2024-05-09T08:57:18.045 INFO:root:teuthology version: 0.0.1.dev271+g1ae7ad8 2024-05-09T08:57:18.045 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} --owner scheduled_teuthology@teuthology --name teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi --verbose -- /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/orig.config.yaml 2024-05-09T08:57:18.100 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T08:57:18.189 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717 branch: squid description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7698717' kernel: client: sha1: distro kdb: 1 sha1: distro ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi mgrmodules: sequential: - print: Enabling mgr modules name: teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 3 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug client: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon down mkfs grace: 300 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - filesystem is online with fewer MDS than max_mds - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - PG_AVAILABILITY - PG_DEGRADED - Reduced data availability - slow metadata IO - SLOW_OPS - slow request sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt install: ceph: flavor: default sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - wsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 2334 sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc sleep_before_teardown: 0 subset: 359/512 suite: fs suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: be85c202969d25af6e130678e7d73a20e9c4ca11 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi069.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX9THh0AhAnU7ZbVL9H/c8OP5pzCaiV1UYunD6EkJPf32epHj6b05KR8z/e5Wj3mGvjRk5aokO9d7SrweOm5pY= smithi103.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM60uisW48ClpMmF1puX7MslpBdWqxj1j9FocMvCZ5jhgBMsWjqySzYa0FxPhOEsSt/m41GP/XP2P9hH1avQ3CM= smithi163.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn8t6I/B1Ndke2vAFl68IOQ3TOq7hSo/j4FuloIK8aChMbV8GlABziQNKfxDENANehxem1LFs8bNCWt6zwliDs= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - sequential: - mgrmodules - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate true - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/dbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: 1ae7ad82388e92a475afff437d49054826c019a1 timestamp: 2024-05-08_21:24:04 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-05-09T08:57:18.189 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa; will attempt to use it 2024-05-09T08:57:18.190 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks 2024-05-09T08:57:18.190 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-05-09T08:57:18.192 INFO:teuthology.task.internal:Checking packages... 2024-05-09T08:57:18.214 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc' 2024-05-09T08:57:18.215 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-05-09T08:57:18.215 INFO:teuthology.packaging:ref: None 2024-05-09T08:57:18.215 INFO:teuthology.packaging:tag: None 2024-05-09T08:57:18.215 INFO:teuthology.packaging:branch: squid 2024-05-09T08:57:18.215 INFO:teuthology.packaging:sha1: 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T08:57:18.215 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-05-09T08:57:18.425 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2529.gbf77c77e 2024-05-09T08:57:18.427 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-05-09T08:57:18.433 INFO:teuthology.task.internal:no buildpackages task found 2024-05-09T08:57:18.433 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-05-09T08:57:18.440 INFO:teuthology.task.internal:Saving configuration 2024-05-09T08:57:18.459 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-05-09T08:57:18.465 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T08:57:18.488 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.282262', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX9THh0AhAnU7ZbVL9H/c8OP5pzCaiV1UYunD6EkJPf32epHj6b05KR8z/e5Wj3mGvjRk5aokO9d7SrweOm5pY='} 2024-05-09T08:57:18.506 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.284308', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM60uisW48ClpMmF1puX7MslpBdWqxj1j9FocMvCZ5jhgBMsWjqySzYa0FxPhOEsSt/m41GP/XP2P9hH1avQ3CM='} 2024-05-09T08:57:18.528 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.283476', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn8t6I/B1Ndke2vAFl68IOQ3TOq7hSo/j4FuloIK8aChMbV8GlABziQNKfxDENANehxem1LFs8bNCWt6zwliDs='} 2024-05-09T08:57:18.528 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-05-09T08:57:18.534 INFO:teuthology.task.internal:roles: ubuntu@smithi069.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-05-09T08:57:18.535 INFO:teuthology.task.internal:roles: ubuntu@smithi103.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-05-09T08:57:18.535 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-05-09T08:57:18.535 INFO:teuthology.run_tasks:Running task console_log... 2024-05-09T08:57:18.643 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f0cb3a7f160>, signals=[15]) 2024-05-09T08:57:18.644 INFO:teuthology.run_tasks:Running task internal.connect... 2024-05-09T08:57:18.650 INFO:teuthology.task.internal:Opening connections... 2024-05-09T08:57:18.650 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T08:57:18.653 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:57:18.734 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi103.front.sepia.ceph.com 2024-05-09T08:57:18.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:57:18.813 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2024-05-09T08:57:18.814 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:57:18.889 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-05-09T08:57:18.895 DEBUG:teuthology.orchestra.run.smithi069:> uname -m 2024-05-09T08:57:18.914 INFO:teuthology.orchestra.run.smithi069.stdout:x86_64 2024-05-09T08:57:18.914 DEBUG:teuthology.orchestra.run.smithi069:> cat /etc/os-release 2024-05-09T08:57:18.971 INFO:teuthology.orchestra.run.smithi069.stdout:NAME="CentOS Stream" 2024-05-09T08:57:18.971 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION="9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:ID="centos" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:VERSION_ID="9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:ANSI_COLOR="0;31" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:57:18.972 INFO:teuthology.orchestra.run.smithi069.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:57:18.973 INFO:teuthology.lock.ops:Updating smithi069.front.sepia.ceph.com on lock server 2024-05-09T08:57:18.991 DEBUG:teuthology.orchestra.run.smithi103:> uname -m 2024-05-09T08:57:19.009 INFO:teuthology.orchestra.run.smithi103.stdout:x86_64 2024-05-09T08:57:19.009 DEBUG:teuthology.orchestra.run.smithi103:> cat /etc/os-release 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:NAME="CentOS Stream" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION="9" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:ID="centos" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:VERSION_ID="9" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:ANSI_COLOR="0;31" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:57:19.067 INFO:teuthology.orchestra.run.smithi103.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:57:19.068 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:57:19.068 INFO:teuthology.orchestra.run.smithi103.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:57:19.068 INFO:teuthology.lock.ops:Updating smithi103.front.sepia.ceph.com on lock server 2024-05-09T08:57:19.091 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2024-05-09T08:57:19.109 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2024-05-09T08:57:19.109 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2024-05-09T08:57:19.167 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="CentOS Stream" 2024-05-09T08:57:19.167 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="9" 2024-05-09T08:57:19.167 INFO:teuthology.orchestra.run.smithi163.stdout:ID="centos" 2024-05-09T08:57:19.167 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE="rhel fedora" 2024-05-09T08:57:19.167 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="9" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:PLATFORM_ID="platform:el9" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="CentOS Stream 9" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:ANSI_COLOR="0;31" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:LOGO="fedora-logo-icon" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://centos.org/" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-05-09T08:57:19.168 INFO:teuthology.orchestra.run.smithi163.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-05-09T08:57:19.168 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2024-05-09T08:57:19.191 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-05-09T08:57:19.204 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-05-09T08:57:19.209 INFO:teuthology.task.internal:Checking for old test directory... 2024-05-09T08:57:19.209 DEBUG:teuthology.orchestra.run.smithi069:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:57:19.211 DEBUG:teuthology.orchestra.run.smithi103:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:57:19.214 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2024-05-09T08:57:19.230 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-05-09T08:57:19.237 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-05-09T08:57:19.237 DEBUG:teuthology.orchestra.run.smithi069:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:57:19.267 DEBUG:teuthology.orchestra.run.smithi103:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:57:19.270 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2024-05-09T08:57:19.400 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-05-09T08:57:19.603 INFO:teuthology.run_tasks:Running task kernel... 2024-05-09T08:57:19.628 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'kdb': 1, 'sha1': 'distro'} 2024-05-09T08:57:19.628 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-05-09T08:57:19.628 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-05-09T08:57:19.628 DEBUG:teuthology.orchestra.run.smithi069:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T08:57:19.645 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T08:57:19.645 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-05-09T08:57:19.703 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-437.el9.x86_64 2024-05-09T08:57:19.703 INFO:teuthology.task.kernel:Running kernel on smithi069: 5.14.0-437.el9.x86_64 2024-05-09T08:57:19.703 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum install -y kernel 2024-05-09T08:57:23.225 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 4.4 MB/s | 8.1 MB 00:01 2024-05-09T08:57:28.500 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 4.5 MB/s | 19 MB 00:04 2024-05-09T08:57:32.801 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 8.8 MB/s | 6.2 MB 00:00 2024-05-09T08:57:34.376 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 38 kB/s | 16 kB 00:00 2024-05-09T08:57:37.611 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 6.8 MB/s | 21 MB 00:03 2024-05-09T08:57:42.736 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-05-09T08:57:44.758 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-05-09T08:57:44.758 INFO:teuthology.orchestra.run.smithi069.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-05-09T08:57:44.846 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-05-09T08:57:44.855 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: Package Architecture Version Repository Size 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: kernel x86_64 5.14.0-444.el9 baseos 484 k 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core x86_64 5.14.0-444.el9 baseos 15 M 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules x86_64 5.14.0-444.el9 baseos 34 M 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core x86_64 5.14.0-444.el9 baseos 28 M 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:================================================================================ 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout:Install 4 Packages 2024-05-09T08:57:44.856 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:57:44.857 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 78 M 2024-05-09T08:57:44.857 INFO:teuthology.orchestra.run.smithi069.stdout:Installed size: 123 M 2024-05-09T08:57:44.857 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-05-09T08:57:45.020 INFO:teuthology.orchestra.run.smithi069.stdout:(1/4): kernel-5.14.0-444.el9.x86_64.rpm 4.8 MB/s | 484 kB 00:00 2024-05-09T08:57:45.604 INFO:teuthology.orchestra.run.smithi069.stdout:(2/4): kernel-core-5.14.0-444.el9.x86_64.rpm 23 MB/s | 15 MB 00:00 2024-05-09T08:57:46.062 INFO:teuthology.orchestra.run.smithi069.stdout:(3/4): kernel-modules-5.14.0-444.el9.x86_64.rpm 30 MB/s | 34 MB 00:01 2024-05-09T08:57:46.413 INFO:teuthology.orchestra.run.smithi069.stdout:(4/4): kernel-modules-core-5.14.0-444.el9.x86_6 20 MB/s | 28 MB 00:01 2024-05-09T08:57:46.413 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-05-09T08:57:46.413 INFO:teuthology.orchestra.run.smithi069.stdout:Total 50 MB/s | 78 MB 00:01 2024-05-09T08:57:47.091 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-05-09T08:57:47.425 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-05-09T08:57:47.426 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-05-09T08:57:48.346 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-05-09T08:57:48.347 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-05-09T08:57:50.741 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-05-09T08:57:51.395 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-core-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:57:51.479 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:57:54.152 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:57:54.289 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:57:59.018 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:57:59.133 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:03.858 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:33.333 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:33.370 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-modules-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:33.718 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: kernel-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:33.718 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-5.14.0-444.el9.x86_64 1/4 2024-05-09T08:58:33.718 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-core-5.14.0-444.el9.x86_64 2/4 2024-05-09T08:58:33.718 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-5.14.0-444.el9.x86_64 3/4 2024-05-09T08:58:34.474 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : kernel-modules-core-5.14.0-444.el9.x86_64 4/4 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-core-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: kernel-modules-core-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:58:34.475 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-05-09T08:58:34.817 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:58:34.888 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.888 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:58:34.888 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-444.el9.x86_64 2024-05-09T08:58:34.889 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-05-09T08:58:34.889 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-05-09T08:58:34.889 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-05-09T08:58:34.889 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi069.front.sepia.ceph.com, path=None, version=distro) 2024-05-09T08:58:34.889 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q kernel | sort -rV | head -n 1 2024-05-09T08:58:34.937 INFO:teuthology.orchestra.run.smithi069.stdout:kernel-5.14.0-444.el9.x86_64 2024-05-09T08:58:34.939 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-444.el9.x86_64 2024-05-09T08:58:34.939 DEBUG:teuthology.orchestra.run.smithi069:> sudo rpm -qi grub2-tools 2024-05-09T08:58:34.969 INFO:teuthology.orchestra.run.smithi069.stdout:Name : grub2-tools 2024-05-09T08:58:34.969 INFO:teuthology.orchestra.run.smithi069.stdout:Epoch : 1 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Version : 2.06 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Release : 68.el9 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Architecture: x86_64 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Group : Unspecified 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Size : 8264254 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:License : GPLv3+ 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-05-09T08:58:34.970 INFO:teuthology.orchestra.run.smithi069.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:Packager : builder@centos.org 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:Vendor : CentOS 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:URL : http://www.gnu.org/software/grub/ 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:Summary : Support tools for GRUB. 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:Description : 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:customizable bootloader with modular architecture. It supports a rich 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:variety of kernel formats, file systems, computer architectures and 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:hardware devices. 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T08:58:34.971 INFO:teuthology.orchestra.run.smithi069.stdout:This subpackage provides tools for support of all platforms. 2024-05-09T08:58:34.972 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-05-09T08:58:34.972 INFO:teuthology.task.kernel:Updating grub on smithi069 to boot 5.14.0-444.el9.x86_64 2024-05-09T08:58:34.973 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-05-09T08:58:35.576 INFO:teuthology.orchestra.run.smithi069.stderr:Generating grub configuration file ... 2024-05-09T08:58:36.990 INFO:teuthology.orchestra.run.smithi069.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-05-09T08:58:37.022 INFO:teuthology.orchestra.run.smithi069.stderr:done 2024-05-09T08:58:37.024 DEBUG:teuthology.orchestra.run.smithi069:> mktemp 2024-05-09T08:58:37.056 INFO:teuthology.orchestra.run.smithi069.stdout:/tmp/tmp.wbLi2IFu4e 2024-05-09T08:58:37.057 DEBUG:teuthology.orchestra.run.smithi069:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.wbLi2IFu4e 2024-05-09T08:58:37.121 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0666 /tmp/tmp.wbLi2IFu4e 2024-05-09T08:58:37.269 DEBUG:teuthology.orchestra.remote:smithi069:/tmp/tmp.wbLi2IFu4e is 6KB 2024-05-09T08:58:37.289 DEBUG:teuthology.orchestra.run.smithi069:> rm -fr /tmp/tmp.wbLi2IFu4e 2024-05-09T08:58:37.303 DEBUG:teuthology.orchestra.run.smithi069:> sudo /bin/ls /boot/loader/entries || true 2024-05-09T08:58:37.364 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-05-09T08:58:37.364 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-05-09T08:58:37.364 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-05-09T08:58:37.364 INFO:teuthology.orchestra.run.smithi069.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64.conf 2024-05-09T08:58:37.365 DEBUG:teuthology.orchestra.run.smithi069:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-444.el9.x86_64 2024-05-09T08:58:37.519 DEBUG:teuthology.orchestra.run.smithi069:> sudo shutdown -r now 2024-05-09T08:59:07.523 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-05-09T08:59:07.523 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T08:59:07.524 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T08:59:25.858 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.69 2024-05-09T08:59:34.859 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T08:59:34.860 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T09:00:34.862 DEBUG:teuthology.orchestra.remote:timed out 2024-05-09T09:00:46.863 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T09:00:46.864 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T09:00:47.225 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-05-09T09:00:47.578 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T09:00:47.578 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-444.el9.x86_64"... 2024-05-09T09:00:47.578 DEBUG:teuthology.orchestra.run.smithi069:> uname -r 2024-05-09T09:00:47.594 INFO:teuthology.orchestra.run.smithi069.stdout:5.14.0-444.el9.x86_64 2024-05-09T09:00:47.595 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-444.el9.x86_64 vs 5.14.0-444.el9.x86_64 2024-05-09T09:00:47.595 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-05-09T09:00:47.595 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-05-09T09:00:48.596 DEBUG:teuthology.parallel:result is None 2024-05-09T09:00:48.597 INFO:teuthology.run_tasks:Running task internal.base... 2024-05-09T09:00:48.604 INFO:teuthology.task.internal:Creating test directory... 2024-05-09T09:00:48.604 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T09:00:48.606 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T09:00:48.609 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-05-09T09:00:48.627 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-05-09T09:00:48.634 INFO:teuthology.run_tasks:Running task internal.archive... 2024-05-09T09:00:48.641 INFO:teuthology.task.internal:Creating archive directory... 2024-05-09T09:00:48.641 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T09:00:48.665 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T09:00:48.668 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-05-09T09:00:48.998 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-05-09T09:00:49.005 INFO:teuthology.task.internal:Enabling coredump saving... 2024-05-09T09:00:49.005 DEBUG:teuthology.orchestra.run.smithi103:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T09:00:49.021 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:00:49.022 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2024-05-09T09:00:49.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:00:49.039 DEBUG:teuthology.orchestra.run.smithi069:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T09:00:49.041 DEBUG:teuthology.orchestra.run.smithi103:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T09:00:49.064 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-05-09T09:00:49.099 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.112 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.117 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.135 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.156 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.165 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-05-09T09:00:49.167 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-05-09T09:00:49.174 INFO:teuthology.task.internal:Configuring sudo... 2024-05-09T09:00:49.174 DEBUG:teuthology.orchestra.run.smithi069:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T09:00:49.177 DEBUG:teuthology.orchestra.run.smithi103:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T09:00:49.179 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-05-09T09:00:49.205 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-05-09T09:00:49.215 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-05-09T09:00:49.215 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T09:00:49.245 DEBUG:teuthology.orchestra.run.smithi103:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T09:00:49.247 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-05-09T09:00:49.263 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.426 DEBUG:teuthology.orchestra.run.smithi069:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:49.469 DEBUG:teuthology.orchestra.run.smithi069:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:49.508 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:00:49.508 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T09:00:49.573 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.615 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.676 DEBUG:teuthology.orchestra.run.smithi103:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:49.757 DEBUG:teuthology.orchestra.run.smithi103:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:49.801 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:00:49.801 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T09:00:49.867 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-05-09T09:00:49.974 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:50.056 DEBUG:teuthology.orchestra.run.smithi163:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-05-09T09:00:50.100 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:00:50.101 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-05-09T09:00:50.167 DEBUG:teuthology.orchestra.run.smithi069:> sudo service rsyslog restart 2024-05-09T09:00:50.170 DEBUG:teuthology.orchestra.run.smithi103:> sudo service rsyslog restart 2024-05-09T09:00:50.172 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2024-05-09T09:00:50.208 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:00:50.211 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:00:50.253 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T09:00:50.659 INFO:teuthology.run_tasks:Running task internal.timer... 2024-05-09T09:00:50.667 INFO:teuthology.task.internal:Starting timer... 2024-05-09T09:00:50.667 INFO:teuthology.run_tasks:Running task pcp... 2024-05-09T09:00:50.676 INFO:teuthology.run_tasks:Running task selinux... 2024-05-09T09:00:50.684 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-05-09T09:00:50.685 DEBUG:teuthology.orchestra.run.smithi069:> sudo service auditd rotate 2024-05-09T09:00:50.763 INFO:teuthology.orchestra.run.smithi069.stdout:Rotating logs: 2024-05-09T09:00:50.765 DEBUG:teuthology.orchestra.run.smithi103:> sudo service auditd rotate 2024-05-09T09:00:50.837 INFO:teuthology.orchestra.run.smithi103.stdout:Rotating logs: 2024-05-09T09:00:50.839 DEBUG:teuthology.orchestra.run.smithi163:> sudo service auditd rotate 2024-05-09T09:00:50.908 INFO:teuthology.orchestra.run.smithi163.stdout:Rotating logs: 2024-05-09T09:00:50.910 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-05-09T09:00:50.910 DEBUG:teuthology.orchestra.run.smithi069:> /usr/sbin/getenforce 2024-05-09T09:00:50.937 INFO:teuthology.orchestra.run.smithi069.stdout:Permissive 2024-05-09T09:00:50.937 DEBUG:teuthology.orchestra.run.smithi103:> /usr/sbin/getenforce 2024-05-09T09:00:50.963 INFO:teuthology.orchestra.run.smithi103.stdout:Permissive 2024-05-09T09:00:50.963 DEBUG:teuthology.orchestra.run.smithi163:> /usr/sbin/getenforce 2024-05-09T09:00:50.988 INFO:teuthology.orchestra.run.smithi163.stdout:Permissive 2024-05-09T09:00:50.988 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi069.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi103.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi163.front.sepia.ceph.com': 'permissive'} 2024-05-09T09:00:50.988 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:00:51.016 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:00:51.017 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:00:51.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:00:51.044 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T09:00:51.073 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:00:51.073 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-05-09T09:00:51.074 DEBUG:teuthology.orchestra.run.smithi069:> sudo /usr/sbin/setenforce permissive 2024-05-09T09:00:51.101 DEBUG:teuthology.orchestra.run.smithi103:> sudo /usr/sbin/setenforce permissive 2024-05-09T09:00:51.126 DEBUG:teuthology.orchestra.run.smithi163:> sudo /usr/sbin/setenforce permissive 2024-05-09T09:00:51.152 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-05-09T09:00:51.160 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-05-09T09:00:51.176 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-05-09T09:00:51.178 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi069.front.sepia.ceph.com,smithi103.front.sepia.ceph.com,smithi163.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-05-09T09:05:28.546 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi069.front.sepia.ceph.com'), Remote(name='ubuntu@smithi103.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com')] 2024-05-09T09:05:28.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T09:05:28.547 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi069.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T09:05:28.623 DEBUG:teuthology.orchestra.run.smithi069:> true 2024-05-09T09:05:28.714 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi069.front.sepia.ceph.com' 2024-05-09T09:05:28.714 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-09T09:05:28.715 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi103.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T09:05:28.787 DEBUG:teuthology.orchestra.run.smithi103:> true 2024-05-09T09:05:28.866 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi103.front.sepia.ceph.com' 2024-05-09T09:05:28.866 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-09T09:05:28.867 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-05-09T09:05:28.940 DEBUG:teuthology.orchestra.run.smithi163:> true 2024-05-09T09:05:29.017 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi163.front.sepia.ceph.com' 2024-05-09T09:05:29.018 INFO:teuthology.run_tasks:Running task clock... 2024-05-09T09:05:29.028 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-05-09T09:05:29.029 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T09:05:29.029 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:05:29.032 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T09:05:29.032 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:05:29.034 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-05-09T09:05:29.034 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T09:05:29.060 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T09:05:29.061 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T09:05:29.075 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T09:05:29.076 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T09:05:29.087 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-05-09T09:05:29.099 INFO:teuthology.orchestra.run.smithi103.stderr:sudo: ntpd: command not found 2024-05-09T09:05:29.099 INFO:teuthology.orchestra.run.smithi069.stderr:sudo: ntpd: command not found 2024-05-09T09:05:29.101 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-05-09T09:05:29.109 INFO:teuthology.orchestra.run.smithi103.stdout:506 Cannot talk to daemon 2024-05-09T09:05:29.110 INFO:teuthology.orchestra.run.smithi069.stdout:506 Cannot talk to daemon 2024-05-09T09:05:29.123 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T09:05:29.124 INFO:teuthology.orchestra.run.smithi163.stderr:sudo: ntpd: command not found 2024-05-09T09:05:29.124 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T09:05:29.134 INFO:teuthology.orchestra.run.smithi163.stdout:506 Cannot talk to daemon 2024-05-09T09:05:29.136 INFO:teuthology.orchestra.run.smithi103.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T09:05:29.138 INFO:teuthology.orchestra.run.smithi069.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T09:05:29.148 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-05-09T09:05:29.161 INFO:teuthology.orchestra.run.smithi163.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-05-09T09:05:29.197 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:05:29.198 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:05:29.200 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:05:29.200 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-05-09T09:05:29.200 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.201 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.201 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.201 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.202 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.217 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.220 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T09:05:29.221 INFO:teuthology.run_tasks:Running task install... 2024-05-09T09:05:29.229 DEBUG:teuthology.task.install:project ceph 2024-05-09T09:05:29.229 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'}} 2024-05-09T09:05:29.229 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc'} 2024-05-09T09:05:29.229 INFO:teuthology.task.install:Using flavor: default 2024-05-09T09:05:29.237 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-05-09T09:05:29.237 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-05-09T09:05:29.237 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T09:05:29.237 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:05:29.239 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T09:05:29.239 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:05:29.240 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'tag': None, 'wait_for_package': False} 2024-05-09T09:05:29.240 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:05:29.383 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T09:05:29.383 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T09:05:29.395 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T09:05:29.396 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T09:05:29.454 INFO:teuthology.task.install.rpm:Pulling from https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/ 2024-05-09T09:05:29.454 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2515.g96805dcf 2024-05-09T09:05:29.491 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T09:05:29.491 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:05:29.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T09:05:29.502 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T09:05:29.502 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:05:29.502 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T09:05:29.515 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T09:05:29.515 DEBUG:teuthology.orchestra.run.smithi103:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T09:05:29.527 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T09:05:29.527 DEBUG:teuthology.orchestra.run.smithi163:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T09:05:29.580 DEBUG:teuthology.orchestra.run.smithi103:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T09:05:29.591 DEBUG:teuthology.orchestra.run.smithi163:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T09:05:29.621 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-05-09T09:05:29.621 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:05:29.621 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-05-09T09:05:29.647 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-05-09T09:05:29.647 DEBUG:teuthology.orchestra.run.smithi069:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-05-09T09:05:29.653 DEBUG:teuthology.orchestra.run.smithi103:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T09:05:29.664 DEBUG:teuthology.orchestra.run.smithi163:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T09:05:29.713 DEBUG:teuthology.orchestra.run.smithi069:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-05-09T09:05:29.716 INFO:teuthology.orchestra.run.smithi103.stdout:check_obsoletes = 1 2024-05-09T09:05:29.718 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum clean all 2024-05-09T09:05:29.727 INFO:teuthology.orchestra.run.smithi163.stdout:check_obsoletes = 1 2024-05-09T09:05:29.729 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum clean all 2024-05-09T09:05:29.788 DEBUG:teuthology.orchestra.run.smithi069:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-05-09T09:05:29.853 INFO:teuthology.orchestra.run.smithi069.stdout:check_obsoletes = 1 2024-05-09T09:05:29.855 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum clean all 2024-05-09T09:05:29.946 INFO:teuthology.orchestra.run.smithi103.stdout:54 files removed 2024-05-09T09:05:29.955 INFO:teuthology.orchestra.run.smithi163.stdout:54 files removed 2024-05-09T09:05:29.972 DEBUG:teuthology.orchestra.run.smithi103:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T09:05:29.982 DEBUG:teuthology.orchestra.run.smithi163:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T09:05:30.102 INFO:teuthology.orchestra.run.smithi069.stdout:54 files removed 2024-05-09T09:05:30.129 DEBUG:teuthology.orchestra.run.smithi069:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-05-09T09:05:30.573 INFO:teuthology.orchestra.run.smithi163.stdout:ceph packages for x86_64 254 kB/s | 83 kB 00:00 2024-05-09T09:05:30.639 INFO:teuthology.orchestra.run.smithi103.stdout:ceph packages for x86_64 205 kB/s | 83 kB 00:00 2024-05-09T09:05:30.754 INFO:teuthology.orchestra.run.smithi069.stdout:ceph packages for x86_64 237 kB/s | 83 kB 00:00 2024-05-09T09:05:30.974 INFO:teuthology.orchestra.run.smithi163.stdout:ceph noarch packages 32 kB/s | 11 kB 00:00 2024-05-09T09:05:31.032 INFO:teuthology.orchestra.run.smithi103.stdout:ceph noarch packages 33 kB/s | 11 kB 00:00 2024-05-09T09:05:31.122 INFO:teuthology.orchestra.run.smithi069.stdout:ceph noarch packages 37 kB/s | 11 kB 00:00 2024-05-09T09:05:31.325 INFO:teuthology.orchestra.run.smithi163.stdout:ceph source packages 6.0 kB/s | 1.9 kB 00:00 2024-05-09T09:05:31.399 INFO:teuthology.orchestra.run.smithi103.stdout:ceph source packages 5.7 kB/s | 1.9 kB 00:00 2024-05-09T09:05:31.539 INFO:teuthology.orchestra.run.smithi069.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-05-09T09:05:31.874 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.1 MB 00:00 2024-05-09T09:05:32.723 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - BaseOS 7.1 MB/s | 8.1 MB 00:01 2024-05-09T09:05:32.900 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - BaseOS 5.2 MB/s | 8.1 MB 00:01 2024-05-09T09:05:34.567 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - AppStream 12 MB/s | 19 MB 00:01 2024-05-09T09:05:34.749 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-05-09T09:05:35.751 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - AppStream 11 MB/s | 19 MB 00:01 2024-05-09T09:05:38.702 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-09T09:05:38.866 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - CRB 11 MB/s | 6.2 MB 00:00 2024-05-09T09:05:40.275 INFO:teuthology.orchestra.run.smithi069.stdout:CentOS Stream 9 - Extras packages 78 kB/s | 16 kB 00:00 2024-05-09T09:05:40.945 INFO:teuthology.orchestra.run.smithi103.stdout:CentOS Stream 9 - Extras packages 15 kB/s | 16 kB 00:01 2024-05-09T09:05:41.351 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - CRB 3.0 MB/s | 6.2 MB 00:02 2024-05-09T09:05:41.713 INFO:teuthology.orchestra.run.smithi069.stdout:Extra Packages for Enterprise Linux 16 MB/s | 21 MB 00:01 2024-05-09T09:05:42.340 INFO:teuthology.orchestra.run.smithi103.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-05-09T09:05:42.860 INFO:teuthology.orchestra.run.smithi163.stdout:CentOS Stream 9 - Extras packages 49 kB/s | 16 kB 00:00 2024-05-09T09:05:43.987 INFO:teuthology.orchestra.run.smithi163.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-05-09T09:05:46.902 INFO:teuthology.orchestra.run.smithi069.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-05-09T09:05:47.530 INFO:teuthology.orchestra.run.smithi103.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-05-09T09:05:49.170 INFO:teuthology.orchestra.run.smithi163.stdout:lab-extras 39 kB/s | 1.7 kB 00:00 2024-05-09T09:05:49.254 INFO:teuthology.orchestra.run.smithi069.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.254 INFO:teuthology.orchestra.run.smithi069.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.265 INFO:teuthology.orchestra.run.smithi069.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T09:05:49.266 INFO:teuthology.orchestra.run.smithi069.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.267 INFO:teuthology.orchestra.run.smithi069.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T09:05:49.268 INFO:teuthology.orchestra.run.smithi069.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T09:05:49.269 INFO:teuthology.orchestra.run.smithi069.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T09:05:49.272 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T09:05:49.273 INFO:teuthology.orchestra.run.smithi069.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T09:05:49.333 INFO:teuthology.orchestra.run.smithi069.stdout:Dependencies resolved. 2024-05-09T09:05:49.349 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-05-09T09:05:49.349 INFO:teuthology.orchestra.run.smithi069.stdout: Package Arch Version Repository Size 2024-05-09T09:05:49.349 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-05-09T09:05:49.349 INFO:teuthology.orchestra.run.smithi069.stdout:Installing: 2024-05-09T09:05:49.349 INFO:teuthology.orchestra.run.smithi069.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T09:05:49.350 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T09:05:49.351 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrading: 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T09:05:49.352 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout:Installing dependencies: 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T09:05:49.353 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T09:05:49.354 INFO:teuthology.orchestra.run.smithi069.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T09:05:49.355 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T09:05:49.356 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T09:05:49.357 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T09:05:49.358 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T09:05:49.359 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T09:05:49.360 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout:Installing weak dependencies: 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction Summary 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout:======================================================================================= 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout:Install 148 Packages 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout:Upgrade 8 Packages 2024-05-09T09:05:49.361 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:05:49.362 INFO:teuthology.orchestra.run.smithi069.stdout:Total download size: 225 M 2024-05-09T09:05:49.362 INFO:teuthology.orchestra.run.smithi069.stdout:Downloading Packages: 2024-05-09T09:05:49.822 INFO:teuthology.orchestra.run.smithi103.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.823 INFO:teuthology.orchestra.run.smithi103.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.833 INFO:teuthology.orchestra.run.smithi103.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T09:05:49.834 INFO:teuthology.orchestra.run.smithi103.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T09:05:49.835 INFO:teuthology.orchestra.run.smithi103.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T09:05:49.836 INFO:teuthology.orchestra.run.smithi103.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T09:05:49.838 INFO:teuthology.orchestra.run.smithi103.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T09:05:49.840 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T09:05:49.841 INFO:teuthology.orchestra.run.smithi103.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T09:05:49.902 INFO:teuthology.orchestra.run.smithi103.stdout:Dependencies resolved. 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: Package Arch Version Repository Size 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout:Installing: 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T09:05:49.916 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T09:05:49.917 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:49.918 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrading: 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout:Installing dependencies: 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T09:05:49.919 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T09:05:49.920 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T09:05:49.921 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T09:05:49.922 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T09:05:49.923 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T09:05:49.924 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T09:05:49.925 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T09:05:49.926 INFO:teuthology.orchestra.run.smithi103.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout:Installing weak dependencies: 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction Summary 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout:======================================================================================= 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout:Install 148 Packages 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout:Upgrade 8 Packages 2024-05-09T09:05:49.927 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:05:49.928 INFO:teuthology.orchestra.run.smithi103.stdout:Total download size: 225 M 2024-05-09T09:05:49.928 INFO:teuthology.orchestra.run.smithi103.stdout:Downloading Packages: 2024-05-09T09:05:50.070 INFO:teuthology.orchestra.run.smithi069.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 30 kB/s | 6.6 kB 00:00 2024-05-09T09:05:50.548 INFO:teuthology.orchestra.run.smithi103.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-05-09T09:05:50.762 INFO:teuthology.orchestra.run.smithi069.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 1.3 MB/s | 889 kB 00:00 2024-05-09T09:05:50.954 INFO:teuthology.orchestra.run.smithi069.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 757 kB/s | 145 kB 00:00 2024-05-09T09:05:51.198 INFO:teuthology.orchestra.run.smithi103.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 1.3 MB/s | 889 kB 00:00 2024-05-09T09:05:51.265 INFO:teuthology.orchestra.run.smithi103.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 2.1 MB/s | 145 kB 00:00 2024-05-09T09:05:51.288 INFO:teuthology.orchestra.run.smithi069.stdout:(4/156): ceph-common-19.0.0-2515.g96805dcf.el9. 15 MB/s | 21 MB 00:01 2024-05-09T09:05:51.363 INFO:teuthology.orchestra.run.smithi069.stdout:(5/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 3.5 MB/s | 5.4 MB 00:01 2024-05-09T09:05:51.446 INFO:teuthology.orchestra.run.smithi163.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:51.447 INFO:teuthology.orchestra.run.smithi163.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-05-09T09:05:51.458 INFO:teuthology.orchestra.run.smithi163.stdout:Package NetworkManager-1:1.47.4-1.el9.x86_64 is already installed. 2024-05-09T09:05:51.458 INFO:teuthology.orchestra.run.smithi163.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-05-09T09:05:51.459 INFO:teuthology.orchestra.run.smithi163.stdout:Package util-linux-2.37.4-18.el9.x86_64 is already installed. 2024-05-09T09:05:51.460 INFO:teuthology.orchestra.run.smithi163.stdout:Package libacl-devel-2.3.1-4.el9.x86_64 is already installed. 2024-05-09T09:05:51.461 INFO:teuthology.orchestra.run.smithi163.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-05-09T09:05:51.464 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsdump-3.1.12-4.el9.x86_64 is already installed. 2024-05-09T09:05:51.464 INFO:teuthology.orchestra.run.smithi163.stdout:Package xfsprogs-6.3.0-1.el9.x86_64 is already installed. 2024-05-09T09:05:51.525 INFO:teuthology.orchestra.run.smithi163.stdout:Dependencies resolved. 2024-05-09T09:05:51.546 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-09T09:05:51.547 INFO:teuthology.orchestra.run.smithi163.stdout: Package Arch Version Repository Size 2024-05-09T09:05:51.547 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-09T09:05:51.547 INFO:teuthology.orchestra.run.smithi163.stdout:Installing: 2024-05-09T09:05:51.548 INFO:teuthology.orchestra.run.smithi163.stdout: bison x86_64 3.7.4-5.el9 appstream 943 k 2024-05-09T09:05:51.548 INFO:teuthology.orchestra.run.smithi163.stdout: ceph x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 6.6 k 2024-05-09T09:05:51.548 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.4 M 2024-05-09T09:05:51.548 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 889 k 2024-05-09T09:05:51.548 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 145 k 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 1.5 M 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 142 k 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 3.5 M 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 7.4 M 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 49 k 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 10 M 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 48 M 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 264 k 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 763 k 2024-05-09T09:05:51.549 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 222 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 27 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 18 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: flex x86_64 2.6.4-9.el9 appstream 313 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel x86_64 0.3.111-13.el9 appstream 11 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 32 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 746 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 127 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: libtool x86_64 2.4.6-45.el9 appstream 582 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-05-09T09:05:51.550 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel x86_64 1:3.2.1-1.el9 appstream 4.4 M 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql x86_64 13.14-1.el9 appstream 1.6 M 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib x86_64 13.14-1.el9 appstream 911 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server x86_64 13.14-1.el9 appstream 5.8 M 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 162 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 321 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 298 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 99 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 85 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 171 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrading: 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager x86_64 1:1.47.5-1.el9 baseos 2.3 M 2024-05-09T09:05:51.551 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm x86_64 1:1.47.5-1.el9 baseos 1.8 M 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team x86_64 1:1.47.5-1.el9 baseos 42 k 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui x86_64 1:1.47.5-1.el9 baseos 248 k 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.4 M 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 3.1 M 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: openssl x86_64 1:3.2.1-1.el9 baseos 1.3 M 2024-05-09T09:05:51.552 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs x86_64 1:3.2.1-1.el9 baseos 2.4 M 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout:Installing dependencies: 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: automake noarch 1.16.2-8.el9 appstream 689 k 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options x86_64 1.75.0-8.el9 appstream 107 k 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 21 M 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 32 k 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 2.4 M 2024-05-09T09:05:51.553 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 249 k 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 4.7 M 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 17 M 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2515.g96805dcf.el9 ceph-noarch 16 k 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 25 k 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-05-09T09:05:51.554 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow x86_64 9.0.0-11.el9 epel 4.4 M 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc noarch 9.0.0-11.el9 epel 26 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 163 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-05-09T09:05:51.555 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 500 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 5.3 M 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel x86_64 1.5.1-2.el9 appstream 47 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-05-09T09:05:51.556 INFO:teuthology.orchestra.run.smithi163.stdout: lua x86_64 5.4.4-4.el9 appstream 188 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs x86_64 9.0.0-11.el9 epel 839 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs x86_64 13.14-1.el9 appstream 133 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf x86_64 3.14.0-13.el9 appstream 1.0 M 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs noarch 20.3.0-7.el9 appstream 87 k 2024-05-09T09:05:51.557 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 45 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common x86_64 2:19.0.0-2515.g96805dcf.el9 ceph 135 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-05-09T09:05:51.558 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth noarch 1:2.29.0-1.el9 epel 206 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-05-09T09:05:51.559 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging noarch 20.9-5.el9 appstream 77 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf noarch 3.14.0-13.el9 appstream 269 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-05-09T09:05:51.560 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth noarch 0.2.5-8.el9 appstream 44 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-05-09T09:05:51.561 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib x86_64 24.02.0-1.el9 appstream 221 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs x86_64 1.2.0-1.el9 appstream 47 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: socat x86_64 1.7.4.1-5.el9 appstream 305 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: uuid x86_64 1.6.2-55.el9 appstream 58 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel x86_64 1.2.11-41.el9 appstream 45 k 2024-05-09T09:05:51.562 INFO:teuthology.orchestra.run.smithi163.stdout:Installing weak dependencies: 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service x86_64 24.02.0-1.el9 appstream 36 k 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction Summary 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout:======================================================================================= 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout:Install 148 Packages 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout:Upgrade 8 Packages 2024-05-09T09:05:51.563 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:05:51.564 INFO:teuthology.orchestra.run.smithi163.stdout:Total download size: 225 M 2024-05-09T09:05:51.564 INFO:teuthology.orchestra.run.smithi163.stdout:Downloading Packages: 2024-05-09T09:05:51.588 INFO:teuthology.orchestra.run.smithi069.stdout:(6/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 5.0 MB/s | 1.5 MB 00:00 2024-05-09T09:05:51.924 INFO:teuthology.orchestra.run.smithi103.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 3.6 MB/s | 2.4 MB 00:00 2024-05-09T09:05:51.938 INFO:teuthology.orchestra.run.smithi069.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 8.2 MB/s | 4.7 MB 00:00 2024-05-09T09:05:52.122 INFO:teuthology.orchestra.run.smithi163.stdout:(1/156): ceph-19.0.0-2515.g96805dcf.el9.x86_64. 23 kB/s | 6.6 kB 00:00 2024-05-09T09:05:52.141 INFO:teuthology.orchestra.run.smithi103.stdout:(5/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-05-09T09:05:52.447 INFO:teuthology.orchestra.run.smithi069.stdout:(8/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.6 MB/s | 2.4 MB 00:01 2024-05-09T09:05:52.466 INFO:teuthology.orchestra.run.smithi103.stdout:(6/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 2.8 MB/s | 1.5 MB 00:00 2024-05-09T09:05:52.539 INFO:teuthology.orchestra.run.smithi069.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 274 kB/s | 25 kB 00:00 2024-05-09T09:05:53.141 INFO:teuthology.orchestra.run.smithi103.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 4.7 MB/s | 4.7 MB 00:01 2024-05-09T09:05:53.163 INFO:teuthology.orchestra.run.smithi163.stdout:(2/156): ceph-fuse-19.0.0-2515.g96805dcf.el9.x8 854 kB/s | 889 kB 00:01 2024-05-09T09:05:53.380 INFO:teuthology.orchestra.run.smithi163.stdout:(3/156): ceph-immutable-object-cache-19.0.0-251 670 kB/s | 145 kB 00:00 2024-05-09T09:05:53.497 INFO:teuthology.orchestra.run.smithi069.stdout:(10/156): ceph-radosgw-19.0.0-2515.g96805dcf.el 6.6 MB/s | 10 MB 00:01 2024-05-09T09:05:53.606 INFO:teuthology.orchestra.run.smithi069.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 2.0 MB/s | 222 kB 00:00 2024-05-09T09:05:53.698 INFO:teuthology.orchestra.run.smithi069.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 295 kB/s | 27 kB 00:00 2024-05-09T09:05:53.790 INFO:teuthology.orchestra.run.smithi069.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 346 kB/s | 32 kB 00:00 2024-05-09T09:05:53.990 INFO:teuthology.orchestra.run.smithi069.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 3.6 MB/s | 746 kB 00:00 2024-05-09T09:05:54.099 INFO:teuthology.orchestra.run.smithi069.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 1.5 MB/s | 163 kB 00:00 2024-05-09T09:05:54.191 INFO:teuthology.orchestra.run.smithi069.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.3 MB/s | 127 kB 00:00 2024-05-09T09:05:54.375 INFO:teuthology.orchestra.run.smithi069.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 2.7 MB/s | 500 kB 00:00 2024-05-09T09:05:54.550 INFO:teuthology.orchestra.run.smithi069.stdout:(18/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 5.7 MB/s | 17 MB 00:02 2024-05-09T09:05:54.600 INFO:teuthology.orchestra.run.smithi069.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 910 kB/s | 45 kB 00:00 2024-05-09T09:05:54.659 INFO:teuthology.orchestra.run.smithi069.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 2.3 MB/s | 135 kB 00:00 2024-05-09T09:05:54.659 INFO:teuthology.orchestra.run.smithi103.stdout:(8/156): ceph-common-19.0.0-2515.g96805dcf.el9. 4.8 MB/s | 21 MB 00:04 2024-05-09T09:05:54.725 INFO:teuthology.orchestra.run.smithi103.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 379 kB/s | 25 kB 00:00 2024-05-09T09:05:54.734 INFO:teuthology.orchestra.run.smithi069.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 2.1 MB/s | 162 kB 00:00 2024-05-09T09:05:54.818 INFO:teuthology.orchestra.run.smithi069.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 3.8 MB/s | 321 kB 00:00 2024-05-09T09:05:54.876 INFO:teuthology.orchestra.run.smithi103.stdout:(10/156): ceph-radosgw-19.0.0-2515.g96805dcf.el 5.9 MB/s | 10 MB 00:01 2024-05-09T09:05:54.897 INFO:teuthology.orchestra.run.smithi163.stdout:(4/156): ceph-mds-19.0.0-2515.g96805dcf.el9.x86 1.6 MB/s | 2.4 MB 00:01 2024-05-09T09:05:54.901 INFO:teuthology.orchestra.run.smithi069.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 3.5 MB/s | 298 kB 00:00 2024-05-09T09:05:54.942 INFO:teuthology.orchestra.run.smithi103.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 3.3 MB/s | 222 kB 00:00 2024-05-09T09:05:54.985 INFO:teuthology.orchestra.run.smithi069.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.2 MB/s | 99 kB 00:00 2024-05-09T09:05:55.018 INFO:teuthology.orchestra.run.smithi103.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 360 kB/s | 27 kB 00:00 2024-05-09T09:05:55.060 INFO:teuthology.orchestra.run.smithi069.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.1 MB/s | 85 kB 00:00 2024-05-09T09:05:55.085 INFO:teuthology.orchestra.run.smithi103.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 477 kB/s | 32 kB 00:00 2024-05-09T09:05:55.252 INFO:teuthology.orchestra.run.smithi103.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 4.4 MB/s | 746 kB 00:00 2024-05-09T09:05:55.277 INFO:teuthology.orchestra.run.smithi069.stdout:(26/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 5.9 MB/s | 5.3 MB 00:00 2024-05-09T09:05:55.319 INFO:teuthology.orchestra.run.smithi103.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 2.4 MB/s | 163 kB 00:00 2024-05-09T09:05:55.394 INFO:teuthology.orchestra.run.smithi103.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 1.7 MB/s | 127 kB 00:00 2024-05-09T09:05:55.394 INFO:teuthology.orchestra.run.smithi069.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 1.4 MB/s | 171 kB 00:00 2024-05-09T09:05:55.503 INFO:teuthology.orchestra.run.smithi069.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 293 kB/s | 32 kB 00:00 2024-05-09T09:05:55.544 INFO:teuthology.orchestra.run.smithi103.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 3.3 MB/s | 500 kB 00:00 2024-05-09T09:05:55.603 INFO:teuthology.orchestra.run.smithi069.stdout:(29/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 5.7 MB/s | 3.1 MB 00:00 2024-05-09T09:05:55.628 INFO:teuthology.orchestra.run.smithi069.stdout:(30/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 1.1 MB/s | 142 kB 00:00 2024-05-09T09:05:55.797 INFO:teuthology.orchestra.run.smithi163.stdout:(5/156): ceph-mgr-19.0.0-2515.g96805dcf.el9.x86 1.7 MB/s | 1.5 MB 00:00 2024-05-09T09:05:56.114 INFO:teuthology.orchestra.run.smithi163.stdout:(6/156): ceph-base-19.0.0-2515.g96805dcf.el9.x8 1.3 MB/s | 5.4 MB 00:04 2024-05-09T09:05:56.412 INFO:teuthology.orchestra.run.smithi069.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 4.3 MB/s | 3.5 MB 00:00 2024-05-09T09:05:56.486 INFO:teuthology.orchestra.run.smithi103.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 5.6 MB/s | 5.3 MB 00:00 2024-05-09T09:05:56.487 INFO:teuthology.orchestra.run.smithi069.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 3.3 MB/s | 249 kB 00:00 2024-05-09T09:05:56.553 INFO:teuthology.orchestra.run.smithi103.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 679 kB/s | 45 kB 00:00 2024-05-09T09:05:56.554 INFO:teuthology.orchestra.run.smithi069.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 739 kB/s | 49 kB 00:00 2024-05-09T09:05:56.628 INFO:teuthology.orchestra.run.smithi103.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 1.8 MB/s | 135 kB 00:00 2024-05-09T09:05:56.629 INFO:teuthology.orchestra.run.smithi069.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 219 kB/s | 16 kB 00:00 2024-05-09T09:05:56.678 INFO:teuthology.orchestra.run.smithi103.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 3.2 MB/s | 162 kB 00:00 2024-05-09T09:05:56.713 INFO:teuthology.orchestra.run.smithi069.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 3.1 MB/s | 264 kB 00:00 2024-05-09T09:05:56.754 INFO:teuthology.orchestra.run.smithi103.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 4.2 MB/s | 321 kB 00:00 2024-05-09T09:05:56.846 INFO:teuthology.orchestra.run.smithi103.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 3.2 MB/s | 298 kB 00:00 2024-05-09T09:05:56.913 INFO:teuthology.orchestra.run.smithi103.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.5 MB/s | 99 kB 00:00 2024-05-09T09:05:56.947 INFO:teuthology.orchestra.run.smithi069.stdout:(36/156): ceph-mgr-diskprediction-local-19.0.0- 5.6 MB/s | 7.4 MB 00:01 2024-05-09T09:05:56.971 INFO:teuthology.orchestra.run.smithi103.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.4 MB/s | 85 kB 00:00 2024-05-09T09:05:56.972 INFO:teuthology.orchestra.run.smithi069.stdout:(37/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 2.9 MB/s | 763 kB 00:00 2024-05-09T09:05:57.022 INFO:teuthology.orchestra.run.smithi069.stdout:(38/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 244 kB/s | 18 kB 00:00 2024-05-09T09:05:57.372 INFO:teuthology.orchestra.run.smithi103.stdout:(26/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 7.7 MB/s | 3.1 MB 00:00 2024-05-09T09:05:57.406 INFO:teuthology.orchestra.run.smithi069.stdout:(39/156): fuse-2.9.9-15.el9.x86_64.rpm 185 kB/s | 80 kB 00:00 2024-05-09T09:05:57.431 INFO:teuthology.orchestra.run.smithi103.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.9 MB/s | 171 kB 00:00 2024-05-09T09:05:57.489 INFO:teuthology.orchestra.run.smithi103.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 546 kB/s | 32 kB 00:00 2024-05-09T09:05:57.549 INFO:teuthology.orchestra.run.smithi103.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 2.4 MB/s | 142 kB 00:00 2024-05-09T09:05:57.673 INFO:teuthology.orchestra.run.smithi069.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 733 kB/s | 195 kB 00:00 2024-05-09T09:05:57.715 INFO:teuthology.orchestra.run.smithi069.stdout:(41/156): libgfortran-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 803 kB 00:00 2024-05-09T09:05:57.747 INFO:teuthology.orchestra.run.smithi163.stdout:(7/156): ceph-mon-19.0.0-2515.g96805dcf.el9.x86 2.4 MB/s | 4.7 MB 00:01 2024-05-09T09:05:57.799 INFO:teuthology.orchestra.run.smithi103.stdout:(30/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 3.1 MB/s | 17 MB 00:05 2024-05-09T09:05:57.857 INFO:teuthology.orchestra.run.smithi069.stdout:(42/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 253 kB 00:00 2024-05-09T09:05:57.965 INFO:teuthology.orchestra.run.smithi069.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 985 kB/s | 106 kB 00:00 2024-05-09T09:05:58.040 INFO:teuthology.orchestra.run.smithi103.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 7.1 MB/s | 3.5 MB 00:00 2024-05-09T09:05:58.065 INFO:teuthology.orchestra.run.smithi069.stdout:(44/156): python3-cryptography-36.0.1-4.el9.x86 3.5 MB/s | 1.2 MB 00:00 2024-05-09T09:05:58.091 INFO:teuthology.orchestra.run.smithi069.stdout:(45/156): python3-pycparser-2.20-6.el9.noarch.r 1.1 MB/s | 135 kB 00:00 2024-05-09T09:05:58.099 INFO:teuthology.orchestra.run.smithi103.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 4.2 MB/s | 249 kB 00:00 2024-05-09T09:05:58.158 INFO:teuthology.orchestra.run.smithi103.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 846 kB/s | 49 kB 00:00 2024-05-09T09:05:58.175 INFO:teuthology.orchestra.run.smithi069.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 1.1 MB/s | 125 kB 00:00 2024-05-09T09:05:58.200 INFO:teuthology.orchestra.run.smithi069.stdout:(47/156): python3-urllib3-1.26.5-5.el9.noarch.r 1.9 MB/s | 215 kB 00:00 2024-05-09T09:05:58.208 INFO:teuthology.orchestra.run.smithi103.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 328 kB/s | 16 kB 00:00 2024-05-09T09:05:58.275 INFO:teuthology.orchestra.run.smithi069.stdout:(48/156): automake-1.16.2-8.el9.noarch.rpm 6.7 MB/s | 689 kB 00:00 2024-05-09T09:05:58.275 INFO:teuthology.orchestra.run.smithi103.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 3.9 MB/s | 264 kB 00:00 2024-05-09T09:05:58.325 INFO:teuthology.orchestra.run.smithi069.stdout:(49/156): bison-3.7.4-5.el9.x86_64.rpm 7.4 MB/s | 943 kB 00:00 2024-05-09T09:05:58.367 INFO:teuthology.orchestra.run.smithi069.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 1.1 MB/s | 107 kB 00:00 2024-05-09T09:05:58.384 INFO:teuthology.orchestra.run.smithi069.stdout:(51/156): elfutils-libelf-devel-0.190-2.el9.x86 366 kB/s | 21 kB 00:00 2024-05-09T09:05:58.400 INFO:teuthology.orchestra.run.smithi103.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 6.0 MB/s | 763 kB 00:00 2024-05-09T09:05:58.426 INFO:teuthology.orchestra.run.smithi069.stdout:(52/156): flexiblas-3.0.4-8.el9.x86_64.rpm 784 kB/s | 33 kB 00:00 2024-05-09T09:05:58.459 INFO:teuthology.orchestra.run.smithi103.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 313 kB/s | 18 kB 00:00 2024-05-09T09:05:58.460 INFO:teuthology.orchestra.run.smithi069.stdout:(53/156): flex-2.6.4-9.el9.x86_64.rpm 3.3 MB/s | 313 kB 00:00 2024-05-09T09:05:58.502 INFO:teuthology.orchestra.run.smithi069.stdout:(54/156): flexiblas-openblas-openmp-3.0.4-8.el9 439 kB/s | 18 kB 00:00 2024-05-09T09:05:58.535 INFO:teuthology.orchestra.run.smithi069.stdout:(55/156): libaio-devel-0.3.111-13.el9.x86_64.rp 334 kB/s | 11 kB 00:00 2024-05-09T09:05:58.594 INFO:teuthology.orchestra.run.smithi069.stdout:(56/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 18 MB/s | 3.0 MB 00:00 2024-05-09T09:05:58.636 INFO:teuthology.orchestra.run.smithi069.stdout:(57/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.6 MB/s | 160 kB 00:00 2024-05-09T09:05:58.661 INFO:teuthology.orchestra.run.smithi069.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 677 kB/s | 45 kB 00:00 2024-05-09T09:05:58.737 INFO:teuthology.orchestra.run.smithi069.stdout:(59/156): librdkafka-1.6.1-102.el9.x86_64.rpm 6.5 MB/s | 662 kB 00:00 2024-05-09T09:05:58.778 INFO:teuthology.orchestra.run.smithi069.stdout:(60/156): libtool-2.4.6-45.el9.x86_64.rpm 4.9 MB/s | 582 kB 00:00 2024-05-09T09:05:58.804 INFO:teuthology.orchestra.run.smithi069.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 381 kB/s | 25 kB 00:00 2024-05-09T09:05:58.854 INFO:teuthology.orchestra.run.smithi069.stdout:(62/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 631 kB/s | 47 kB 00:00 2024-05-09T09:05:58.884 INFO:teuthology.orchestra.run.smithi103.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 189 kB/s | 80 kB 00:00 2024-05-09T09:05:58.888 INFO:teuthology.orchestra.run.smithi069.stdout:(63/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.4 MB/s | 292 kB 00:00 2024-05-09T09:05:58.938 INFO:teuthology.orchestra.run.smithi069.stdout:(64/156): openblas-0.3.21-2.el9.x86_64.rpm 706 kB/s | 35 kB 00:00 2024-05-09T09:05:58.964 INFO:teuthology.orchestra.run.smithi069.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 1.7 MB/s | 188 kB 00:00 2024-05-09T09:05:59.156 INFO:teuthology.orchestra.run.smithi069.stdout:(66/156): openblas-openmp-0.3.21-2.el9.x86_64.r 22 MB/s | 4.7 MB 00:00 2024-05-09T09:05:59.214 INFO:teuthology.orchestra.run.smithi069.stdout:(67/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-09T09:05:59.314 INFO:teuthology.orchestra.run.smithi069.stdout:(68/156): postgresql-contrib-13.14-1.el9.x86_64 8.9 MB/s | 911 kB 00:00 2024-05-09T09:05:59.373 INFO:teuthology.orchestra.run.smithi069.stdout:(69/156): postgresql-13.14-1.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-05-09T09:05:59.418 INFO:teuthology.orchestra.run.smithi103.stdout:(39/156): ceph-mgr-diskprediction-local-19.0.0- 4.6 MB/s | 7.4 MB 00:01 2024-05-09T09:05:59.431 INFO:teuthology.orchestra.run.smithi069.stdout:(70/156): postgresql-private-libs-13.14-1.el9.x 1.1 MB/s | 133 kB 00:00 2024-05-09T09:05:59.451 INFO:teuthology.orchestra.run.smithi103.stdout:(40/156): libgfortran-11.4.1-3.el9.x86_64.rpm 1.4 MB/s | 803 kB 00:00 2024-05-09T09:05:59.540 INFO:teuthology.orchestra.run.smithi069.stdout:(71/156): protobuf-3.14.0-13.el9.x86_64.rpm 9.3 MB/s | 1.0 MB 00:00 2024-05-09T09:05:59.560 INFO:teuthology.orchestra.run.smithi103.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.3 MB/s | 253 kB 00:00 2024-05-09T09:05:59.649 INFO:teuthology.orchestra.run.smithi069.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 21 MB/s | 5.8 MB 00:00 2024-05-09T09:05:59.674 INFO:teuthology.orchestra.run.smithi069.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 654 kB/s | 87 kB 00:00 2024-05-09T09:05:59.749 INFO:teuthology.orchestra.run.smithi069.stdout:(74/156): python3-devel-3.9.18-3.el9.x86_64.rpm 3.2 MB/s | 244 kB 00:00 2024-05-09T09:05:59.824 INFO:teuthology.orchestra.run.smithi069.stdout:(75/156): python3-jinja2-2.11.3-5.el9.noarch.rp 3.2 MB/s | 248 kB 00:00 2024-05-09T09:05:59.900 INFO:teuthology.orchestra.run.smithi069.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 6.0 MB 00:00 2024-05-09T09:05:59.919 INFO:teuthology.orchestra.run.smithi103.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 3.4 MB/s | 1.2 MB 00:00 2024-05-09T09:05:59.942 INFO:teuthology.orchestra.run.smithi069.stdout:(77/156): python3-jsonpatch-1.21-16.el9.noarch. 219 kB/s | 26 kB 00:00 2024-05-09T09:05:59.961 INFO:teuthology.orchestra.run.smithi103.stdout:(43/156): libquadmath-11.4.1-3.el9.x86_64.rpm 360 kB/s | 195 kB 00:00 2024-05-09T09:05:59.967 INFO:teuthology.orchestra.run.smithi069.stdout:(78/156): python3-jsonpointer-2.0-4.el9.noarch. 286 kB/s | 19 kB 00:00 2024-05-09T09:06:00.026 INFO:teuthology.orchestra.run.smithi069.stdout:(79/156): python3-markupsafe-1.1.1-12.el9.x86_6 593 kB/s | 35 kB 00:00 2024-05-09T09:06:00.027 INFO:teuthology.orchestra.run.smithi103.stdout:(44/156): python3-ply-3.11-14.el9.noarch.rpm 981 kB/s | 106 kB 00:00 2024-05-09T09:06:00.051 INFO:teuthology.orchestra.run.smithi069.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-05-09T09:06:00.064 INFO:teuthology.orchestra.run.smithi163.stdout:(8/156): ceph-radosgw-19.0.0-2515.g96805dcf.el9 4.4 MB/s | 10 MB 00:02 2024-05-09T09:06:00.106 INFO:teuthology.orchestra.run.smithi163.stdout:(9/156): ceph-selinux-19.0.0-2515.g96805dcf.el9 606 kB/s | 25 kB 00:00 2024-05-09T09:06:00.126 INFO:teuthology.orchestra.run.smithi069.stdout:(81/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.3 MB/s | 250 kB 00:00 2024-05-09T09:06:00.202 INFO:teuthology.orchestra.run.smithi069.stdout:(82/156): python3-oauthlib-3.1.1-5.el9.noarch.r 2.9 MB/s | 222 kB 00:00 2024-05-09T09:06:00.277 INFO:teuthology.orchestra.run.smithi069.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 5.1 MB 00:00 2024-05-09T09:06:00.327 INFO:teuthology.orchestra.run.smithi069.stdout:(84/156): python3-packaging-20.9-5.el9.noarch.r 617 kB/s | 77 kB 00:00 2024-05-09T09:06:00.354 INFO:teuthology.orchestra.run.smithi069.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 562 kB/s | 42 kB 00:00 2024-05-09T09:06:00.420 INFO:teuthology.orchestra.run.smithi069.stdout:(86/156): python3-protobuf-3.14.0-13.el9.noarch 2.9 MB/s | 269 kB 00:00 2024-05-09T09:06:00.445 INFO:teuthology.orchestra.run.smithi069.stdout:(87/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 159 kB 00:00 2024-05-09T09:06:00.461 INFO:teuthology.orchestra.run.smithi103.stdout:(45/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 8.4 MB/s | 48 MB 00:05 2024-05-09T09:06:00.487 INFO:teuthology.orchestra.run.smithi103.stdout:(46/156): python3-requests-2.25.1-8.el9.noarch. 272 kB/s | 125 kB 00:00 2024-05-09T09:06:00.495 INFO:teuthology.orchestra.run.smithi069.stdout:(88/156): python3-pytz-2021.1-5.el9.noarch.rpm 1.0 MB/s | 51 kB 00:00 2024-05-09T09:06:00.512 INFO:teuthology.orchestra.run.smithi103.stdout:(47/156): python3-pycparser-2.20-6.el9.noarch.r 245 kB/s | 135 kB 00:00 2024-05-09T09:06:00.521 INFO:teuthology.orchestra.run.smithi069.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 279 kB 00:00 2024-05-09T09:06:00.562 INFO:teuthology.orchestra.run.smithi069.stdout:(90/156): python3-requests-oauthlib-1.3.0-12.el 802 kB/s | 54 kB 00:00 2024-05-09T09:06:00.613 INFO:teuthology.orchestra.run.smithi069.stdout:(91/156): python3-toml-0.10.2-6.el9.noarch.rpm 831 kB/s | 42 kB 00:00 2024-05-09T09:06:00.663 INFO:teuthology.orchestra.run.smithi069.stdout:(92/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 885 kB/s | 44 kB 00:00 2024-05-09T09:06:00.738 INFO:teuthology.orchestra.run.smithi069.stdout:(93/156): qatlib-24.02.0-1.el9.x86_64.rpm 2.9 MB/s | 221 kB 00:00 2024-05-09T09:06:00.780 INFO:teuthology.orchestra.run.smithi069.stdout:(94/156): qatlib-service-24.02.0-1.el9.x86_64.r 862 kB/s | 36 kB 00:00 2024-05-09T09:06:00.839 INFO:teuthology.orchestra.run.smithi069.stdout:(95/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 811 kB/s | 47 kB 00:00 2024-05-09T09:06:00.906 INFO:teuthology.orchestra.run.smithi163.stdout:(10/156): ceph-osd-19.0.0-2515.g96805dcf.el9.x8 3.5 MB/s | 17 MB 00:04 2024-05-09T09:06:00.962 INFO:teuthology.orchestra.run.smithi103.stdout:(48/156): python3-urllib3-1.26.5-5.el9.noarch.r 429 kB/s | 215 kB 00:00 2024-05-09T09:06:00.973 INFO:teuthology.orchestra.run.smithi163.stdout:(11/156): cephfs-mirror-19.0.0-2515.g96805dcf.e 3.3 MB/s | 222 kB 00:00 2024-05-09T09:06:01.018 INFO:teuthology.orchestra.run.smithi163.stdout:(12/156): cephfs-shell-19.0.0-2515.g96805dcf.el 652 kB/s | 27 kB 00:00 2024-05-09T09:06:01.022 INFO:teuthology.orchestra.run.smithi069.stdout:(96/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 32 MB/s | 16 MB 00:00 2024-05-09T09:06:01.057 INFO:teuthology.orchestra.run.smithi163.stdout:(13/156): libcephfs-devel-19.0.0-2515.g96805dcf 760 kB/s | 32 kB 00:00 2024-05-09T09:06:01.089 INFO:teuthology.orchestra.run.smithi069.stdout:(97/156): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 305 kB 00:00 2024-05-09T09:06:01.115 INFO:teuthology.orchestra.run.smithi069.stdout:(98/156): uuid-1.6.2-55.el9.x86_64.rpm 631 kB/s | 58 kB 00:00 2024-05-09T09:06:01.165 INFO:teuthology.orchestra.run.smithi069.stdout:(99/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rpm 1.2 MB/s | 94 kB 00:00 2024-05-09T09:06:01.190 INFO:teuthology.orchestra.run.smithi069.stdout:(100/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 845 kB/s | 64 kB 00:00 2024-05-09T09:06:01.191 INFO:teuthology.orchestra.run.smithi163.stdout:(14/156): libcephfs2-19.0.0-2515.g96805dcf.el9. 5.5 MB/s | 746 kB 00:00 2024-05-09T09:06:01.233 INFO:teuthology.orchestra.run.smithi163.stdout:(15/156): libcephsqlite-19.0.0-2515.g96805dcf.e 3.8 MB/s | 163 kB 00:00 2024-05-09T09:06:01.241 INFO:teuthology.orchestra.run.smithi069.stdout:(101/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 594 kB/s | 45 kB 00:00 2024-05-09T09:06:01.283 INFO:teuthology.orchestra.run.smithi163.stdout:(16/156): librados-devel-19.0.0-2515.g96805dcf. 2.5 MB/s | 127 kB 00:00 2024-05-09T09:06:01.358 INFO:teuthology.orchestra.run.smithi069.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-05-09T09:06:01.383 INFO:teuthology.orchestra.run.smithi163.stdout:(17/156): libradosstriper1-19.0.0-2515.g96805dc 4.9 MB/s | 500 kB 00:00 2024-05-09T09:06:01.462 INFO:teuthology.orchestra.run.smithi103.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 706 kB/s | 689 kB 00:00 2024-05-09T09:06:01.504 INFO:teuthology.orchestra.run.smithi103.stdout:(50/156): bison-3.7.4-5.el9.x86_64.rpm 950 kB/s | 943 kB 00:00 2024-05-09T09:06:01.525 INFO:teuthology.orchestra.run.smithi069.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 3.0 MB/s | 863 kB 00:00 2024-05-09T09:06:01.666 INFO:teuthology.orchestra.run.smithi069.stdout:(104/156): abseil-cpp-20211102.0-3.el9.x86_64.r 1.7 MB/s | 548 kB 00:00 2024-05-09T09:06:01.696 INFO:teuthology.orchestra.run.smithi103.stdout:(51/156): boost-program-options-1.75.0-8.el9.x8 146 kB/s | 107 kB 00:00 2024-05-09T09:06:01.733 INFO:teuthology.orchestra.run.smithi069.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 291 kB/s | 19 kB 00:00 2024-05-09T09:06:01.825 INFO:teuthology.orchestra.run.smithi069.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-05-09T09:06:01.892 INFO:teuthology.orchestra.run.smithi069.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 388 kB/s | 26 kB 00:00 2024-05-09T09:06:01.976 INFO:teuthology.orchestra.run.smithi069.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 18 MB/s | 4.4 MB 00:00 2024-05-09T09:06:02.001 INFO:teuthology.orchestra.run.smithi069.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 447 kB/s | 49 kB 00:00 2024-05-09T09:06:02.043 INFO:teuthology.orchestra.run.smithi069.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 1.0 MB/s | 67 kB 00:00 2024-05-09T09:06:02.063 INFO:teuthology.orchestra.run.smithi103.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 36 kB/s | 21 kB 00:00 2024-05-09T09:06:02.069 INFO:teuthology.orchestra.run.smithi069.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 2.2 MB/s | 151 kB 00:00 2024-05-09T09:06:02.084 INFO:teuthology.orchestra.run.smithi163.stdout:(18/156): librgw2-19.0.0-2515.g96805dcf.el9.x86 7.6 MB/s | 5.3 MB 00:00 2024-05-09T09:06:02.119 INFO:teuthology.orchestra.run.smithi069.stdout:(112/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 11 MB/s | 839 kB 00:00 2024-05-09T09:06:02.134 INFO:teuthology.orchestra.run.smithi163.stdout:(19/156): python3-ceph-argparse-19.0.0-2515.g96 907 kB/s | 45 kB 00:00 2024-05-09T09:06:02.184 INFO:teuthology.orchestra.run.smithi163.stdout:(20/156): python3-ceph-common-19.0.0-2515.g9680 2.6 MB/s | 135 kB 00:00 2024-05-09T09:06:02.186 INFO:teuthology.orchestra.run.smithi069.stdout:(113/156): python3-asyncssh-2.13.2-5.el9.noarch 4.6 MB/s | 548 kB 00:00 2024-05-09T09:06:02.211 INFO:teuthology.orchestra.run.smithi069.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 473 kB/s | 43 kB 00:00 2024-05-09T09:06:02.243 INFO:teuthology.orchestra.run.smithi163.stdout:(21/156): python3-cephfs-19.0.0-2515.g96805dcf. 2.7 MB/s | 162 kB 00:00 2024-05-09T09:06:02.253 INFO:teuthology.orchestra.run.smithi069.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 481 kB/s | 32 kB 00:00 2024-05-09T09:06:02.278 INFO:teuthology.orchestra.run.smithi069.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 223 kB/s | 15 kB 00:00 2024-05-09T09:06:02.318 INFO:teuthology.orchestra.run.smithi163.stdout:(22/156): python3-rados-19.0.0-2515.g96805dcf.e 4.2 MB/s | 321 kB 00:00 2024-05-09T09:06:02.320 INFO:teuthology.orchestra.run.smithi069.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.5 MB/s | 172 kB 00:00 2024-05-09T09:06:02.347 INFO:teuthology.orchestra.run.smithi103.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-09T09:06:02.354 INFO:teuthology.orchestra.run.smithi069.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 4.6 MB/s | 358 kB 00:00 2024-05-09T09:06:02.372 INFO:teuthology.orchestra.run.smithi103.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 362 kB/s | 313 kB 00:00 2024-05-09T09:06:02.387 INFO:teuthology.orchestra.run.smithi069.stdout:(119/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 4.4 MB/s | 303 kB 00:00 2024-05-09T09:06:02.393 INFO:teuthology.orchestra.run.smithi163.stdout:(23/156): python3-rbd-19.0.0-2515.g96805dcf.el9 3.9 MB/s | 298 kB 00:00 2024-05-09T09:06:02.421 INFO:teuthology.orchestra.run.smithi069.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 528 kB/s | 35 kB 00:00 2024-05-09T09:06:02.444 INFO:teuthology.orchestra.run.smithi163.stdout:(24/156): python3-rgw-19.0.0-2515.g96805dcf.el9 1.9 MB/s | 99 kB 00:00 2024-05-09T09:06:02.455 INFO:teuthology.orchestra.run.smithi069.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-05-09T09:06:02.494 INFO:teuthology.orchestra.run.smithi163.stdout:(25/156): rbd-fuse-19.0.0-2515.g96805dcf.el9.x8 1.7 MB/s | 85 kB 00:00 2024-05-09T09:06:02.522 INFO:teuthology.orchestra.run.smithi069.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-05-09T09:06:02.580 INFO:teuthology.orchestra.run.smithi069.stdout:(123/156): python3-grpcio-1.46.7-10.el9.x86_64. 13 MB/s | 2.0 MB 00:00 2024-05-09T09:06:02.605 INFO:teuthology.orchestra.run.smithi069.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 128 kB/s | 11 kB 00:00 2024-05-09T09:06:02.647 INFO:teuthology.orchestra.run.smithi069.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 263 kB/s | 18 kB 00:00 2024-05-09T09:06:02.673 INFO:teuthology.orchestra.run.smithi069.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 346 kB/s | 23 kB 00:00 2024-05-09T09:06:02.723 INFO:teuthology.orchestra.run.smithi069.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 258 kB/s | 19 kB 00:00 2024-05-09T09:06:02.748 INFO:teuthology.orchestra.run.smithi069.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 261 kB/s | 20 kB 00:00 2024-05-09T09:06:02.832 INFO:teuthology.orchestra.run.smithi069.stdout:(129/156): python3-kubernetes-26.1.0-2.el9.noar 9.4 MB/s | 1.0 MB 00:00 2024-05-09T09:06:02.849 INFO:teuthology.orchestra.run.smithi069.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 462 kB/s | 46 kB 00:00 2024-05-09T09:06:02.864 INFO:teuthology.orchestra.run.smithi103.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 35 kB/s | 18 kB 00:00 2024-05-09T09:06:02.889 INFO:teuthology.orchestra.run.smithi103.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 22 kB/s | 11 kB 00:00 2024-05-09T09:06:02.911 INFO:teuthology.orchestra.run.smithi163.stdout:(26/156): rbd-mirror-19.0.0-2515.g96805dcf.el9. 7.4 MB/s | 3.1 MB 00:00 2024-05-09T09:06:02.924 INFO:teuthology.orchestra.run.smithi069.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 857 kB/s | 79 kB 00:00 2024-05-09T09:06:02.941 INFO:teuthology.orchestra.run.smithi069.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 628 kB/s | 58 kB 00:00 2024-05-09T09:06:02.969 INFO:teuthology.orchestra.run.smithi163.stdout:(27/156): rbd-nbd-19.0.0-2515.g96805dcf.el9.x86 2.9 MB/s | 171 kB 00:00 2024-05-09T09:06:03.008 INFO:teuthology.orchestra.run.smithi069.stdout:(133/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-05-09T09:06:03.011 INFO:teuthology.orchestra.run.smithi163.stdout:(28/156): ceph-grafana-dashboards-19.0.0-2515.g 762 kB/s | 32 kB 00:00 2024-05-09T09:06:03.025 INFO:teuthology.orchestra.run.smithi069.stdout:(134/156): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-05-09T09:06:03.062 INFO:teuthology.orchestra.run.smithi163.stdout:(29/156): ceph-mgr-cephadm-19.0.0-2515.g96805dc 2.8 MB/s | 142 kB 00:00 2024-05-09T09:06:03.092 INFO:teuthology.orchestra.run.smithi069.stdout:(135/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.1 MB/s | 90 kB 00:00 2024-05-09T09:06:03.117 INFO:teuthology.orchestra.run.smithi069.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 306 kB/s | 28 kB 00:00 2024-05-09T09:06:03.176 INFO:teuthology.orchestra.run.smithi069.stdout:(137/156): python3-repoze-lru-0.7-16.el9.noarch 367 kB/s | 31 kB 00:00 2024-05-09T09:06:03.206 INFO:teuthology.orchestra.run.smithi103.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-05-09T09:06:03.210 INFO:teuthology.orchestra.run.smithi069.stdout:(138/156): python3-routes-2.5.1-5.el9.noarch.rp 2.0 MB/s | 188 kB 00:00 2024-05-09T09:06:03.268 INFO:teuthology.orchestra.run.smithi069.stdout:(139/156): python3-rsa-4.9-2.el9.noarch.rpm 643 kB/s | 59 kB 00:00 2024-05-09T09:06:03.302 INFO:teuthology.orchestra.run.smithi069.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 390 kB/s | 36 kB 00:00 2024-05-09T09:06:03.352 INFO:teuthology.orchestra.run.smithi069.stdout:(141/156): python3-typing-extensions-4.4.0-2.el 610 kB/s | 51 kB 00:00 2024-05-09T09:06:03.394 INFO:teuthology.orchestra.run.smithi069.stdout:(142/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-09T09:06:03.404 INFO:teuthology.orchestra.run.smithi163.stdout:(30/156): ceph-common-19.0.0-2515.g96805dcf.el9 1.8 MB/s | 21 MB 00:11 2024-05-09T09:06:03.461 INFO:teuthology.orchestra.run.smithi069.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 827 kB/s | 90 kB 00:00 2024-05-09T09:06:03.486 INFO:teuthology.orchestra.run.smithi069.stdout:(144/156): python3-werkzeug-2.0.3-3.el9.1.noarc 4.5 MB/s | 427 kB 00:00 2024-05-09T09:06:03.515 INFO:teuthology.orchestra.run.smithi103.stdout:(58/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 72 kB/s | 45 kB 00:00 2024-05-09T09:06:03.545 INFO:teuthology.orchestra.run.smithi069.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 238 kB/s | 20 kB 00:00 2024-05-09T09:06:03.570 INFO:teuthology.orchestra.run.smithi069.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 2.2 MB/s | 192 kB 00:00 2024-05-09T09:06:03.637 INFO:teuthology.orchestra.run.smithi069.stdout:(147/156): thrift-0.15.0-2.el9.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-05-09T09:06:03.648 INFO:teuthology.orchestra.run.smithi103.stdout:(59/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 205 kB/s | 160 kB 00:00 2024-05-09T09:06:03.713 INFO:teuthology.orchestra.run.smithi163.stdout:(31/156): ceph-mgr-dashboard-19.0.0-2515.g96805 5.4 MB/s | 3.5 MB 00:00 2024-05-09T09:06:03.813 INFO:teuthology.orchestra.run.smithi163.stdout:(32/156): ceph-mgr-modules-core-19.0.0-2515.g96 2.4 MB/s | 249 kB 00:00 2024-05-09T09:06:03.871 INFO:teuthology.orchestra.run.smithi163.stdout:(33/156): ceph-mgr-rook-19.0.0-2515.g96805dcf.e 846 kB/s | 49 kB 00:00 2024-05-09T09:06:03.930 INFO:teuthology.orchestra.run.smithi163.stdout:(34/156): ceph-prometheus-alerts-19.0.0-2515.g9 282 kB/s | 16 kB 00:00 2024-05-09T09:06:04.055 INFO:teuthology.orchestra.run.smithi163.stdout:(35/156): ceph-volume-19.0.0-2515.g96805dcf.el9 2.1 MB/s | 264 kB 00:00 2024-05-09T09:06:04.188 INFO:teuthology.orchestra.run.smithi069.stdout:(148/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 5.7 MB/s | 3.1 MB 00:00 2024-05-09T09:06:04.215 INFO:teuthology.orchestra.run.smithi103.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 657 kB/s | 662 kB 00:01 2024-05-09T09:06:04.238 INFO:teuthology.orchestra.run.smithi069.stdout:(149/156): librados2-19.0.0-2515.g96805dcf.el9. 5.1 MB/s | 3.4 MB 00:00 2024-05-09T09:06:04.239 INFO:teuthology.orchestra.run.smithi163.stdout:(36/156): cephadm-19.0.0-2515.g96805dcf.el9.noa 4.1 MB/s | 763 kB 00:00 2024-05-09T09:06:04.283 INFO:teuthology.orchestra.run.smithi103.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 40 kB/s | 25 kB 00:00 2024-05-09T09:06:04.297 INFO:teuthology.orchestra.run.smithi163.stdout:(37/156): cephfs-top-19.0.0-2515.g96805dcf.el9. 313 kB/s | 18 kB 00:00 2024-05-09T09:06:04.364 INFO:teuthology.orchestra.run.smithi163.stdout:(38/156): fuse-2.9.9-15.el9.x86_64.rpm 1.2 MB/s | 80 kB 00:00 2024-05-09T09:06:04.465 INFO:teuthology.orchestra.run.smithi163.stdout:(39/156): libgfortran-11.4.1-3.el9.x86_64.rpm 7.9 MB/s | 803 kB 00:00 2024-05-09T09:06:04.533 INFO:teuthology.orchestra.run.smithi163.stdout:(40/156): libquadmath-11.4.1-3.el9.x86_64.rpm 2.9 MB/s | 195 kB 00:00 2024-05-09T09:06:04.558 INFO:teuthology.orchestra.run.smithi103.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 559 kB/s | 582 kB 00:01 2024-05-09T09:06:04.607 INFO:teuthology.orchestra.run.smithi163.stdout:(41/156): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.3 MB/s | 253 kB 00:00 2024-05-09T09:06:04.724 INFO:teuthology.orchestra.run.smithi163.stdout:(42/156): python3-cryptography-36.0.1-4.el9.x86 11 MB/s | 1.2 MB 00:00 2024-05-09T09:06:04.791 INFO:teuthology.orchestra.run.smithi163.stdout:(43/156): python3-ply-3.11-14.el9.noarch.rpm 1.6 MB/s | 106 kB 00:00 2024-05-09T09:06:04.858 INFO:teuthology.orchestra.run.smithi163.stdout:(44/156): python3-pycparser-2.20-6.el9.noarch.r 2.0 MB/s | 135 kB 00:00 2024-05-09T09:06:04.866 INFO:teuthology.orchestra.run.smithi103.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-05-09T09:06:04.925 INFO:teuthology.orchestra.run.smithi163.stdout:(45/156): python3-requests-2.25.1-8.el9.noarch. 1.8 MB/s | 125 kB 00:00 2024-05-09T09:06:05.000 INFO:teuthology.orchestra.run.smithi163.stdout:(46/156): python3-urllib3-1.26.5-5.el9.noarch.r 2.8 MB/s | 215 kB 00:00 2024-05-09T09:06:05.098 INFO:teuthology.orchestra.run.smithi069.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 2.1 MB/s | 1.8 MB 00:00 2024-05-09T09:06:05.116 INFO:teuthology.orchestra.run.smithi103.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 351 kB/s | 292 kB 00:00 2024-05-09T09:06:05.147 INFO:teuthology.orchestra.run.smithi069.stdout:(151/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.4 MB/s | 2.3 MB 00:00 2024-05-09T09:06:05.205 INFO:teuthology.orchestra.run.smithi069.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 385 kB/s | 42 kB 00:00 2024-05-09T09:06:05.256 INFO:teuthology.orchestra.run.smithi069.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.2 MB/s | 248 kB 00:00 2024-05-09T09:06:05.350 INFO:teuthology.orchestra.run.smithi103.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 238 kB/s | 188 kB 00:00 2024-05-09T09:06:05.534 INFO:teuthology.orchestra.run.smithi103.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 53 kB/s | 35 kB 00:00 2024-05-09T09:06:05.534 INFO:teuthology.orchestra.run.smithi069.stdout:(154/156): openssl-3.2.1-1.el9.x86_64.rpm 4.1 MB/s | 1.3 MB 00:00 2024-05-09T09:06:05.726 INFO:teuthology.orchestra.run.smithi163.stdout:(47/156): ceph-test-19.0.0-2515.g96805dcf.el9.x 8.6 MB/s | 48 MB 00:05 2024-05-09T09:06:05.809 INFO:teuthology.orchestra.run.smithi163.stdout:(48/156): ceph-mgr-diskprediction-local-19.0.0- 3.1 MB/s | 7.4 MB 00:02 2024-05-09T09:06:06.318 INFO:teuthology.orchestra.run.smithi103.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 4.0 MB/s | 4.7 MB 00:01 2024-05-09T09:06:06.348 INFO:teuthology.orchestra.run.smithi069.stdout:(155/156): ceph-test-19.0.0-2515.g96805dcf.el9. 3.5 MB/s | 48 MB 00:13 2024-05-09T09:06:06.351 INFO:teuthology.orchestra.run.smithi163.stdout:(49/156): automake-1.16.2-8.el9.noarch.rpm 510 kB/s | 689 kB 00:01 2024-05-09T09:06:06.473 INFO:teuthology.orchestra.run.smithi069.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.0 MB/s | 2.4 MB 00:01 2024-05-09T09:06:06.477 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-05-09T09:06:06.477 INFO:teuthology.orchestra.run.smithi069.stdout:Total 13 MB/s | 225 MB 00:17 2024-05-09T09:06:06.551 INFO:teuthology.orchestra.run.smithi163.stdout:(50/156): boost-program-options-1.75.0-8.el9.x8 144 kB/s | 107 kB 00:00 2024-05-09T09:06:06.643 INFO:teuthology.orchestra.run.smithi103.stdout:(68/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.4 MB/s | 4.4 MB 00:01 2024-05-09T09:06:06.685 INFO:teuthology.orchestra.run.smithi103.stdout:(69/156): postgresql-13.14-1.el9.x86_64.rpm 1.4 MB/s | 1.6 MB 00:01 2024-05-09T09:06:06.685 INFO:teuthology.orchestra.run.smithi163.stdout:(51/156): bison-3.7.4-5.el9.x86_64.rpm 985 kB/s | 943 kB 00:00 2024-05-09T09:06:06.969 INFO:teuthology.orchestra.run.smithi163.stdout:(52/156): elfutils-libelf-devel-0.190-2.el9.x86 35 kB/s | 21 kB 00:00 2024-05-09T09:06:07.286 INFO:teuthology.orchestra.run.smithi163.stdout:(53/156): flexiblas-3.0.4-8.el9.x86_64.rpm 55 kB/s | 33 kB 00:00 2024-05-09T09:06:07.327 INFO:teuthology.orchestra.run.smithi103.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 904 kB/s | 911 kB 00:01 2024-05-09T09:06:07.385 INFO:teuthology.orchestra.run.smithi103.stdout:(71/156): postgresql-private-libs-13.14-1.el9.x 180 kB/s | 133 kB 00:00 2024-05-09T09:06:07.419 INFO:teuthology.orchestra.run.smithi163.stdout:(54/156): flex-2.6.4-9.el9.x86_64.rpm 362 kB/s | 313 kB 00:00 2024-05-09T09:06:07.697 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction check 2024-05-09T09:06:07.828 INFO:teuthology.orchestra.run.smithi163.stdout:(55/156): flexiblas-openblas-openmp-3.0.4-8.el9 34 kB/s | 18 kB 00:00 2024-05-09T09:06:07.850 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction check succeeded. 2024-05-09T09:06:07.850 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction test 2024-05-09T09:06:07.902 INFO:teuthology.orchestra.run.smithi103.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 4.8 MB/s | 5.8 MB 00:01 2024-05-09T09:06:07.944 INFO:teuthology.orchestra.run.smithi163.stdout:(56/156): libaio-devel-0.3.111-13.el9.x86_64.rp 21 kB/s | 11 kB 00:00 2024-05-09T09:06:08.103 INFO:teuthology.orchestra.run.smithi103.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 122 kB/s | 87 kB 00:00 2024-05-09T09:06:08.136 INFO:teuthology.orchestra.run.smithi163.stdout:(57/156): flexiblas-netlib-3.0.4-8.el9.x86_64.r 2.6 MB/s | 3.0 MB 00:01 2024-05-09T09:06:08.403 INFO:teuthology.orchestra.run.smithi103.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 961 kB/s | 1.0 MB 00:01 2024-05-09T09:06:08.637 INFO:teuthology.orchestra.run.smithi163.stdout:(58/156): libpmemobj-1.12.1-1.el9.x86_64.rpm 198 kB/s | 160 kB 00:00 2024-05-09T09:06:08.662 INFO:teuthology.orchestra.run.smithi163.stdout:(59/156): librabbitmq-0.11.0-7.el9.x86_64.rpm 63 kB/s | 45 kB 00:00 2024-05-09T09:06:08.970 INFO:teuthology.orchestra.run.smithi103.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 282 kB/s | 244 kB 00:00 2024-05-09T09:06:09.162 INFO:teuthology.orchestra.run.smithi163.stdout:(60/156): librdkafka-1.6.1-102.el9.x86_64.rpm 646 kB/s | 662 kB 00:01 2024-05-09T09:06:09.203 INFO:teuthology.orchestra.run.smithi103.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.6 MB/s | 6.0 MB 00:01 2024-05-09T09:06:09.254 INFO:teuthology.orchestra.run.smithi163.stdout:(61/156): libuuid-devel-2.37.4-18.el9.x86_64.rp 43 kB/s | 25 kB 00:00 2024-05-09T09:06:09.287 INFO:teuthology.orchestra.run.smithi103.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 281 kB/s | 248 kB 00:00 2024-05-09T09:06:09.562 INFO:teuthology.orchestra.run.smithi103.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 43 kB/s | 26 kB 00:00 2024-05-09T09:06:09.596 INFO:teuthology.orchestra.run.smithi163.stdout:(62/156): libtool-2.4.6-45.el9.x86_64.rpm 608 kB/s | 582 kB 00:00 2024-05-09T09:06:09.779 INFO:teuthology.orchestra.run.smithi103.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 35 kB/s | 19 kB 00:00 2024-05-09T09:06:09.822 INFO:teuthology.orchestra.run.smithi163.stdout:(63/156): libzstd-devel-1.5.1-2.el9.x86_64.rpm 72 kB/s | 47 kB 00:00 2024-05-09T09:06:10.051 INFO:teuthology.orchestra.run.smithi069.stdout:Transaction test succeeded. 2024-05-09T09:06:10.052 INFO:teuthology.orchestra.run.smithi069.stdout:Running transaction 2024-05-09T09:06:10.063 INFO:teuthology.orchestra.run.smithi103.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 222 kB/s | 172 kB 00:00 2024-05-09T09:06:10.130 INFO:teuthology.orchestra.run.smithi163.stdout:(64/156): lttng-ust-2.12.0-6.el9.x86_64.rpm 334 kB/s | 292 kB 00:00 2024-05-09T09:06:10.171 INFO:teuthology.orchestra.run.smithi103.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 57 kB/s | 35 kB 00:00 2024-05-09T09:06:10.397 INFO:teuthology.orchestra.run.smithi163.stdout:(65/156): lua-5.4.4-4.el9.x86_64.rpm 235 kB/s | 188 kB 00:00 2024-05-09T09:06:10.439 INFO:teuthology.orchestra.run.smithi163.stdout:(66/156): openblas-0.3.21-2.el9.x86_64.rpm 60 kB/s | 35 kB 00:00 2024-05-09T09:06:10.897 INFO:teuthology.orchestra.run.smithi103.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 301 kB/s | 250 kB 00:00 2024-05-09T09:06:10.972 INFO:teuthology.orchestra.run.smithi103.stdout:(83/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.3 MB/s | 5.1 MB 00:01 2024-05-09T09:06:10.998 INFO:teuthology.orchestra.run.smithi103.stdout:(84/156): python3-oauthlib-3.1.1-5.el9.noarch.r 269 kB/s | 222 kB 00:00 2024-05-09T09:06:11.298 INFO:teuthology.orchestra.run.smithi163.stdout:(67/156): openblas-openmp-0.3.21-2.el9.x86_64.r 4.1 MB/s | 4.7 MB 00:01 2024-05-09T09:06:11.489 INFO:teuthology.orchestra.run.smithi163.stdout:(68/156): postgresql-13.14-1.el9.x86_64.rpm 1.5 MB/s | 1.6 MB 00:01 2024-05-09T09:06:11.573 INFO:teuthology.orchestra.run.smithi103.stdout:(85/156): python3-packaging-20.9-5.el9.noarch.r 114 kB/s | 77 kB 00:00 2024-05-09T09:06:11.598 INFO:teuthology.orchestra.run.smithi103.stdout:(86/156): python3-prettytable-0.7.2-27.el9.noar 68 kB/s | 42 kB 00:00 2024-05-09T09:06:11.615 INFO:teuthology.orchestra.run.smithi163.stdout:(69/156): openssl-devel-3.2.1-1.el9.x86_64.rpm 3.6 MB/s | 4.4 MB 00:01 2024-05-09T09:06:11.823 INFO:teuthology.orchestra.run.smithi103.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 326 kB/s | 269 kB 00:00 2024-05-09T09:06:12.273 INFO:teuthology.orchestra.run.smithi163.stdout:(70/156): postgresql-contrib-13.14-1.el9.x86_64 934 kB/s | 911 kB 00:00 2024-05-09T09:06:12.298 INFO:teuthology.orchestra.run.smithi163.stdout:(71/156): postgresql-private-libs-13.14-1.el9.x 165 kB/s | 133 kB 00:00 2024-05-09T09:06:12.382 INFO:teuthology.orchestra.run.smithi103.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 197 kB/s | 159 kB 00:00 2024-05-09T09:06:12.451 INFO:teuthology.orchestra.run.smithi103.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 328 kB/s | 279 kB 00:00 2024-05-09T09:06:12.541 INFO:teuthology.orchestra.run.smithi103.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 77 kB/s | 51 kB 00:00 2024-05-09T09:06:12.857 INFO:teuthology.orchestra.run.smithi163.stdout:(72/156): postgresql-server-13.14-1.el9.x86_64. 4.7 MB/s | 5.8 MB 00:01 2024-05-09T09:06:13.024 INFO:teuthology.orchestra.run.smithi163.stdout:(73/156): python3-attrs-20.3.0-7.el9.noarch.rpm 121 kB/s | 87 kB 00:00 2024-05-09T09:06:13.033 INFO:teuthology.orchestra.run.smithi103.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 82 kB/s | 54 kB 00:00 2024-05-09T09:06:13.167 INFO:teuthology.orchestra.run.smithi103.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 67 kB/s | 42 kB 00:00 2024-05-09T09:06:13.266 INFO:teuthology.orchestra.run.smithi163.stdout:(74/156): protobuf-3.14.0-13.el9.x86_64.rpm 1.0 MB/s | 1.0 MB 00:00 2024-05-09T09:06:13.650 INFO:teuthology.orchestra.run.smithi103.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 72 kB/s | 44 kB 00:00 2024-05-09T09:06:13.849 INFO:teuthology.orchestra.run.smithi163.stdout:(75/156): python3-devel-3.9.18-3.el9.x86_64.rpm 296 kB/s | 244 kB 00:00 2024-05-09T09:06:13.975 INFO:teuthology.orchestra.run.smithi103.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 11 MB/s | 16 MB 00:01 2024-05-09T09:06:14.026 INFO:teuthology.orchestra.run.smithi103.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 257 kB/s | 221 kB 00:00 2024-05-09T09:06:14.116 INFO:teuthology.orchestra.run.smithi163.stdout:(76/156): python3-babel-2.9.1-2.el9.noarch.rpm 4.7 MB/s | 6.0 MB 00:01 2024-05-09T09:06:14.167 INFO:teuthology.orchestra.run.smithi163.stdout:(77/156): python3-jinja2-2.11.3-5.el9.noarch.rp 275 kB/s | 248 kB 00:00 2024-05-09T09:06:14.392 INFO:teuthology.orchestra.run.smithi103.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 48 kB/s | 36 kB 00:00 2024-05-09T09:06:14.475 INFO:teuthology.orchestra.run.smithi163.stdout:(78/156): python3-jsonpatch-1.21-16.el9.noarch. 41 kB/s | 26 kB 00:00 2024-05-09T09:06:14.676 INFO:teuthology.orchestra.run.smithi103.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 70 kB/s | 47 kB 00:00 2024-05-09T09:06:14.759 INFO:teuthology.orchestra.run.smithi163.stdout:(79/156): python3-jsonpointer-2.0-4.el9.noarch. 31 kB/s | 19 kB 00:00 2024-05-09T09:06:14.860 INFO:teuthology.orchestra.run.smithi103.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 366 kB/s | 305 kB 00:00 2024-05-09T09:06:15.009 INFO:teuthology.orchestra.run.smithi163.stdout:(80/156): python3-mako-1.1.4-6.el9.noarch.rpm 204 kB/s | 172 kB 00:00 2024-05-09T09:06:15.051 INFO:teuthology.orchestra.run.smithi103.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-05-09T09:06:15.151 INFO:teuthology.orchestra.run.smithi163.stdout:(81/156): python3-markupsafe-1.1.1-12.el9.x86_6 51 kB/s | 35 kB 00:00 2024-05-09T09:06:15.368 INFO:teuthology.orchestra.run.smithi103.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 135 kB/s | 94 kB 00:00 2024-05-09T09:06:15.485 INFO:teuthology.orchestra.run.smithi103.stdout:(101/156): lua-devel-5.4.4-4.el9.x86_64.rpm 191 kB/s | 22 kB 00:00 2024-05-09T09:06:15.519 INFO:teuthology.orchestra.run.smithi103.stdout:(102/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 97 kB/s | 64 kB 00:00 2024-05-09T09:06:15.669 INFO:teuthology.orchestra.run.smithi103.stdout:(103/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 73 kB/s | 45 kB 00:00 2024-05-09T09:06:15.851 INFO:teuthology.orchestra.run.smithi163.stdout:(82/156): python3-numpy-f2py-1.20.1-5.el9.x86_6 308 kB/s | 250 kB 00:00 2024-05-09T09:06:15.886 INFO:teuthology.orchestra.run.smithi103.stdout:(104/156): protobuf-compiler-3.14.0-13.el9.x86_ 2.1 MB/s | 863 kB 00:00 2024-05-09T09:06:15.960 INFO:teuthology.orchestra.run.smithi163.stdout:(83/156): python3-oauthlib-3.1.1-5.el9.noarch.r 275 kB/s | 222 kB 00:00 2024-05-09T09:06:16.041 INFO:teuthology.orchestra.run.smithi163.stdout:(84/156): python3-numpy-1.20.1-5.el9.x86_64.rpm 4.1 MB/s | 5.1 MB 00:01 2024-05-09T09:06:16.257 INFO:teuthology.orchestra.run.smithi069.stdout: Preparing : 1/1 2024-05-09T09:06:16.361 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T09:06:16.378 INFO:teuthology.orchestra.run.smithi103.stdout:(105/156): grpc-data-1.46.7-10.el9.noarch.rpm 42 kB/s | 19 kB 00:00 2024-05-09T09:06:16.400 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T09:06:16.460 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T09:06:16.494 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T09:06:16.520 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T09:06:16.566 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T09:06:16.602 INFO:teuthology.orchestra.run.smithi163.stdout:(85/156): python3-prettytable-0.7.2-27.el9.noar 66 kB/s | 42 kB 00:00 2024-05-09T09:06:16.627 INFO:teuthology.orchestra.run.smithi163.stdout:(86/156): python3-packaging-20.9-5.el9.noarch.r 100 kB/s | 77 kB 00:00 2024-05-09T09:06:16.695 INFO:teuthology.orchestra.run.smithi103.stdout:(106/156): gperftools-libs-2.9.1-2.el9.x86_64.r 301 kB/s | 309 kB 00:01 2024-05-09T09:06:16.796 INFO:teuthology.orchestra.run.smithi103.stdout:(107/156): abseil-cpp-20211102.0-3.el9.x86_64.r 430 kB/s | 548 kB 00:01 2024-05-09T09:06:16.812 INFO:teuthology.orchestra.run.smithi103.stdout:(108/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 222 kB/s | 26 kB 00:00 2024-05-09T09:06:16.888 INFO:teuthology.orchestra.run.smithi103.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-05-09T09:06:16.894 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T09:06:16.902 INFO:teuthology.orchestra.run.smithi163.stdout:(87/156): python3-protobuf-3.14.0-13.el9.noarch 313 kB/s | 269 kB 00:00 2024-05-09T09:06:16.996 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:16.997 INFO:teuthology.orchestra.run.smithi103.stdout:(110/156): libunwind-1.6.2-1.el9.x86_64.rpm 366 kB/s | 67 kB 00:00 2024-05-09T09:06:17.118 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:17.156 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:17.178 INFO:teuthology.orchestra.run.smithi103.stdout:(111/156): luarocks-3.9.2-1.el9.noarch.rpm 525 kB/s | 151 kB 00:00 2024-05-09T09:06:17.247 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:17.277 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:06:17.361 INFO:teuthology.orchestra.run.smithi163.stdout:(88/156): python3-pyasn1-0.4.8-6.el9.noarch.rpm 210 kB/s | 159 kB 00:00 2024-05-09T09:06:17.445 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T09:06:17.478 INFO:teuthology.orchestra.run.smithi163.stdout:(89/156): python3-pyasn1-modules-0.4.8-6.el9.no 328 kB/s | 279 kB 00:00 2024-05-09T09:06:17.522 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T09:06:17.535 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:17.536 INFO:teuthology.orchestra.run.smithi163.stdout:(90/156): python3-pytz-2021.1-5.el9.noarch.rpm 80 kB/s | 51 kB 00:00 2024-05-09T09:06:17.638 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:17.670 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:17.735 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:17.762 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:17.790 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:17.964 INFO:teuthology.orchestra.run.smithi103.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 696 kB/s | 548 kB 00:00 2024-05-09T09:06:17.972 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:18.081 INFO:teuthology.orchestra.run.smithi103.stdout:(113/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 373 kB/s | 43 kB 00:00 2024-05-09T09:06:18.095 INFO:teuthology.orchestra.run.smithi163.stdout:(91/156): python3-requests-oauthlib-1.3.0-12.el 73 kB/s | 54 kB 00:00 2024-05-09T09:06:18.123 INFO:teuthology.orchestra.run.smithi103.stdout:(114/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 745 kB/s | 839 kB 00:01 2024-05-09T09:06:18.153 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:18.191 INFO:teuthology.orchestra.run.smithi103.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 296 kB/s | 32 kB 00:00 2024-05-09T09:06:18.232 INFO:teuthology.orchestra.run.smithi103.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-05-09T09:06:18.237 INFO:teuthology.orchestra.run.smithi163.stdout:(92/156): python3-toml-0.10.2-6.el9.noarch.rpm 59 kB/s | 42 kB 00:00 2024-05-09T09:06:18.293 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:18.353 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:18.465 INFO:teuthology.orchestra.run.smithi103.stdout:(117/156): python3-cheroot-8.6.0-4.el9.noarch.r 624 kB/s | 172 kB 00:00 2024-05-09T09:06:18.469 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:18.537 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:18.749 INFO:teuthology.orchestra.run.smithi103.stdout:(118/156): python3-cherrypy-18.6.1-2.el9.noarch 693 kB/s | 358 kB 00:00 2024-05-09T09:06:18.751 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:18.754 INFO:teuthology.orchestra.run.smithi163.stdout:(93/156): python3-wcwidth-0.2.5-8.el9.noarch.rp 67 kB/s | 44 kB 00:00 2024-05-09T09:06:18.857 INFO:teuthology.orchestra.run.smithi103.stdout:(119/156): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-05-09T09:06:18.865 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T09:06:18.916 INFO:teuthology.orchestra.run.smithi103.stdout:(120/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 673 kB/s | 303 kB 00:00 2024-05-09T09:06:18.961 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T09:06:18.992 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T09:06:19.026 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T09:06:19.091 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T09:06:19.112 INFO:teuthology.orchestra.run.smithi163.stdout:(94/156): python3-scipy-1.6.2-8.el9.x86_64.rpm 9.9 MB/s | 16 MB 00:01 2024-05-09T09:06:19.126 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T09:06:19.162 INFO:teuthology.orchestra.run.smithi163.stdout:(95/156): qatlib-24.02.0-1.el9.x86_64.rpm 238 kB/s | 221 kB 00:00 2024-05-09T09:06:19.169 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T09:06:19.203 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T09:06:19.208 INFO:teuthology.orchestra.run.smithi103.stdout:(121/156): python3-google-auth-2.29.0-1.el9.noa 587 kB/s | 206 kB 00:00 2024-05-09T09:06:19.230 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T09:06:19.298 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T09:06:19.351 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T09:06:19.383 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T09:06:19.418 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T09:06:19.467 INFO:teuthology.orchestra.run.smithi103.stdout:(122/156): python3-grpcio-tools-1.46.7-10.el9.x 560 kB/s | 144 kB 00:00 2024-05-09T09:06:19.492 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T09:06:19.521 INFO:teuthology.orchestra.run.smithi163.stdout:(96/156): qatlib-service-24.02.0-1.el9.x86_64.r 47 kB/s | 36 kB 00:00 2024-05-09T09:06:19.529 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T09:06:19.572 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:06:19.584 INFO:teuthology.orchestra.run.smithi103.stdout:(123/156): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-05-09T09:06:19.600 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T09:06:19.669 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T09:06:19.692 INFO:teuthology.orchestra.run.smithi103.stdout:(124/156): python3-jaraco-classes-3.2.1-5.el9.n 163 kB/s | 18 kB 00:00 2024-05-09T09:06:19.715 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T09:06:19.788 INFO:teuthology.orchestra.run.smithi163.stdout:(97/156): qatzip-libs-1.2.0-1.el9.x86_64.rpm 73 kB/s | 47 kB 00:00 2024-05-09T09:06:19.789 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T09:06:19.793 INFO:teuthology.orchestra.run.smithi103.stdout:(125/156): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-05-09T09:06:19.901 INFO:teuthology.orchestra.run.smithi103.stdout:(126/156): python3-jaraco-functools-3.5.0-2.el9 180 kB/s | 19 kB 00:00 2024-05-09T09:06:19.945 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T09:06:20.001 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T09:06:20.010 INFO:teuthology.orchestra.run.smithi103.stdout:(127/156): python3-jaraco-text-3.2.0-6.el9.noar 181 kB/s | 20 kB 00:00 2024-05-09T09:06:20.030 INFO:teuthology.orchestra.run.smithi163.stdout:(98/156): socat-1.7.4.1-5.el9.x86_64.rpm 351 kB/s | 305 kB 00:00 2024-05-09T09:06:20.180 INFO:teuthology.orchestra.run.smithi163.stdout:(99/156): uuid-1.6.2-55.el9.x86_64.rpm 88 kB/s | 58 kB 00:00 2024-05-09T09:06:20.519 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T09:06:20.596 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T09:06:20.622 INFO:teuthology.orchestra.run.smithi163.stdout:(100/156): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 112 kB/s | 94 kB 00:00 2024-05-09T09:06:20.626 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T09:06:20.661 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T09:06:20.706 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T09:06:20.723 INFO:teuthology.orchestra.run.smithi163.stdout:(101/156): xmlstarlet-1.6.1-20.el9.x86_64.rpm 95 kB/s | 64 kB 00:00 2024-05-09T09:06:20.739 INFO:teuthology.orchestra.run.smithi163.stdout:(102/156): lua-devel-5.4.4-4.el9.x86_64.rpm 192 kB/s | 22 kB 00:00 2024-05-09T09:06:20.748 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T09:06:20.902 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T09:06:20.932 INFO:teuthology.orchestra.run.smithi163.stdout:(103/156): protobuf-compiler-3.14.0-13.el9.x86_ 4.0 MB/s | 863 kB 00:00 2024-05-09T09:06:20.948 INFO:teuthology.orchestra.run.smithi163.stdout:(104/156): zlib-devel-1.2.11-41.el9.x86_64.rpm 58 kB/s | 45 kB 00:00 2024-05-09T09:06:20.960 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T09:06:20.982 INFO:teuthology.orchestra.run.smithi163.stdout:(105/156): abseil-cpp-20211102.0-3.el9.x86_64.r 2.2 MB/s | 548 kB 00:00 2024-05-09T09:06:21.014 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T09:06:21.051 INFO:teuthology.orchestra.run.smithi163.stdout:(106/156): grpc-data-1.46.7-10.el9.noarch.rpm 194 kB/s | 19 kB 00:00 2024-05-09T09:06:21.086 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T09:06:21.099 INFO:teuthology.orchestra.run.smithi163.stdout:(107/156): libarrow-doc-9.0.0-11.el9.noarch.rpm 521 kB/s | 26 kB 00:00 2024-05-09T09:06:21.118 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T09:06:21.147 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T09:06:21.175 INFO:teuthology.orchestra.run.smithi163.stdout:(108/156): libarrow-9.0.0-11.el9.x86_64.rpm 23 MB/s | 4.4 MB 00:00 2024-05-09T09:06:21.193 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T09:06:21.200 INFO:teuthology.orchestra.run.smithi163.stdout:(109/156): liboath-2.6.7-2.el9.x86_64.rpm 485 kB/s | 49 kB 00:00 2024-05-09T09:06:21.225 INFO:teuthology.orchestra.run.smithi163.stdout:(110/156): gperftools-libs-2.9.1-2.el9.x86_64.r 1.0 MB/s | 309 kB 00:00 2024-05-09T09:06:21.229 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T09:06:21.250 INFO:teuthology.orchestra.run.smithi163.stdout:(111/156): libunwind-1.6.2-1.el9.x86_64.rpm 895 kB/s | 67 kB 00:00 2024-05-09T09:06:21.272 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T09:06:21.309 INFO:teuthology.orchestra.run.smithi163.stdout:(112/156): python3-asyncssh-2.13.2-5.el9.noarch 9.2 MB/s | 548 kB 00:00 2024-05-09T09:06:21.314 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T09:06:21.334 INFO:teuthology.orchestra.run.smithi163.stdout:(113/156): luarocks-3.9.2-1.el9.noarch.rpm 1.1 MB/s | 151 kB 00:00 2024-05-09T09:06:21.360 INFO:teuthology.orchestra.run.smithi163.stdout:(114/156): python3-bcrypt-3.2.2-1.el9.x86_64.rp 864 kB/s | 43 kB 00:00 2024-05-09T09:06:21.385 INFO:teuthology.orchestra.run.smithi163.stdout:(115/156): python3-cachetools-4.2.4-1.el9.noarc 640 kB/s | 32 kB 00:00 2024-05-09T09:06:21.402 INFO:teuthology.orchestra.run.smithi163.stdout:(116/156): python3-certifi-2021.10.8-2.el9.noar 359 kB/s | 15 kB 00:00 2024-05-09T09:06:21.427 INFO:teuthology.orchestra.run.smithi103.stdout:(128/156): python3-kubernetes-26.1.0-2.el9.noar 738 kB/s | 1.0 MB 00:01 2024-05-09T09:06:21.435 INFO:teuthology.orchestra.run.smithi163.stdout:(117/156): parquet-libs-9.0.0-11.el9.x86_64.rpm 3.9 MB/s | 839 kB 00:00 2024-05-09T09:06:21.449 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T09:06:21.461 INFO:teuthology.orchestra.run.smithi163.stdout:(118/156): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-05-09T09:06:21.477 INFO:teuthology.orchestra.run.smithi103.stdout:(129/156): python3-grpcio-1.46.7-10.el9.x86_64. 815 kB/s | 2.0 MB 00:02 2024-05-09T09:06:21.486 INFO:teuthology.orchestra.run.smithi163.stdout:(119/156): python3-cherrypy-18.6.1-2.el9.noarch 4.2 MB/s | 358 kB 00:00 2024-05-09T09:06:21.503 INFO:teuthology.orchestra.run.smithi163.stdout:(120/156): python3-colorama-0.4.4-12.el9.noarch 846 kB/s | 35 kB 00:00 2024-05-09T09:06:21.503 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T09:06:21.527 INFO:teuthology.orchestra.run.smithi103.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-05-09T09:06:21.528 INFO:teuthology.orchestra.run.smithi163.stdout:(121/156): python3-cmd2-2.4.2-2.el9.noarch.rpm 3.2 MB/s | 303 kB 00:00 2024-05-09T09:06:21.534 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T09:06:21.553 INFO:teuthology.orchestra.run.smithi163.stdout:(122/156): python3-google-auth-2.29.0-1.el9.noa 3.0 MB/s | 206 kB 00:00 2024-05-09T09:06:21.575 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T09:06:21.579 INFO:teuthology.orchestra.run.smithi163.stdout:(123/156): python3-grpcio-tools-1.46.7-10.el9.x 2.8 MB/s | 144 kB 00:00 2024-05-09T09:06:21.604 INFO:teuthology.orchestra.run.smithi163.stdout:(124/156): python3-jaraco-8.2.1-3.el9.noarch.rp 213 kB/s | 11 kB 00:00 2024-05-09T09:06:21.629 INFO:teuthology.orchestra.run.smithi163.stdout:(125/156): python3-jaraco-classes-3.2.1-5.el9.n 353 kB/s | 18 kB 00:00 2024-05-09T09:06:21.655 INFO:teuthology.orchestra.run.smithi163.stdout:(126/156): python3-jaraco-collections-3.0.0-8.e 460 kB/s | 23 kB 00:00 2024-05-09T09:06:21.669 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T09:06:21.670 INFO:teuthology.orchestra.run.smithi103.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 410 kB/s | 79 kB 00:00 2024-05-09T09:06:21.680 INFO:teuthology.orchestra.run.smithi163.stdout:(127/156): python3-jaraco-functools-3.5.0-2.el9 385 kB/s | 19 kB 00:00 2024-05-09T09:06:21.705 INFO:teuthology.orchestra.run.smithi163.stdout:(128/156): python3-jaraco-text-3.2.0-6.el9.noar 392 kB/s | 20 kB 00:00 2024-05-09T09:06:21.709 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T09:06:21.720 INFO:teuthology.orchestra.run.smithi103.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 300 kB/s | 58 kB 00:00 2024-05-09T09:06:21.750 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T09:06:21.756 INFO:teuthology.orchestra.run.smithi163.stdout:(129/156): python3-grpcio-1.46.7-10.el9.x86_64. 8.1 MB/s | 2.0 MB 00:00 2024-05-09T09:06:21.781 INFO:teuthology.orchestra.run.smithi163.stdout:(130/156): python3-logutils-0.3.5-21.el9.noarch 615 kB/s | 46 kB 00:00 2024-05-09T09:06:21.783 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T09:06:21.806 INFO:teuthology.orchestra.run.smithi163.stdout:(131/156): python3-more-itertools-8.12.0-2.el9. 1.5 MB/s | 79 kB 00:00 2024-05-09T09:06:21.807 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T09:06:21.820 INFO:teuthology.orchestra.run.smithi103.stdout:(133/156): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-05-09T09:06:21.831 INFO:teuthology.orchestra.run.smithi163.stdout:(132/156): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-05-09T09:06:21.865 INFO:teuthology.orchestra.run.smithi163.stdout:(133/156): python3-kubernetes-26.1.0-2.el9.noar 5.5 MB/s | 1.0 MB 00:00 2024-05-09T09:06:21.890 INFO:teuthology.orchestra.run.smithi163.stdout:(134/156): python3-pecan-1.4.2-3.el9.noarch.rpm 3.2 MB/s | 272 kB 00:00 2024-05-09T09:06:21.915 INFO:teuthology.orchestra.run.smithi163.stdout:(135/156): python3-portend-3.1.0-2.el9.noarch.r 196 kB/s | 16 kB 00:00 2024-05-09T09:06:21.933 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T09:06:21.941 INFO:teuthology.orchestra.run.smithi163.stdout:(136/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 1.2 MB/s | 90 kB 00:00 2024-05-09T09:06:21.966 INFO:teuthology.orchestra.run.smithi163.stdout:(137/156): python3-pyperclip-1.8.2-2.el9.noarch 564 kB/s | 28 kB 00:00 2024-05-09T09:06:21.991 INFO:teuthology.orchestra.run.smithi163.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 409 kB/s | 31 kB 00:00 2024-05-09T09:06:22.004 INFO:teuthology.orchestra.run.smithi103.stdout:(134/156): python3-pyOpenSSL-21.0.0-1.el9.noarc 490 kB/s | 90 kB 00:00 2024-05-09T09:06:22.016 INFO:teuthology.orchestra.run.smithi163.stdout:(139/156): python3-routes-2.5.1-5.el9.noarch.rp 2.4 MB/s | 188 kB 00:00 2024-05-09T09:06:22.042 INFO:teuthology.orchestra.run.smithi163.stdout:(140/156): python3-tempora-5.0.0-2.el9.noarch.r 716 kB/s | 36 kB 00:00 2024-05-09T09:06:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T09:06:22.067 INFO:teuthology.orchestra.run.smithi163.stdout:(141/156): python3-rsa-4.9-2.el9.noarch.rpm 785 kB/s | 59 kB 00:00 2024-05-09T09:06:22.092 INFO:teuthology.orchestra.run.smithi163.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 676 kB/s | 51 kB 00:00 2024-05-09T09:06:22.098 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T09:06:22.109 INFO:teuthology.orchestra.run.smithi163.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 2.1 MB/s | 90 kB 00:00 2024-05-09T09:06:22.112 INFO:teuthology.orchestra.run.smithi103.stdout:(135/156): python3-pecan-1.4.2-3.el9.noarch.rpm 615 kB/s | 272 kB 00:00 2024-05-09T09:06:22.126 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T09:06:22.134 INFO:teuthology.orchestra.run.smithi163.stdout:(144/156): python3-webob-1.8.7-6.el9.noarch.rpm 2.4 MB/s | 230 kB 00:00 2024-05-09T09:06:22.138 INFO:teuthology.orchestra.run.smithi103.stdout:(136/156): python3-pyperclip-1.8.2-2.el9.noarch 211 kB/s | 28 kB 00:00 2024-05-09T09:06:22.160 INFO:teuthology.orchestra.run.smithi163.stdout:(145/156): python3-zc-lockfile-2.0-10.el9.noarc 398 kB/s | 20 kB 00:00 2024-05-09T09:06:22.161 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T09:06:22.185 INFO:teuthology.orchestra.run.smithi163.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 3.7 MB/s | 192 kB 00:00 2024-05-09T09:06:22.199 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T09:06:22.219 INFO:teuthology.orchestra.run.smithi163.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 3.3 MB/s | 427 kB 00:00 2024-05-09T09:06:22.231 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T09:06:22.261 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T09:06:22.263 INFO:teuthology.orchestra.run.smithi103.stdout:(137/156): libarrow-9.0.0-11.el9.x86_64.rpm 768 kB/s | 4.4 MB 00:05 2024-05-09T09:06:22.288 INFO:teuthology.orchestra.run.smithi103.stdout:(138/156): python3-repoze-lru-0.7-16.el9.noarch 175 kB/s | 31 kB 00:00 2024-05-09T09:06:22.295 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T09:06:22.319 INFO:teuthology.orchestra.run.smithi163.stdout:(148/156): thrift-0.15.0-2.el9.x86_64.rpm 10 MB/s | 1.6 MB 00:00 2024-05-09T09:06:22.329 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T09:06:22.370 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T09:06:22.397 INFO:teuthology.orchestra.run.smithi103.stdout:(139/156): python3-tempora-5.0.0-2.el9.noarch.r 331 kB/s | 36 kB 00:00 2024-05-09T09:06:22.405 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T09:06:22.440 INFO:teuthology.orchestra.run.smithi103.stdout:(140/156): python3-routes-2.5.1-5.el9.noarch.rp 627 kB/s | 188 kB 00:00 2024-05-09T09:06:22.453 INFO:teuthology.orchestra.run.smithi163.stdout:(149/156): NetworkManager-1.47.5-1.el9.x86_64.r 17 MB/s | 2.3 MB 00:00 2024-05-09T09:06:22.473 INFO:teuthology.orchestra.run.smithi103.stdout:(141/156): python3-rsa-4.9-2.el9.noarch.rpm 322 kB/s | 59 kB 00:00 2024-05-09T09:06:22.498 INFO:teuthology.orchestra.run.smithi103.stdout:(142/156): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-05-09T09:06:22.502 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T09:06:22.544 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T09:06:22.569 INFO:teuthology.orchestra.run.smithi163.stdout:(150/156): NetworkManager-libnm-1.47.5-1.el9.x8 16 MB/s | 1.8 MB 00:00 2024-05-09T09:06:22.584 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T09:06:22.628 INFO:teuthology.orchestra.run.smithi163.stdout:(151/156): librados2-19.0.0-2515.g96805dcf.el9. 7.7 MB/s | 3.4 MB 00:00 2024-05-09T09:06:22.630 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T09:06:22.648 INFO:teuthology.orchestra.run.smithi103.stdout:(143/156): python3-websocket-client-1.2.3-2.el9 511 kB/s | 90 kB 00:00 2024-05-09T09:06:22.678 INFO:teuthology.orchestra.run.smithi163.stdout:(152/156): NetworkManager-team-1.47.5-1.el9.x86 385 kB/s | 42 kB 00:00 2024-05-09T09:06:22.728 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T09:06:22.729 INFO:teuthology.orchestra.run.smithi163.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-05-09T09:06:22.748 INFO:teuthology.orchestra.run.smithi103.stdout:(144/156): python3-zc-lockfile-2.0-10.el9.noarc 200 kB/s | 20 kB 00:00 2024-05-09T09:06:22.776 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T09:06:22.782 INFO:teuthology.orchestra.run.smithi103.stdout:(145/156): python3-webob-1.8.7-6.el9.noarch.rpm 670 kB/s | 230 kB 00:00 2024-05-09T09:06:22.787 INFO:teuthology.orchestra.run.smithi163.stdout:(154/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 5.5 MB/s | 3.1 MB 00:00 2024-05-09T09:06:22.846 INFO:teuthology.orchestra.run.smithi163.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 8.1 MB/s | 1.3 MB 00:00 2024-05-09T09:06:22.979 INFO:teuthology.orchestra.run.smithi163.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 9.6 MB/s | 2.4 MB 00:00 2024-05-09T09:06:22.983 INFO:teuthology.orchestra.run.smithi163.stdout:-------------------------------------------------------------------------------- 2024-05-09T09:06:22.983 INFO:teuthology.orchestra.run.smithi163.stdout:Total 7.1 MB/s | 225 MB 00:31 2024-05-09T09:06:23.007 INFO:teuthology.orchestra.run.smithi103.stdout:(146/156): re2-20211101-3.el9.x86_64.rpm 741 kB/s | 192 kB 00:00 2024-05-09T09:06:23.116 INFO:teuthology.orchestra.run.smithi103.stdout:(147/156): python3-werkzeug-2.0.3-3.el9.1.noarc 691 kB/s | 427 kB 00:00 2024-05-09T09:06:23.441 INFO:teuthology.orchestra.run.smithi103.stdout:(148/156): librbd1-19.0.0-2515.g96805dcf.el9.x8 9.7 MB/s | 3.1 MB 00:00 2024-05-09T09:06:23.486 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T09:06:23.532 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T09:06:23.566 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T09:06:23.583 INFO:teuthology.orchestra.run.smithi103.stdout:(149/156): librados2-19.0.0-2515.g96805dcf.el9. 5.9 MB/s | 3.4 MB 00:00 2024-05-09T09:06:23.671 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T09:06:23.731 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T09:06:23.818 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T09:06:23.851 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T09:06:24.027 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T09:06:24.065 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T09:06:24.138 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction check 2024-05-09T09:06:24.143 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T09:06:24.290 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction check succeeded. 2024-05-09T09:06:24.290 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction test 2024-05-09T09:06:24.375 INFO:teuthology.orchestra.run.smithi103.stdout:(150/156): NetworkManager-1.47.5-1.el9.x86_64.r 2.5 MB/s | 2.3 MB 00:00 2024-05-09T09:06:24.475 INFO:teuthology.orchestra.run.smithi103.stdout:(151/156): NetworkManager-team-1.47.5-1.el9.x86 418 kB/s | 42 kB 00:00 2024-05-09T09:06:24.601 INFO:teuthology.orchestra.run.smithi103.stdout:(152/156): NetworkManager-libnm-1.47.5-1.el9.x8 1.8 MB/s | 1.8 MB 00:01 2024-05-09T09:06:24.626 INFO:teuthology.orchestra.run.smithi103.stdout:(153/156): NetworkManager-tui-1.47.5-1.el9.x86_ 1.6 MB/s | 248 kB 00:00 2024-05-09T09:06:24.760 INFO:teuthology.orchestra.run.smithi103.stdout:(154/156): thrift-0.15.0-2.el9.x86_64.rpm 819 kB/s | 1.6 MB 00:01 2024-05-09T09:06:24.785 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T09:06:24.810 INFO:teuthology.orchestra.run.smithi103.stdout:(155/156): openssl-3.2.1-1.el9.x86_64.rpm 6.5 MB/s | 1.3 MB 00:00 2024-05-09T09:06:24.895 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T09:06:25.643 INFO:teuthology.orchestra.run.smithi103.stdout:(156/156): openssl-libs-3.2.1-1.el9.x86_64.rpm 2.4 MB/s | 2.4 MB 00:01 2024-05-09T09:06:25.650 INFO:teuthology.orchestra.run.smithi103.stdout:-------------------------------------------------------------------------------- 2024-05-09T09:06:25.650 INFO:teuthology.orchestra.run.smithi103.stdout:Total 6.3 MB/s | 225 MB 00:35 2024-05-09T09:06:26.083 INFO:teuthology.orchestra.run.smithi163.stdout:Transaction test succeeded. 2024-05-09T09:06:26.083 INFO:teuthology.orchestra.run.smithi163.stdout:Running transaction 2024-05-09T09:06:26.123 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T09:06:26.219 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T09:06:26.246 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T09:06:26.280 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T09:06:26.601 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T09:06:26.643 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:26.732 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:26.764 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T09:06:26.800 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T09:06:26.836 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction check 2024-05-09T09:06:26.986 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction check succeeded. 2024-05-09T09:06:26.987 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction test 2024-05-09T09:06:27.273 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T09:06:27.304 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:27.366 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:27.389 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T09:06:28.800 INFO:teuthology.orchestra.run.smithi103.stdout:Transaction test succeeded. 2024-05-09T09:06:28.800 INFO:teuthology.orchestra.run.smithi103.stdout:Running transaction 2024-05-09T09:06:29.577 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:29.619 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:30.163 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:30.205 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:30.298 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T09:06:30.298 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T09:06:30.298 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:30.348 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:06:32.192 INFO:teuthology.orchestra.run.smithi163.stdout: Preparing : 1/1 2024-05-09T09:06:32.258 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T09:06:32.297 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T09:06:32.350 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T09:06:32.385 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T09:06:32.410 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T09:06:32.453 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T09:06:32.780 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T09:06:32.903 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:33.008 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:33.037 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:33.129 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:33.158 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:06:33.310 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T09:06:33.380 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T09:06:33.407 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:33.510 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:33.533 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:33.596 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:33.618 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:33.644 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:33.843 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:33.974 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:34.140 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:34.184 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:34.305 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:34.369 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:34.567 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:34.673 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T09:06:34.770 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T09:06:34.816 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T09:06:34.839 INFO:teuthology.orchestra.run.smithi103.stdout: Preparing : 1/1 2024-05-09T09:06:34.851 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T09:06:34.916 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T09:06:34.925 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-libs-1:3.2.1-1.el9.x86_64 1/164 2024-05-09T09:06:34.950 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T09:06:34.961 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 2/164 2024-05-09T09:06:34.993 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T09:06:35.024 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 3/164 2024-05-09T09:06:35.026 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T09:06:35.054 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T09:06:35.058 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : thrift-0.15.0-2.el9.x86_64 4/164 2024-05-09T09:06:35.083 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 5/164 2024-05-09T09:06:35.115 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T09:06:35.125 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 6/164 2024-05-09T09:06:35.168 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T09:06:35.198 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T09:06:35.233 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T09:06:35.310 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T09:06:35.344 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T09:06:35.387 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:06:35.416 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T09:06:35.446 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 7/164 2024-05-09T09:06:35.488 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T09:06:35.539 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T09:06:35.567 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:35.615 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T09:06:35.689 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:06:35.727 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:35.750 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T09:06:35.800 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T09:06:35.818 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:06:35.848 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:06:36.007 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-private-libs-13.14-1.el9.x86_64 11/164 2024-05-09T09:06:36.062 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-13.14-1.el9.x86_64 12/164 2024-05-09T09:06:36.090 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:36.193 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:06:36.216 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:36.278 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 14/164 2024-05-09T09:06:36.300 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:36.322 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-service-24.02.0-1.el9.x86_64 15/164 2024-05-09T09:06:36.338 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T09:06:36.421 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T09:06:36.449 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T09:06:36.500 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T09:06:36.533 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:36.547 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T09:06:36.589 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T09:06:36.661 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatlib-24.02.0-1.el9.x86_64 16/164 2024-05-09T09:06:36.751 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T09:06:36.814 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:36.825 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T09:06:36.875 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 17/164 2024-05-09T09:06:36.880 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T09:06:36.953 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T09:06:36.981 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:36.983 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T09:06:37.020 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T09:06:37.043 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:37.067 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T09:06:37.103 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T09:06:37.146 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T09:06:37.188 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T09:06:37.234 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.5-1.el9.x86_64 18/164 2024-05-09T09:06:37.326 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T09:06:37.340 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : re2-1:20211101-3.el9.x86_64 19/164 2024-05-09T09:06:37.377 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T09:06:37.407 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T09:06:37.445 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-9.0.0-11.el9.x86_64 20/164 2024-05-09T09:06:37.448 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T09:06:37.474 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 21/164 2024-05-09T09:06:37.509 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : liboath-2.6.7-2.el9.x86_64 22/164 2024-05-09T09:06:37.543 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T09:06:37.574 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 23/164 2024-05-09T09:06:37.581 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T09:06:37.617 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 24/164 2024-05-09T09:06:37.621 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T09:06:37.647 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T09:06:37.660 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/164 2024-05-09T09:06:37.671 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T09:06:37.692 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 26/164 2024-05-09T09:06:37.720 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-5.4.4-4.el9.x86_64 27/164 2024-05-09T09:06:37.747 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T09:06:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 28/164 2024-05-09T09:06:37.843 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 29/164 2024-05-09T09:06:37.874 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 30/164 2024-05-09T09:06:37.875 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T09:06:37.910 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 31/164 2024-05-09T09:06:37.913 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T09:06:37.940 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T09:06:37.974 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T09:06:37.986 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 32/164 2024-05-09T09:06:38.014 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T09:06:38.020 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 33/164 2024-05-09T09:06:38.045 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T09:06:38.063 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:06:38.075 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T09:06:38.092 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 35/164 2024-05-09T09:06:38.109 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T09:06:38.143 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T09:06:38.163 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 36/164 2024-05-09T09:06:38.184 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T09:06:38.207 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : luarocks-3.9.2-1.el9.noarch 37/164 2024-05-09T09:06:38.219 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T09:06:38.283 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 38/164 2024-05-09T09:06:38.307 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T09:06:38.349 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T09:06:38.380 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T09:06:38.419 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 39/164 2024-05-09T09:06:38.427 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T09:06:38.469 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 40/164 2024-05-09T09:06:38.543 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T09:06:38.598 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T09:06:39.009 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rsa-4.9-2.el9.noarch 41/164 2024-05-09T09:06:39.090 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 42/164 2024-05-09T09:06:39.118 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 43/164 2024-05-09T09:06:39.161 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : qatzip-libs-1.2.0-1.el9.x86_64 44/164 2024-05-09T09:06:39.200 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 45/164 2024-05-09T09:06:39.242 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 46/164 2024-05-09T09:06:39.336 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T09:06:39.387 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T09:06:39.405 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 47/164 2024-05-09T09:06:39.421 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T09:06:39.461 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : openssl-1:3.2.1-1.el9.x86_64 48/164 2024-05-09T09:06:39.516 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 49/164 2024-05-09T09:06:39.524 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T09:06:39.579 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T09:06:39.589 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 50/164 2024-05-09T09:06:39.620 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 51/164 2024-05-09T09:06:39.656 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 52/164 2024-05-09T09:06:39.668 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T09:06:39.698 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T09:06:39.704 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 53/164 2024-05-09T09:06:39.782 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 54/164 2024-05-09T09:06:39.868 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T09:06:39.895 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T09:06:39.958 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 55/164 2024-05-09T09:06:39.973 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T09:06:40.151 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 56/164 2024-05-09T09:06:40.316 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 57/164 2024-05-09T09:06:40.363 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 58/164 2024-05-09T09:06:40.394 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 59/164 2024-05-09T09:06:40.435 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 60/164 2024-05-09T09:06:40.531 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 61/164 2024-05-09T09:06:40.569 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-google-auth-1:2.29.0-1.el9.noarch 62/164 2024-05-09T09:06:40.609 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 63/164 2024-05-09T09:06:40.626 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T09:06:40.634 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 64/164 2024-05-09T09:06:40.659 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libarrow-doc-9.0.0-11.el9.noarch 65/164 2024-05-09T09:06:40.726 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 66/164 2024-05-09T09:06:40.746 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T09:06:40.854 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 67/164 2024-05-09T09:06:40.892 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 68/164 2024-05-09T09:06:40.920 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : zlib-devel-1.2.11-41.el9.x86_64 69/164 2024-05-09T09:06:40.954 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 70/164 2024-05-09T09:06:40.994 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : uuid-1.6.2-55.el9.x86_64 71/164 2024-05-09T09:06:41.025 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 72/164 2024-05-09T09:06:41.055 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 73/164 2024-05-09T09:06:41.089 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 74/164 2024-05-09T09:06:41.123 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 75/164 2024-05-09T09:06:41.164 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 76/164 2024-05-09T09:06:41.199 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 77/164 2024-05-09T09:06:41.287 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 78/164 2024-05-09T09:06:41.338 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 79/164 2024-05-09T09:06:41.369 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 80/164 2024-05-09T09:06:41.417 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 81/164 2024-05-09T09:06:41.516 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-packaging-20.9-5.el9.noarch 82/164 2024-05-09T09:06:41.563 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 83/164 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /sys 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /proc 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /mnt 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /var/tmp 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /home 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /root 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout:skipping the directory /tmp 2024-05-09T09:06:41.576 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:41.811 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:06:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:06:41.829 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:41.829 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T09:06:41.829 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:41.999 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T09:06:42.083 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T09:06:42.111 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T09:06:42.144 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T09:06:42.309 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 84/164 2024-05-09T09:06:42.352 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 85/164 2024-05-09T09:06:42.385 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 86/164 2024-05-09T09:06:42.464 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T09:06:42.488 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 87/164 2024-05-09T09:06:42.507 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:42.544 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 88/164 2024-05-09T09:06:42.598 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:42.628 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T09:06:42.633 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 89/164 2024-05-09T09:06:42.664 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T09:06:42.671 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 90/164 2024-05-09T09:06:42.848 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-0.3.21-2.el9.x86_64 91/164 2024-05-09T09:06:42.886 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 92/164 2024-05-09T09:06:42.963 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 93/164 2024-05-09T09:06:43.138 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T09:06:43.168 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:43.239 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:43.261 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T09:06:43.314 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:06:43.341 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:06:43.341 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:43.342 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T09:06:43.342 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:06:43.342 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:06:43.342 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:43.564 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:06:43.588 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:06:43.588 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:43.588 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T09:06:43.588 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:06:43.588 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:06:43.589 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:43.624 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 94/164 2024-05-09T09:06:43.738 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 95/164 2024-05-09T09:06:44.040 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:06:44.064 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:44.178 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T09:06:44.244 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T09:06:44.307 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T09:06:44.496 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T09:06:44.549 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T09:06:44.634 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T09:06:44.722 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T09:06:44.860 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T09:06:44.934 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T09:06:44.976 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T09:06:44.983 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 96/164 2024-05-09T09:06:45.008 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T09:06:45.075 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 97/164 2024-05-09T09:06:45.103 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 98/164 2024-05-09T09:06:45.136 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 99/164 2024-05-09T09:06:45.359 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:06:45.382 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:06:45.413 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T09:06:45.448 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 100/164 2024-05-09T09:06:45.452 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:45.475 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T09:06:45.491 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:45.491 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:45.504 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:06:45.580 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 101/164 2024-05-09T09:06:45.612 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 102/164 2024-05-09T09:06:45.649 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 103/164 2024-05-09T09:06:46.033 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:46.077 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:46.121 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : parquet-libs-9.0.0-11.el9.x86_64 104/164 2024-05-09T09:06:46.152 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:46.162 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T09:06:46.162 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T09:06:46.162 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:06:46.204 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:06:46.223 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 105/164 2024-05-09T09:06:46.254 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 106/164 2024-05-09T09:06:46.538 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:06:46.573 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:06:47.534 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:06:47.557 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:06:47.659 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:06:47.826 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T09:06:47.861 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:06:47.885 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:47.936 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:06:47.948 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:06:48.046 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T09:06:48.099 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T09:06:48.533 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:48.567 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:48.989 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T09:06:49.039 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:06:49.063 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:06:49.064 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:49.064 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T09:06:49.064 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:06:49.064 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:06:49.064 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:49.104 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 107/164 2024-05-09T09:06:49.106 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:06:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:06:49.130 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:49.130 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T09:06:49.130 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:49.146 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:49.182 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:06:49.205 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:06:49.206 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:49.206 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T09:06:49.206 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:06:49.206 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:06:49.206 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:49.243 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 108/164 2024-05-09T09:06:49.243 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-05-09T09:06:49.244 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-05-09T09:06:49.244 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:06:49.280 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:06:49.507 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:06:49.531 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:06:54.061 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T09:06:54.102 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T09:06:54.142 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T09:06:54.183 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T09:06:54.210 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T09:06:54.399 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T09:06:54.448 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T09:06:54.478 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T09:06:54.508 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T09:06:54.546 INFO:teuthology.orchestra.run.smithi069.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T09:06:55.151 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:06:55.185 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:06:55.206 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:06:56.040 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T09:06:56.131 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T09:06:56.178 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T09:06:56.275 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T09:06:56.309 INFO:teuthology.orchestra.run.smithi069.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T09:06:56.335 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T09:06:56.335 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:06:56.402 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:06:56.444 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T09:06:56.444 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:06:56.529 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:06:56.562 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T09:06:56.568 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:06:56.568 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:06:56.747 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:06:56.747 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:06:56.831 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:06:56.896 INFO:teuthology.orchestra.run.smithi069.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /sys 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /proc 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /mnt 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /var/tmp 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /home 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /root 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout:skipping the directory /tmp 2024-05-09T09:06:57.665 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:06:57.831 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:06:57.846 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:06:57.847 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:57.847 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T09:06:57.847 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:06:59.343 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:06:59.368 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:06:59.368 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:59.368 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T09:06:59.368 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:06:59.368 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:06:59.369 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:06:59.603 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:06:59.625 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:00.062 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:07:00.084 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:00.192 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T09:07:00.249 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T09:07:00.305 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T09:07:00.491 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T09:07:00.537 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T09:07:00.615 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T09:07:00.704 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T09:07:00.845 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T09:07:00.895 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T09:07:00.927 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T09:07:00.980 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 109/164 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /sys 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /proc 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /mnt 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /var/tmp 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /home 2024-05-09T09:07:01.049 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /root 2024-05-09T09:07:01.050 INFO:teuthology.orchestra.run.smithi103.stdout:skipping the directory /tmp 2024-05-09T09:07:01.050 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:01.210 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:07:01.225 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 110/164 2024-05-09T09:07:01.225 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:01.225 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-05-09T09:07:01.225 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:01.437 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:07:01.461 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:07:01.493 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T09:07:01.555 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T09:07:01.584 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:07:02.407 INFO:teuthology.orchestra.run.smithi069.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:02.407 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T09:07:02.407 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T09:07:02.407 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T09:07:02.408 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T09:07:02.409 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T09:07:02.410 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T09:07:02.411 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T09:07:02.412 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T09:07:02.413 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T09:07:02.414 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T09:07:02.415 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T09:07:02.416 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T09:07:02.417 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T09:07:02.418 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T09:07:02.419 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T09:07:02.619 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:07:02.652 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:07:02.687 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 111/164 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-05-09T09:07:02.713 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:02.955 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:07:02.977 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 112/164 2024-05-09T09:07:02.977 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:02.978 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-05-09T09:07:02.978 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:07:02.978 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-05-09T09:07:02.978 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:03.433 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 113/164 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-05-09T09:07:03.456 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:03.562 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : automake-1.16.2-8.el9.noarch 114/164 2024-05-09T09:07:03.617 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:07:03.619 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-ply-3.11-14.el9.noarch 115/164 2024-05-09T09:07:03.653 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:07:03.674 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 116/164 2024-05-09T09:07:03.756 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:07:03.860 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 117/164 2024-05-09T09:07:03.965 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 118/164 2024-05-09T09:07:03.965 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T09:07:03.965 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:07:03.979 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:03.994 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 119/164 2024-05-09T09:07:04.023 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:07:04.033 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:07:04.083 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 120/164 2024-05-09T09:07:04.131 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T09:07:04.179 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T09:07:04.222 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 121/164 2024-05-09T09:07:04.275 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 122/164 2024-05-09T09:07:04.307 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 123/164 2024-05-09T09:07:04.335 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : fuse-2.9.9-15.el9.x86_64 124/164 2024-05-09T09:07:04.775 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:07:04.800 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 125/164 2024-05-09T09:07:04.831 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 126/164 2024-05-09T09:07:04.893 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 127/164 2024-05-09T09:07:04.923 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:07:04.985 INFO:teuthology.orchestra.run.smithi069.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout:Upgraded: 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout:Installed: 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T09:07:04.986 INFO:teuthology.orchestra.run.smithi069.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.987 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T09:07:04.988 INFO:teuthology.orchestra.run.smithi069.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.989 INFO:teuthology.orchestra.run.smithi069.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T09:07:04.990 INFO:teuthology.orchestra.run.smithi069.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T09:07:04.991 INFO:teuthology.orchestra.run.smithi069.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T09:07:04.992 INFO:teuthology.orchestra.run.smithi069.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T09:07:04.993 INFO:teuthology.orchestra.run.smithi069.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T09:07:04.994 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T09:07:04.995 INFO:teuthology.orchestra.run.smithi069.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:04.996 INFO:teuthology.orchestra.run.smithi069.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:07:04.997 INFO:teuthology.orchestra.run.smithi069.stdout:Complete! 2024-05-09T09:07:05.079 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T09:07:05.108 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:07:05.131 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:07:05.131 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:05.131 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T09:07:05.131 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:07:05.131 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:07:05.132 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:05.176 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:07:05.197 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:07:05.197 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:05.197 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T09:07:05.197 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:05.243 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:07:05.265 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:07:05.265 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:05.265 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T09:07:05.265 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:07:05.265 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:07:05.266 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:05.479 DEBUG:teuthology.parallel:result is None 2024-05-09T09:07:05.585 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:07:05.608 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:05.950 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 128/164 2024-05-09T09:07:05.992 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:07:06.948 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 129/164 2024-05-09T09:07:06.984 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:07:07.086 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 130/164 2024-05-09T09:07:07.256 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 131/164 2024-05-09T09:07:07.294 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:07:07.317 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 132/164 2024-05-09T09:07:07.317 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:07.317 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-05-09T09:07:07.318 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:07:07.318 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-05-09T09:07:07.318 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:07.361 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:07:07.373 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 133/164 2024-05-09T09:07:07.469 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 134/164 2024-05-09T09:07:07.518 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 135/164 2024-05-09T09:07:08.408 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libtool-2.4.6-45.el9.x86_64 136/164 2024-05-09T09:07:08.439 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:07:08.462 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 137/164 2024-05-09T09:07:08.462 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:08.462 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-05-09T09:07:08.462 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:07:08.462 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-05-09T09:07:08.463 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:08.506 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:07:08.528 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2515.g96805 138/164 2024-05-09T09:07:08.528 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:08.529 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-05-09T09:07:08.529 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:08.582 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 139/164 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-05-09T09:07:08.605 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:08.916 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:07:08.939 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 140/164 2024-05-09T09:07:08.939 INFO:teuthology.orchestra.run.smithi103.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-05-09T09:07:08.939 INFO:teuthology.orchestra.run.smithi103.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-05-09T09:07:08.939 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:07:08.940 INFO:teuthology.orchestra.run.smithi103.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-05-09T09:07:08.940 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:10.105 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T09:07:10.147 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T09:07:10.188 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T09:07:10.219 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T09:07:10.247 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T09:07:10.439 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T09:07:10.492 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T09:07:10.523 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T09:07:10.553 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T09:07:10.591 INFO:teuthology.orchestra.run.smithi163.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T09:07:11.207 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:11.246 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:11.265 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:12.110 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T09:07:12.209 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T09:07:12.250 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T09:07:12.334 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T09:07:12.378 INFO:teuthology.orchestra.run.smithi163.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T09:07:12.404 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T09:07:12.404 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:07:12.488 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:07:12.513 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T09:07:12.513 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:07:12.597 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:07:12.622 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T09:07:12.627 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:12.627 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:12.824 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:07:12.908 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:07:12.973 INFO:teuthology.orchestra.run.smithi163.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:13.445 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 141/164 2024-05-09T09:07:13.486 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 142/164 2024-05-09T09:07:13.528 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 143/164 2024-05-09T09:07:13.559 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 144/164 2024-05-09T09:07:13.586 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 145/164 2024-05-09T09:07:13.770 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 146/164 2024-05-09T09:07:13.815 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-contrib-13.14-1.el9.x86_64 147/164 2024-05-09T09:07:13.854 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 148/164 2024-05-09T09:07:13.884 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-team-1:1.47.5-1.el9.x86_64 149/164 2024-05-09T09:07:13.930 INFO:teuthology.orchestra.run.smithi103.stdout: Upgrading : NetworkManager-tui-1:1.47.5-1.el9.x86_64 150/164 2024-05-09T09:07:14.563 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:14.602 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:14.623 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: postgresql-server-13.14-1.el9.x86_64 151/164 2024-05-09T09:07:15.463 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 152/164 2024-05-09T09:07:15.549 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : openssl-devel-1:3.2.1-1.el9.x86_64 153/164 2024-05-09T09:07:15.590 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 154/164 2024-05-09T09:07:15.681 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : flex-2.6.4-9.el9.x86_64 155/164 2024-05-09T09:07:15.719 INFO:teuthology.orchestra.run.smithi103.stdout: Installing : bison-3.7.4-5.el9.x86_64 156/164 2024-05-09T09:07:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-tui-1:1.47.4-1.el9.x86_64 157/164 2024-05-09T09:07:15.745 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:07:15.829 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 158/164 2024-05-09T09:07:15.854 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-1:3.0.7-27.el9.x86_64 159/164 2024-05-09T09:07:15.854 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:07:15.938 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 160/164 2024-05-09T09:07:15.963 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-team-1:1.47.4-1.el9.x86_64 161/164 2024-05-09T09:07:15.969 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:15.969 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:16.157 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-1:1.47.4-1.el9.x86_64 162/164 2024-05-09T09:07:16.157 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:07:16.241 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.4-1.el9.x86_64 163/164 2024-05-09T09:07:16.305 INFO:teuthology.orchestra.run.smithi103.stdout: Cleanup : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:18.578 INFO:teuthology.orchestra.run.smithi163.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:18.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T09:07:18.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T09:07:18.578 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:07:18.579 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T09:07:18.580 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T09:07:18.581 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T09:07:18.582 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T09:07:18.583 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T09:07:18.583 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:07:18.583 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T09:07:18.584 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T09:07:18.585 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T09:07:18.586 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T09:07:18.587 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T09:07:18.588 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T09:07:18.589 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T09:07:18.590 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T09:07:18.591 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T09:07:18.592 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout:Upgraded: 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:21.213 INFO:teuthology.orchestra.run.smithi163.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout:Installed: 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.214 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.215 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T09:07:21.216 INFO:teuthology.orchestra.run.smithi163.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T09:07:21.217 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T09:07:21.218 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T09:07:21.219 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T09:07:21.220 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T09:07:21.221 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T09:07:21.222 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T09:07:21.223 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T09:07:21.224 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T09:07:21.225 INFO:teuthology.orchestra.run.smithi163.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T09:07:21.225 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:07:21.225 INFO:teuthology.orchestra.run.smithi163.stdout:Complete! 2024-05-09T09:07:21.610 DEBUG:teuthology.parallel:result is None 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Running scriptlet: openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 1/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 3/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 4/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2515.g96805 5/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 6/164 2024-05-09T09:07:22.070 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 7/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 8/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 9/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 10/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 11/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 12/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 13/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 14/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_ 15/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 16/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 17/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_6 18/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86 19/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 20/164 2024-05-09T09:07:22.071 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el 21/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ceph-common-2:19.0.0-2515.g96805dcf.el9. 22/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_6 23/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 24/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 25/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 26/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 27/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 28/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 29/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf. 30/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noa 31/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.n 32/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2515.g968 33/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el 34/164 2024-05-09T09:07:22.072 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 35/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.e 36/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 37/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 38/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 39/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 40/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 41/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 42/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 43/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 44/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-ply-3.11-14.el9.noarch 45/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 46/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 47/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 48/164 2024-05-09T09:07:22.073 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : automake-1.16.2-8.el9.noarch 49/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : bison-3.7.4-5.el9.x86_64 50/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 51/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 52/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flex-2.6.4-9.el9.x86_64 53/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 54/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 55/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 56/164 2024-05-09T09:07:22.074 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 57/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 58/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 59/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 60/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 61/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 62/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 63/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 64/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-5.4.4-4.el9.x86_64 65/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 66/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 67/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-devel-1:3.2.1-1.el9.x86_64 68/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-13.14-1.el9.x86_64 69/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-contrib-13.14-1.el9.x86_64 70/164 2024-05-09T09:07:22.075 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-private-libs-13.14-1.el9.x86_64 71/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : postgresql-server-13.14-1.el9.x86_64 72/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 73/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 74/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 75/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 76/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 77/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 78/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 79/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 80/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 81/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 82/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 83/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 84/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 85/164 2024-05-09T09:07:22.076 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 86/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 87/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 88/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 89/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 90/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 91/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 92/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 93/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 94/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-24.02.0-1.el9.x86_64 95/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatlib-service-24.02.0-1.el9.x86_64 96/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : qatzip-libs-1.2.0-1.el9.x86_64 97/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 98/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 99/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 100/164 2024-05-09T09:07:22.077 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 101/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : zlib-devel-1.2.11-41.el9.x86_64 102/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 103/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 104/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 105/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 106/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 107/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-9.0.0-11.el9.x86_64 108/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libarrow-doc-9.0.0-11.el9.noarch 109/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 110/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 111/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 112/164 2024-05-09T09:07:22.078 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : parquet-libs-9.0.0-11.el9.x86_64 113/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 114/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 115/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 116/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 117/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 118/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 119/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 120/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 121/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-google-auth-1:2.29.0-1.el9.noarch 122/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 123/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 124/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 125/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 126/164 2024-05-09T09:07:22.079 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 127/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 128/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 129/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 130/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 131/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 132/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 133/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 134/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 135/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 136/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 137/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 138/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 139/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 140/164 2024-05-09T09:07:22.080 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 141/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 142/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 143/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 144/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 145/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 146/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : re2-1:20211101-3.el9.x86_64 147/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 148/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 149/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 150/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 151/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 152/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.5-1.el9.x86_64 153/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-1:1.47.4-1.el9.x86_64 154/164 2024-05-09T09:07:22.081 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.5-1.el9.x86_64 155/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-libnm-1:1.47.4-1.el9.x86_64 156/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.5-1.el9.x86_64 157/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-team-1:1.47.4-1.el9.x86_64 158/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.5-1.el9.x86_64 159/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : NetworkManager-tui-1:1.47.4-1.el9.x86_64 160/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.2.1-1.el9.x86_64 161/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 162/164 2024-05-09T09:07:22.082 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.2.1-1.el9.x86_64 163/164 2024-05-09T09:07:24.509 INFO:teuthology.orchestra.run.smithi103.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 164/164 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout:Upgraded: 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-libnm-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-team-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: NetworkManager-tui-1:1.47.5-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: librados2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: librbd1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-libs-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout:Installed: 2024-05-09T09:07:24.510 INFO:teuthology.orchestra.run.smithi103.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: automake-1.16.2-8.el9.noarch 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: bison-3.7.4-5.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-base-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-grafana-dashboards-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-immutable-object-cache-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mds-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-dashboard-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.511 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-modules-core-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mgr-rook-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-mon-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-osd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-prometheus-alerts-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-radosgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-selinux-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-test-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: ceph-volume-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: cephadm-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-shell-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: cephfs-top-2:19.0.0-2515.g96805dcf.el9.noarch 2024-05-09T09:07:24.512 INFO:teuthology.orchestra.run.smithi103.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: flex-2.6.4-9.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: fuse-2.9.9-15.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: grpc-data-1.46.7-10.el9.noarch 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-9.0.0-11.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libarrow-doc-9.0.0-11.el9.noarch 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libcephfs2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libcephsqlite-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-05-09T09:07:24.513 INFO:teuthology.orchestra.run.smithi103.stdout: liboath-2.6.7-2.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: librados-devel-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libradosstriper1-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: librgw2-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libtool-2.4.6-45.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libunwind-1.6.2-1.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-05-09T09:07:24.514 INFO:teuthology.orchestra.run.smithi103.stdout: lua-5.4.4-4.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: luarocks-3.9.2-1.el9.noarch 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-0.3.21-2.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: openssl-devel-1:3.2.1-1.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: parquet-libs-9.0.0-11.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-13.14-1.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-contrib-13.14-1.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-private-libs-13.14-1.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: postgresql-server-13.14-1.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-3.14.0-13.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-05-09T09:07:24.515 INFO:teuthology.orchestra.run.smithi103.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-babel-2.9.1-2.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-argparse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ceph-common-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cephfs-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-05-09T09:07:24.516 INFO:teuthology.orchestra.run.smithi103.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-google-auth-1:2.29.0-1.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-mako-1.1.4-6.el9.noarch 2024-05-09T09:07:24.517 INFO:teuthology.orchestra.run.smithi103.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-packaging-20.9-5.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-ply-3.11-14.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-portend-3.1.0-2.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-05-09T09:07:24.518 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pycparser-2.20-6.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-pytz-2021.1-5.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rados-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-2.25.1-8.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rgw-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-routes-2.5.1-5.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-rsa-4.9-2.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-05-09T09:07:24.519 INFO:teuthology.orchestra.run.smithi103.stdout: python3-toml-0.10.2-6.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-webob-1.8.7-6.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-24.02.0-1.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: qatlib-service-24.02.0-1.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: qatzip-libs-1.2.0-1.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-fuse-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-mirror-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: rbd-nbd-2:19.0.0-2515.g96805dcf.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: re2-1:20211101-3.el9.x86_64 2024-05-09T09:07:24.520 INFO:teuthology.orchestra.run.smithi103.stdout: socat-1.7.4.1-5.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: thrift-0.15.0-2.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: uuid-1.6.2-55.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: zlib-devel-1.2.11-41.el9.x86_64 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout: 2024-05-09T09:07:24.521 INFO:teuthology.orchestra.run.smithi103.stdout:Complete! 2024-05-09T09:07:24.932 DEBUG:teuthology.parallel:result is None 2024-05-09T09:07:24.933 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:25.164 DEBUG:teuthology.orchestra.run.smithi069:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T09:07:25.193 INFO:teuthology.orchestra.run.smithi069.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.193 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.193 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T09:07:25.195 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:25.422 DEBUG:teuthology.orchestra.run.smithi103:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T09:07:25.446 INFO:teuthology.orchestra.run.smithi103.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.447 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.447 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T09:07:25.449 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:25.586 DEBUG:teuthology.orchestra.run.smithi163:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-05-09T09:07:25.612 INFO:teuthology.orchestra.run.smithi163.stdout:19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.612 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2515.g96805dcf.el9 2024-05-09T09:07:25.613 INFO:teuthology.task.install:The correct ceph version 19.0.0-2515.g96805dcf is installed. 2024-05-09T09:07:25.615 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-05-09T09:07:25.615 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:07:25.615 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T09:07:25.647 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:07:25.647 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T09:07:25.678 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:07:25.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-05-09T09:07:25.708 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-05-09T09:07:25.709 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:07:25.709 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T09:07:25.737 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T09:07:25.805 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:07:25.805 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T09:07:25.833 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T09:07:25.900 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:07:25.900 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2024-05-09T09:07:25.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-05-09T09:07:25.996 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-05-09T09:07:25.996 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:07:25.996 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T09:07:26.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T09:07:26.093 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:07:26.093 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T09:07:26.121 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T09:07:26.189 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:07:26.189 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2024-05-09T09:07:26.216 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-05-09T09:07:26.285 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-05-09T09:07:26.285 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:07:26.285 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T09:07:26.311 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T09:07:26.376 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:07:26.376 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T09:07:26.404 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T09:07:26.472 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:07:26.473 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/stdin-killer 2024-05-09T09:07:26.501 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-05-09T09:07:26.569 INFO:teuthology.run_tasks:Running task cephadm... 2024-05-09T09:07:26.670 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug client': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon down mkfs grace': 300, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'filesystem is online with fewer MDS than max_mds', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'PG_AVAILABILITY', 'PG_DEGRADED', 'Reduced data availability', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-09T09:07:26.670 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:26.671 INFO:tasks.cephadm:Cluster fsid is 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:07:26.671 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-05-09T09:07:26.671 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.69', 'mon.b': '172.21.15.103', 'mon.c': '172.21.15.163'} 2024-05-09T09:07:26.671 INFO:tasks.cephadm:First mon is mon.a on smithi069 2024-05-09T09:07:26.671 INFO:tasks.cephadm:First mgr is x 2024-05-09T09:07:26.671 INFO:tasks.cephadm:Normalizing hostnames... 2024-05-09T09:07:26.671 DEBUG:teuthology.orchestra.run.smithi069:> sudo hostname $(hostname -s) 2024-05-09T09:07:26.697 DEBUG:teuthology.orchestra.run.smithi103:> sudo hostname $(hostname -s) 2024-05-09T09:07:26.725 DEBUG:teuthology.orchestra.run.smithi163:> sudo hostname $(hostname -s) 2024-05-09T09:07:26.754 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-05-09T09:07:26.754 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:26.911 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/55097/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2515-g96805dcf', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2515.g96805dcf'}, 'url': 'https://3.chacra.ceph.com/r/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-05-07 19:00:25.721799', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'squid', 'chacra_url': 'https://3.chacra.ceph.com/repos/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/flavors/default/', 'archs': ['x86_64', 'source'], 'distro': 'centos'}] 2024-05-09T09:07:27.136 INFO:tasks.util.chacra:got chacra host 3.chacra.ceph.com, ref squid, sha1 96805dcf909541c4ca9a1e90258de8ae5b6fa1bc from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:07:27.138 INFO:tasks.cephadm:Discovered cachra url: https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm 2024-05-09T09:07:27.138 DEBUG:teuthology.orchestra.run.smithi069:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:27.524 INFO:teuthology.orchestra.run.smithi069.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 09:07 /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:27.525 DEBUG:teuthology.orchestra.run.smithi103:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:27.918 INFO:teuthology.orchestra.run.smithi103.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 09:07 /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:27.919 DEBUG:teuthology.orchestra.run.smithi163:> curl --silent -L https://3.chacra.ceph.com/binaries/ceph/squid/96805dcf909541c4ca9a1e90258de8ae5b6fa1bc/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:28.296 INFO:teuthology.orchestra.run.smithi163.stdout:-rw-r--r--. 1 ubuntu ubuntu 782039 May 9 09:07 /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:28.297 DEBUG:teuthology.orchestra.run.smithi069:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:28.316 DEBUG:teuthology.orchestra.run.smithi103:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:28.335 DEBUG:teuthology.orchestra.run.smithi163:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-05-09T09:07:28.371 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc on all hosts... 2024-05-09T09:07:28.371 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T09:07:28.373 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T09:07:28.377 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc pull 2024-05-09T09:07:29.926 INFO:teuthology.orchestra.run.smithi103.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T09:07:29.985 INFO:teuthology.orchestra.run.smithi069.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T09:07:29.991 INFO:teuthology.orchestra.run.smithi163.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout: "repo_digests": [ 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout: ] 2024-05-09T09:08:01.850 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-09T09:08:16.348 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout: "repo_digests": [ 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout: ] 2024-05-09T09:08:16.349 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "image_id": "d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662", 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "repo_digests": [ 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac" 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:08:18.936 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:08:18.960 DEBUG:teuthology.orchestra.run.smithi069:> sudo mkdir -p /etc/ceph 2024-05-09T09:08:19.013 DEBUG:teuthology.orchestra.run.smithi103:> sudo mkdir -p /etc/ceph 2024-05-09T09:08:19.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph 2024-05-09T09:08:19.078 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 777 /etc/ceph 2024-05-09T09:08:19.103 DEBUG:teuthology.orchestra.run.smithi103:> sudo chmod 777 /etc/ceph 2024-05-09T09:08:19.130 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /etc/ceph 2024-05-09T09:08:19.158 INFO:tasks.cephadm:Writing seed config... 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [client] debug client = 20 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-05-09T09:08:19.159 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-05-09T09:08:19.160 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mgr] debug client = 20 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mon] mon down mkfs grace = 300 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-05-09T09:08:19.161 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-05-09T09:08:19.162 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-05-09T09:08:19.163 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:08:19.163 DEBUG:teuthology.orchestra.run.smithi069:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-05-09T09:08:19.179 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 8e3a7b52-0de3-11ef-bc98-c7b262605968 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug client = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon down mkfs grace = 300 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-05-09T09:08:19.180 DEBUG:teuthology.orchestra.run.smithi069:mon.a> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service 2024-05-09T09:08:19.222 DEBUG:teuthology.orchestra.run.smithi069:mgr.x> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x.service 2024-05-09T09:08:19.264 INFO:tasks.cephadm:Bootstrapping... 2024-05-09T09:08:19.264 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc -v bootstrap --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.69 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:08:19.453 INFO:teuthology.orchestra.run.smithi069.stdout:-------------------------------------------------------------------------------- 2024-05-09T09:08:19.453 INFO:teuthology.orchestra.run.smithi069.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc', '-v', 'bootstrap', '--fsid', '8e3a7b52-0de3-11ef-bc98-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.69', '--skip-admin-label'] 2024-05-09T09:08:19.453 INFO:teuthology.orchestra.run.smithi069.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-05-09T09:08:19.454 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying podman|docker is present... 2024-05-09T09:08:19.479 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T09:08:19.479 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying lvm2 is present... 2024-05-09T09:08:19.479 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying time synchronization is in place... 2024-05-09T09:08:19.486 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T09:08:19.487 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T09:08:19.493 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T09:08:19.493 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:08:19.499 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2024-05-09T09:08:19.506 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2024-05-09T09:08:19.506 INFO:teuthology.orchestra.run.smithi069.stdout:Unit chronyd.service is enabled and running 2024-05-09T09:08:19.506 INFO:teuthology.orchestra.run.smithi069.stdout:Repeating the final host check... 2024-05-09T09:08:19.529 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout 4.9.4-dev 2024-05-09T09:08:19.529 INFO:teuthology.orchestra.run.smithi069.stdout:podman (/bin/podman) version 4.9.4 is present 2024-05-09T09:08:19.529 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl is present 2024-05-09T09:08:19.529 INFO:teuthology.orchestra.run.smithi069.stdout:lvcreate is present 2024-05-09T09:08:19.536 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-05-09T09:08:19.536 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-05-09T09:08:19.542 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-05-09T09:08:19.542 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:08:19.548 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout enabled 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout active 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:Unit chronyd.service is enabled and running 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:Host looks OK 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:Cluster fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:Acquiring lock 140439247319152 on /run/cephadm/8e3a7b52-0de3-11ef-bc98-c7b262605968.lock 2024-05-09T09:08:19.555 INFO:teuthology.orchestra.run.smithi069.stdout:Lock 140439247319152 acquired on /run/cephadm/8e3a7b52-0de3-11ef-bc98-c7b262605968.lock 2024-05-09T09:08:19.556 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 3300 ... 2024-05-09T09:08:19.556 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying IP 172.21.15.69 port 6789 ... 2024-05-09T09:08:19.556 INFO:teuthology.orchestra.run.smithi069.stdout:Base mon IP(s) is [172.21.15.69:3300, 172.21.15.69:6789], mon addrv is [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-05-09T09:08:19.559 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.69 metric 100 2024-05-09T09:08:19.559 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.69 metric 100 2024-05-09T09:08:19.562 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-05-09T09:08:19.563 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-05-09T09:08:19.563 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1784sec hoplimit 64 pref medium 2024-05-09T09:08:19.564 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-05-09T09:08:19.564 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:Mon IP `172.21.15.69` is in CIDR network `172.21.0.0/20` 2024-05-09T09:08:19.565 INFO:teuthology.orchestra.run.smithi069.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-05-09T09:08:19.566 INFO:teuthology.orchestra.run.smithi069.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-05-09T09:08:19.567 INFO:teuthology.orchestra.run.smithi069.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T09:08:20.047 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stdout d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T09:08:20.047 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc... 2024-05-09T09:08:20.047 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Getting image source signatures 2024-05-09T09:08:20.047 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying blob sha256:9852312f87792557815c3e860a5c4f8cbf2b59bbbde9b5f86ab07b438f905b2d 2024-05-09T09:08:20.047 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying blob sha256:bf5b95bd68251da39c6cd2eb96c6ba4138bb37a62bdc39287908b25e92047d7e 2024-05-09T09:08:20.048 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Copying config sha256:d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 2024-05-09T09:08:20.048 INFO:teuthology.orchestra.run.smithi069.stdout:/bin/podman: stderr Writing manifest to image destination 2024-05-09T09:08:20.947 INFO:teuthology.orchestra.run.smithi069.stdout:ceph: stdout ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T09:08:20.947 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph version: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev) 2024-05-09T09:08:20.947 INFO:teuthology.orchestra.run.smithi069.stdout:Extracting ceph user uid/gid from container image... 2024-05-09T09:08:21.913 INFO:teuthology.orchestra.run.smithi069.stdout:stat: stdout 167 167 2024-05-09T09:08:21.913 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial keys... 2024-05-09T09:08:22.819 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCGkjxm7Lg/DhAASfJENMgzUKlvdVRD6PMu4A== 2024-05-09T09:08:23.600 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCHkjxmejhOCBAAshM+qgLc+dmn0F0eOw4mlA== 2024-05-09T09:08:24.672 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph-authtool: stdout AQCHkjxmjyNKOhAASlIyR0P0c1iHeEPWPGVqPA== 2024-05-09T09:08:24.673 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial monmap... 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:monmaptool for a [v2:172.21.15.69:3300,v1:172.21.15.69:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-05-09T09:08:25.529 INFO:teuthology.orchestra.run.smithi069.stdout:setting min_mon_release = quincy 2024-05-09T09:08:25.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: set fsid to 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:25.530 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-05-09T09:08:25.530 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:08:25.530 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mon... 2024-05-09T09:08:26.434 INFO:teuthology.orchestra.run.smithi069.stdout:create mon.a on 2024-05-09T09:08:26.630 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-05-09T09:08:26.789 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-05-09T09:08:26.969 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target. 2024-05-09T09:08:26.969 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target → /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target. 2024-05-09T09:08:27.234 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a 2024-05-09T09:08:27.234 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service: Unit ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service not loaded. 2024-05-09T09:08:27.411 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target.wants/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service → /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@.service. 2024-05-09T09:08:27.465 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 systemd[1]: Starting Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:08:27.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 podman[29731]: 2024-05-09 09:08:27.5667812 +0000 UTC m=+0.018130052 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:08:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 podman[29731]: 2024-05-09 09:08:27.678088774 +0000 UTC m=+0.129437607 container create 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-05-09T09:08:27.958 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T09:08:27.958 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2024-05-09T09:08:27.965 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T09:08:27.966 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:08:27.966 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2024-05-09T09:08:27.966 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T09:08:27.966 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon to start... 2024-05-09T09:08:27.966 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mon... 2024-05-09T09:08:28.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 podman[29731]: 2024-05-09 09:08:27.88865214 +0000 UTC m=+0.340000986 container init 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:08:28.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 podman[29731]: 2024-05-09 09:08:27.891890535 +0000 UTC m=+0.343239377 container start 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T09:08:28.031 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 ceph-mon[29749]: mkfs 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:28.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 ceph-mon[29749]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T09:08:28.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 bash[29731]: 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce 2024-05-09T09:08:28.032 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:27 smithi069 systemd[1]: Started Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout cluster: 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout id: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout services: 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.534968s) 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-05-09T09:08:29.053 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout data: 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout pgs: 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.390+0000 7f37cb571700 1 Processor -- start 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.390+0000 7f37cb571700 1 -- start start 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.390+0000 7f37cb571700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.390+0000 7f37cb571700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f37c41087c0 con 0x7f37c4107dd0 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37c930d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37c930d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36556/0 (socket says 172.21.15.69:36556) 2024-05-09T09:08:29.054 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37c930d700 1 -- 172.21.15.69:0/1029211270 learned_addr learned my addr 172.21.15.69:0/1029211270 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37c930d700 1 -- 172.21.15.69:0/1029211270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c4108fd0 con 0x7f37c4107dd0 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37c930d700 1 --2- 172.21.15.69:0/1029211270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098a0 tx=0x7f37c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61413e5d7fc78b88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37bbfff700 1 -- 172.21.15.69:0/1029211270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c0004160 con 0x7f37c4107dd0 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37bbfff700 1 -- 172.21.15.69:0/1029211270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c0004300 con 0x7f37c4107dd0 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.391+0000 7f37bbfff700 1 -- 172.21.15.69:0/1029211270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c00036a0 con 0x7f37c4107dd0 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 -- 172.21.15.69:0/1029211270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 msgr2=0x7f37c41081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 --2- 172.21.15.69:0/1029211270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f37c00098a0 tx=0x7f37c0009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 -- 172.21.15.69:0/1029211270 shutdown_connections 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 --2- 172.21.15.69:0/1029211270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c41081f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 -- 172.21.15.69:0/1029211270 >> 172.21.15.69:0/1029211270 conn(0x7f37c41033b0 msgr2=0x7f37c41057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 -- 172.21.15.69:0/1029211270 shutdown_connections 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.392+0000 7f37cb571700 1 -- 172.21.15.69:0/1029211270 wait complete. 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37cb571700 1 Processor -- start 2024-05-09T09:08:29.055 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37cb571700 1 -- start start 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37cb571700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37cb571700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f37c4109500 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37c930d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37c930d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36560/0 (socket says 172.21.15.69:36560) 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37c930d700 1 -- 172.21.15.69:0/3739794246 learned_addr learned my addr 172.21.15.69:0/3739794246 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37c930d700 1 -- 172.21.15.69:0/3739794246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f37c407e930 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37c930d700 1 --2- 172.21.15.69:0/3739794246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c0004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c0003db0 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.393+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f37c0011630 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f37c407cb20 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f37c0011950 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f37c407d060 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f37c0011af0 con 0x7f37c4107dd0 2024-05-09T09:08:29.056 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f37c001c450 con 0x7f37c4107dd0 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.394+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f37c4063980 con 0x7f37c4107dd0 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.397+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f37c0032020 con 0x7f37c4107dd0 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.465+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status"} v 0) -- 0x7f37c407d2e0 con 0x7f37c4107dd0 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37ba7fc700 1 -- 172.21.15.69:0/3739794246 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) ==== 54+0+317 (secure 0 0 0) 0x7f37c0004260 con 0x7f37c4107dd0 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 msgr2=0x7f37c407e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 --2- 172.21.15.69:0/3739794246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f37c0000c00 tx=0x7f37c0004750 comp rx=0 tx=0).stop 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 shutdown_connections 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 --2- 172.21.15.69:0/3739794246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f37c4107dd0 0x7f37c407e3f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 >> 172.21.15.69:0/3739794246 conn(0x7f37c41033b0 msgr2=0x7f37c418f790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 shutdown_connections 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:28.466+0000 7f37cb571700 1 -- 172.21.15.69:0/3739794246 wait complete. 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:mon is available 2024-05-09T09:08:29.057 INFO:teuthology.orchestra.run.smithi069.stdout:Assimilating anything we can from ceph.conf... 2024-05-09T09:08:29.152 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: monmap epoch 1 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: last_changed 2024-05-09T09:08:24.964355+0000 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: min_mon_release 19 (squid) 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: election_strategy: 1 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: fsmap 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: mgrmap e1: no daemons active 2024-05-09T09:08:29.153 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:28 smithi069 ceph-mon[29749]: from='client.? 172.21.15.69:0/3739794246' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-05-09T09:08:30.141 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:29 smithi069 ceph-mon[29749]: from='client.? 172.21.15.69:0/231685656' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T09:08:30.142 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:29 smithi069 ceph-mon[29749]: from='client.? 172.21.15.69:0/231685656' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.69:3300,v1:172.21.15.69:6789] 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T09:08:30.151 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.464+0000 7f345e3f7700 1 Processor -- start 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.464+0000 7f345e3f7700 1 -- start start 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.464+0000 7f345e3f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.464+0000 7f345e3f7700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f345807d150 con 0x7f345807e720 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.464+0000 7f3457fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3457fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36562/0 (socket says 172.21.15.69:36562) 2024-05-09T09:08:30.152 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3457fff700 1 -- 172.21.15.69:0/1439711824 learned_addr learned my addr 172.21.15.69:0/1439711824 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3457fff700 1 -- 172.21.15.69:0/1439711824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345807d2d0 con 0x7f345807e720 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3457fff700 1 --2- 172.21.15.69:0/1439711824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34480098a0 tx=0x7f3448009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af0e712cfe197ec3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3456ffd700 1 -- 172.21.15.69:0/1439711824 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3448004160 con 0x7f345807e720 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3456ffd700 1 -- 172.21.15.69:0/1439711824 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3448004300 con 0x7f345807e720 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.465+0000 7f3456ffd700 1 -- 172.21.15.69:0/1439711824 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f34480036a0 con 0x7f345807e720 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- 172.21.15.69:0/1439711824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 msgr2=0x7f345807cb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 --2- 172.21.15.69:0/1439711824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f34480098a0 tx=0x7f3448009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- 172.21.15.69:0/1439711824 shutdown_connections 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 --2- 172.21.15.69:0/1439711824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345807cb80 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- 172.21.15.69:0/1439711824 >> 172.21.15.69:0/1439711824 conn(0x7f34581033b0 msgr2=0x7f34581057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- 172.21.15.69:0/1439711824 shutdown_connections 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- 172.21.15.69:0/1439711824 wait complete. 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 Processor -- start 2024-05-09T09:08:30.153 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.466+0000 7f345e3f7700 1 -- start start 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f345e3f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f345e3f7700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f345807d800 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f3457fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f3457fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36572/0 (socket says 172.21.15.69:36572) 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f3457fff700 1 -- 172.21.15.69:0/231685656 learned_addr learned my addr 172.21.15.69:0/231685656 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f3457fff700 1 -- 172.21.15.69:0/231685656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f345819dbb0 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f3457fff700 1 --2- 172.21.15.69:0/231685656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3448000c00 tx=0x7f3448004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3448003db0 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(0 keys) ==== 4+0+0 (secure 0 0 0) 0x7f3448011630 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f345819de40 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3448011950 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34581a0b30 con 0x7f345807e720 2024-05-09T09:08:30.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.467+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3448011af0 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.468+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f344801c450 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.468+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3458063980 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.470+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3448032020 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.537+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f345802dd00 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.540+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) ==== 70+0+429 (secure 0 0 0) 0x7f3448004260 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.541+0000 7f34557fa700 1 -- 172.21.15.69:0/231685656 <== mon.0 v2:172.21.15.69:3300/0 8 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f344801cb60 con 0x7f345807e720 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.541+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 msgr2=0x7f345819d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.541+0000 7f345e3f7700 1 --2- 172.21.15.69:0/231685656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3448000c00 tx=0x7f3448004750 comp rx=0 tx=0).stop 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.542+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 shutdown_connections 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.542+0000 7f345e3f7700 1 --2- 172.21.15.69:0/231685656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f345807e720 0x7f345819d670 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.542+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 >> 172.21.15.69:0/231685656 conn(0x7f34581033b0 msgr2=0x7f34581056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.542+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 shutdown_connections 2024-05-09T09:08:30.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:29.542+0000 7f345e3f7700 1 -- 172.21.15.69:0/231685656 wait complete. 2024-05-09T09:08:30.156 INFO:teuthology.orchestra.run.smithi069.stdout:Generating new minimal ceph.conf... 2024-05-09T09:08:32.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:31 smithi069 ceph-mon[29749]: from='client.? 172.21.15.69:0/3974215961' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:08:37.880 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.658+0000 7fddd2f1a700 1 Processor -- start 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.658+0000 7fddd2f1a700 1 -- start start 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.658+0000 7fddd2f1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.658+0000 7fddd2f1a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc07c6f0 con 0x7fddcc108810 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddd0cb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddd0cb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36580/0 (socket says 172.21.15.69:36580) 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddd0cb6700 1 -- 172.21.15.69:0/926261413 learned_addr learned my addr 172.21.15.69:0/926261413 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddd0cb6700 1 -- 172.21.15.69:0/926261413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc10b140 con 0x7fddcc108810 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddd0cb6700 1 --2- 172.21.15.69:0/926261413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fddc80098a0 tx=0x7fddc8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a4a871d7cafe61e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddc37fe700 1 -- 172.21.15.69:0/926261413 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddc8004160 con 0x7fddcc108810 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddc37fe700 1 -- 172.21.15.69:0/926261413 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fddc8004300 con 0x7fddcc108810 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.659+0000 7fddc37fe700 1 -- 172.21.15.69:0/926261413 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddc8011b50 con 0x7fddcc108810 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- 172.21.15.69:0/926261413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 msgr2=0x7fddcc10ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:37.881 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 --2- 172.21.15.69:0/926261413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fddc80098a0 tx=0x7fddc8009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- 172.21.15.69:0/926261413 shutdown_connections 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 --2- 172.21.15.69:0/926261413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc10ac00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- 172.21.15.69:0/926261413 >> 172.21.15.69:0/926261413 conn(0x7fddcc102200 msgr2=0x7fddcc104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- 172.21.15.69:0/926261413 shutdown_connections 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- 172.21.15.69:0/926261413 wait complete. 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 Processor -- start 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.660+0000 7fddd2f1a700 1 -- start start 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd2f1a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd2f1a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fddcc10c0f0 con 0x7fddcc108810 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd0cb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd0cb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36588/0 (socket says 172.21.15.69:36588) 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd0cb6700 1 -- 172.21.15.69:0/3974215961 learned_addr learned my addr 172.21.15.69:0/3974215961 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd0cb6700 1 -- 172.21.15.69:0/3974215961 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fddcc19c260 con 0x7fddcc108810 2024-05-09T09:08:37.882 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd0cb6700 1 --2- 172.21.15.69:0/3974215961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fddc8011670 tx=0x7fddc8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddc80039c0 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fddc8011e50 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fddcc19c4f0 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fddc801a5c0 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.661+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fddcc19f180 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.662+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fddc801ac00 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.662+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fddc8019700 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.662+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fddcc063980 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.665+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fddc8003b60 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.731+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7fddcc19f350 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.731+0000 7fddc1ffb700 1 -- 172.21.15.69:0/3974215961 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) ==== 76+0+175 (secure 0 0 0) 0x7fddc80199a0 con 0x7fddcc108810 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 msgr2=0x7fddcc19bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 --2- 172.21.15.69:0/3974215961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fddc8011670 tx=0x7fddc8011b60 comp rx=0 tx=0).stop 2024-05-09T09:08:37.883 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 shutdown_connections 2024-05-09T09:08:37.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 --2- 172.21.15.69:0/3974215961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fddcc108810 0x7fddcc19bd20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:37.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 >> 172.21.15.69:0/3974215961 conn(0x7fddcc102200 msgr2=0x7fddcc102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:37.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 shutdown_connections 2024-05-09T09:08:37.884 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:31.732+0000 7fddd2f1a700 1 -- 172.21.15.69:0/3974215961 wait complete. 2024-05-09T09:08:37.884 INFO:teuthology.orchestra.run.smithi069.stdout:Restarting the monitor... 2024-05-09T09:08:38.131 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:37 smithi069 systemd[1]: Stopping Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:08:39.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:39 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[29745]: 2024-05-09T09:08:39.005+0000 7fdd086ff700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:08:39.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:39 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[29745]: 2024-05-09T09:08:39.005+0000 7fdd086ff700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-05-09T09:08:39.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:39 smithi069 podman[29994]: 2024-05-09 09:08:39.057117022 +0000 UTC m=+0.957013661 container died 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True) 2024-05-09T09:08:40.662 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 podman[29994]: 2024-05-09 09:08:40.438820841 +0000 UTC m=+2.338717485 container cleanup 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T09:08:40.663 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 bash[29994]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a 2024-05-09T09:08:40.996 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 podman[30007]: 2024-05-09 09:08:40.664593218 +0000 UTC m=+1.610993445 container remove 9b2dceb92be2e9066608a1db603b6d913481efe4f23cb8262a0f15f61ecd09ce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-09T09:08:40.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service: Deactivated successfully. 2024-05-09T09:08:40.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 systemd[1]: Stopped Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:08:40.997 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 systemd[1]: Starting Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:08:41.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:40 smithi069 podman[30104]: 2024-05-09 09:08:40.998703533 +0000 UTC m=+0.109515628 container create 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:08:41.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 podman[30104]: 2024-05-09 09:08:40.906824031 +0000 UTC m=+0.017636091 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 podman[30104]: 2024-05-09 09:08:41.209574638 +0000 UTC m=+0.320386692 container init 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 podman[30104]: 2024-05-09 09:08:41.212550042 +0000 UTC m=+0.323362096 container start 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: pidfile_write: ignore empty --pid-file 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: load: jerasure load: lrc 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: RocksDB version: 7.9.2 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Git sha 0 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: DB SUMMARY 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: DB Session ID: W145KUU5OG95S1GU0T6L 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: CURRENT file: CURRENT 2024-05-09T09:08:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T09:08:41.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-05-09T09:08:41.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-05-09T09:08:41.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 104832 ; 2024-05-09T09:08:41.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.error_if_exists: 0 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.create_if_missing: 0 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.paranoid_checks: 1 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T09:08:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.env: 0x55f4f036cc60 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.info_log: 0x55f4f1ebbac0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.statistics: (nil) 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.use_fsync: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_log_file_size: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_fallocate: 1 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T09:08:41.263 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.use_direct_reads: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.db_log_dir: 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.wal_dir: 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.write_buffer_manager: 0x55f4f1e8e1e0 2024-05-09T09:08:41.264 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.unordered_write: 0 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.row_cache: None 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.wal_filter: None 2024-05-09T09:08:41.265 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.two_write_queues: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.wal_compression: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.atomic_flush: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.log_readahead_size: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T09:08:41.266 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_background_jobs: 2 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_background_compactions: -1 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_subcompactions: 1 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_open_files: -1 2024-05-09T09:08:41.267 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_background_flushes: -1 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Compression algorithms supported: 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kZSTD supported: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kXpressCompression supported: 0 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kZlibCompression supported: 1 2024-05-09T09:08:41.268 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kSnappyCompression supported: 1 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kLZ4Compression supported: 1 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: kBZip2Compression supported: 0 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.merge_operator: 2024-05-09T09:08:41.269 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_filter: None 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55f4f1ebbbc0) 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks: 1 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T09:08:41.270 INFO:journalctl@ceph.mon.a.smithi069.stdout: pin_top_level_index_and_filter: 1 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_type: 0 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_index_type: 0 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_shortening: 1 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: checksum: 4 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: no_block_cache: 0 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache: 0x55f4f1ebf090 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_name: BinnedLRUCache 2024-05-09T09:08:41.271 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_options: 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: capacity : 536870912 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_shard_bits : 4 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: strict_capacity_limit : 0 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: high_pri_pool_ratio: 0.000 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_cache_compressed: (nil) 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: persistent_cache: (nil) 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size: 4096 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_size_deviation: 10 2024-05-09T09:08:41.272 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_restart_interval: 16 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: index_block_restart_interval: 1 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: metadata_block_size: 4096 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: partition_filters: 0 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: use_delta_encoding: 1 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: filter_policy: bloomfilter 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: whole_key_filtering: 1 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: verify_compression: 0 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: read_amp_bytes_per_bit: 0 2024-05-09T09:08:41.273 INFO:journalctl@ceph.mon.a.smithi069.stdout: format_version: 5 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_index_compression: 1 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: block_align: 0 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: max_auto_readahead_size: 262144 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: prepopulate_block_cache: 0 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: initial_auto_readahead_size: 8192 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T09:08:41.274 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression: NoCompression 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.num_levels: 7 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T09:08:41.275 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T09:08:41.276 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T09:08:41.277 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T09:08:41.278 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T09:08:41.279 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T09:08:41.280 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.table_properties_collectors: 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.inplace_update_support: 0 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T09:08:41.281 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.bloom_locality: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.max_successive_merges: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T09:08:41.282 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.ttl: 2592000 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enable_blob_files: false 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.min_blob_size: 0 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T09:08:41.283 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 81d0bb3e-b9cb-4ad3-9205-7522e941eea2 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245721241600, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245721242562, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 99942, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 326, "table_properties": {"data_size": 97853, "index_size": 307, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 773, "raw_key_size": 14187, "raw_average_key_size": 49, "raw_value_size": 90150, "raw_average_value_size": 316, "num_data_blocks": 12, "num_entries": 285, "num_filter_entries": 285, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715245721, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "81d0bb3e-b9cb-4ad3-9205-7522e941eea2", "db_session_id": "W145KUU5OG95S1GU0T6L", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245721242644, "job": 1, "event": "recovery_finished"} 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-05-09T09:08:41.284 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55f4f1fb0000 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: DB pointer 0x55f4f1f9a000 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** DB Stats ** 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-05-09T09:08:41.285 INFO:journalctl@ceph.mon.a.smithi069.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: L0 2/0 99.43 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Sum 2/0 99.43 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** Compaction Stats [default] ** 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.2 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T09:08:41.286 INFO:journalctl@ceph.mon.a.smithi069.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: Cumulative compaction: 0.00 GB write, 32.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: Interval compaction: 0.00 GB write, 32.22 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache BinnedLRUCache@0x55f4f1ebf090#2 capacity: 512.00 MB usage: 1.52 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,1.02 KB,0.000193715%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: starting mon.a rank 0 at public addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] at bind addrs [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???) e1 preinit fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).mds e1 new map 2024-05-09T09:08:41.287 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).mds e1 print_map 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: e1 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: legacy client fscid: -1 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout: No filesystems configured 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:08:41.288 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-05-09T09:08:41.289 INFO:teuthology.orchestra.run.smithi069.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-05-09T09:08:41.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 bash[30104]: 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 2024-05-09T09:08:41.602 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 systemd[1]: Started Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mon.a is new leader, mons a in quorum (ranks 0) 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: monmap epoch 1 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: last_changed 2024-05-09T09:08:24.964355+0000 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: min_mon_release 19 (squid) 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: election_strategy: 1 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: fsmap 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: osdmap e1: 0 total, 0 up, 0 in 2024-05-09T09:08:41.603 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:41 smithi069 ceph-mon[30123]: mgrmap e1: no daemons active 2024-05-09T09:08:42.345 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.705+0000 7fb747b7d700 1 Processor -- start 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.705+0000 7fb747b7d700 1 -- start start 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.705+0000 7fb747b7d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.705+0000 7fb747b7d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb7401087c0 con 0x7fb740107dd0 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb745919700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb745919700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47618/0 (socket says 172.21.15.69:47618) 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb745919700 1 -- 172.21.15.69:0/3954953491 learned_addr learned my addr 172.21.15.69:0/3954953491 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb745919700 1 -- 172.21.15.69:0/3954953491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb740108fd0 con 0x7fb740107dd0 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb745919700 1 --2- 172.21.15.69:0/3954953491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5534eaaeb59fbcc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb744917700 1 -- 172.21.15.69:0/3954953491 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c004160 con 0x7fb740107dd0 2024-05-09T09:08:42.346 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb744917700 1 -- 172.21.15.69:0/3954953491 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb73c004300 con 0x7fb740107dd0 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.706+0000 7fb744917700 1 -- 172.21.15.69:0/3954953491 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c011b50 con 0x7fb740107dd0 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 -- 172.21.15.69:0/3954953491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 msgr2=0x7fb7401081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 --2- 172.21.15.69:0/3954953491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7fb73c0098a0 tx=0x7fb73c009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 -- 172.21.15.69:0/3954953491 shutdown_connections 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 --2- 172.21.15.69:0/3954953491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb7401081f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 -- 172.21.15.69:0/3954953491 >> 172.21.15.69:0/3954953491 conn(0x7fb7401033b0 msgr2=0x7fb7401057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 -- 172.21.15.69:0/3954953491 shutdown_connections 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.707+0000 7fb747b7d700 1 -- 172.21.15.69:0/3954953491 wait complete. 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb747b7d700 1 Processor -- start 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb747b7d700 1 -- start start 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb747b7d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb747b7d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb74010c380 con 0x7fb740107dd0 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb745919700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:42.347 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb745919700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47624/0 (socket says 172.21.15.69:47624) 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb745919700 1 -- 172.21.15.69:0/1693470964 learned_addr learned my addr 172.21.15.69:0/1693470964 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb745919700 1 -- 172.21.15.69:0/1693470964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb74019c270 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb745919700 1 --2- 172.21.15.69:0/1693470964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c003b20 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.708+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb73c003cc0 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb74019c500 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb73c024910 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb74019f1f0 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fb73c022020 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb73c011e00 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.709+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb740063980 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.712+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fb73c042dd0 con 0x7fb740107dd0 2024-05-09T09:08:42.348 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.777+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) -- 0x7fb74019f3c0 con 0x7fb740107dd0 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.782+0000 7fb736ffd700 1 -- 172.21.15.69:0/1693470964 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) ==== 124+0+0 (secure 0 0 0) 0x7fb73c031020 con 0x7fb740107dd0 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.782+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 msgr2=0x7fb74019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.782+0000 7fb747b7d700 1 --2- 172.21.15.69:0/1693470964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb73c000c00 tx=0x7fb73c011c10 comp rx=0 tx=0).stop 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.783+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 shutdown_connections 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.783+0000 7fb747b7d700 1 --2- 172.21.15.69:0/1693470964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb740107dd0 0x7fb74019bd30 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.783+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 >> 172.21.15.69:0/1693470964 conn(0x7fb7401033b0 msgr2=0x7fb740103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.783+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 shutdown_connections 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:41.783+0000 7fb747b7d700 1 -- 172.21.15.69:0/1693470964 wait complete. 2024-05-09T09:08:42.349 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote config to /etc/ceph/ceph.conf 2024-05-09T09:08:42.372 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:08:42.372 INFO:teuthology.orchestra.run.smithi069.stdout:Creating mgr... 2024-05-09T09:08:42.372 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 0.0.0.0:9283 ... 2024-05-09T09:08:42.373 INFO:teuthology.orchestra.run.smithi069.stdout:Verifying port 0.0.0.0:8765 ... 2024-05-09T09:08:42.628 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x 2024-05-09T09:08:42.628 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Failed to reset failed state of unit ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x.service: Unit ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x.service not loaded. 2024-05-09T09:08:42.772 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968.target.wants/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x.service → /etc/systemd/system/ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@.service. 2024-05-09T09:08:43.008 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:42 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1693470964' entity='client.admin' 2024-05-09T09:08:43.008 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:42 smithi069 systemd[1]: Starting Ceph mgr.x for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:08:43.260 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 podman[30337]: 2024-05-09 09:08:43.009798351 +0000 UTC m=+0.094361812 container create 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T09:08:43.260 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 podman[30337]: 2024-05-09 09:08:42.933106595 +0000 UTC m=+0.017670028 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc 2024-05-09T09:08:43.260 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 podman[30337]: 2024-05-09 09:08:43.195475966 +0000 UTC m=+0.280039390 container init 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_BRANCH=HEAD) 2024-05-09T09:08:43.260 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 podman[30337]: 2024-05-09 09:08:43.198166241 +0000 UTC m=+0.282729661 container start 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=) 2024-05-09T09:08:43.301 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T09:08:43.301 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2024-05-09T09:08:43.309 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T09:08:43.309 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:08:43.309 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2024-05-09T09:08:43.309 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to enable service . firewalld.service is not available 2024-05-09T09:08:43.317 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T09:08:43.318 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2024-05-09T09:08:43.325 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T09:08:43.325 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:08:43.325 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2024-05-09T09:08:43.325 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-05-09T09:08:43.326 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr to start... 2024-05-09T09:08:43.326 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr... 2024-05-09T09:08:43.539 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 bash[30337]: 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 2024-05-09T09:08:43.540 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 systemd[1]: Started Ceph mgr.x for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:08:43.540 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:43.350+0000 7f4b9c6fb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T09:08:43.540 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:43.540+0000 7f4b9c6fb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:08:43.841 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:43.645+0000 7f4b9c6fb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T09:08:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:43 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2202109109' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T09:08:44.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:43.967+0000 7f4b9c6fb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T09:08:44.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:44 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:44.052+0000 7f4b9c6fb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T09:08:44.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:44.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:08:44.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:08:44.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T09:08:44.484 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-05-09T09:08:44.485 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.486 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.487 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T09:08:27.932718+0000", 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:44.488 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.765+0000 7f394c422700 1 Processor -- start 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.765+0000 7f394c422700 1 -- start start 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.765+0000 7f394c422700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.765+0000 7f394c422700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39441076b0 con 0x7f3944106cc0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f394a1be700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f394a1be700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47646/0 (socket says 172.21.15.69:47646) 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f394a1be700 1 -- 172.21.15.69:0/2569583533 learned_addr learned my addr 172.21.15.69:0/2569583533 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f394a1be700 1 -- 172.21.15.69:0/2569583533 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3944107ec0 con 0x7f3944106cc0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f394a1be700 1 --2- 172.21.15.69:0/2569583533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f39400098a0 tx=0x7f3940009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cd496b7de95817a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f39491bc700 1 -- 172.21.15.69:0/2569583533 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3940004160 con 0x7f3944106cc0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f39491bc700 1 -- 172.21.15.69:0/2569583533 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3940004300 con 0x7f3944106cc0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.766+0000 7f39491bc700 1 -- 172.21.15.69:0/2569583533 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3940011b50 con 0x7f3944106cc0 2024-05-09T09:08:44.489 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 -- 172.21.15.69:0/2569583533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 msgr2=0x7f39441070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 --2- 172.21.15.69:0/2569583533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f39400098a0 tx=0x7f3940009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 -- 172.21.15.69:0/2569583533 shutdown_connections 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 --2- 172.21.15.69:0/2569583533 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f39441070e0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 -- 172.21.15.69:0/2569583533 >> 172.21.15.69:0/2569583533 conn(0x7f39441022c0 msgr2=0x7f39441046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 -- 172.21.15.69:0/2569583533 shutdown_connections 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.767+0000 7f394c422700 1 -- 172.21.15.69:0/2569583533 wait complete. 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394c422700 1 Processor -- start 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394c422700 1 -- start start 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394c422700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394c422700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f394410b270 con 0x7f3944106cc0 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394a1be700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394a1be700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47658/0 (socket says 172.21.15.69:47658) 2024-05-09T09:08:44.490 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394a1be700 1 -- 172.21.15.69:0/2202109109 learned_addr learned my addr 172.21.15.69:0/2202109109 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.768+0000 7f394a1be700 1 -- 172.21.15.69:0/2202109109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f394419c8d0 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f394a1be700 1 --2- 172.21.15.69:0/2202109109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3940011850 tx=0x7f3940011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3940003af0 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f3940020070 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f39441a0ee0 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f39400195a0 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f39441a1280 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f3940019be0 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.769+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f3940011e00 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.770+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3944063980 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.772+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f3940003c90 con 0x7f3944106cc0 2024-05-09T09:08:44.491 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.842+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f39441a1660 con 0x7f3944106cc0 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.842+0000 7f393b7fe700 1 -- 172.21.15.69:0/2202109109 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f3940025460 con 0x7f3944106cc0 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.843+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 msgr2=0x7f394407c2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 --2- 172.21.15.69:0/2202109109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3940011850 tx=0x7f3940011c10 comp rx=0 tx=0).stop 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 shutdown_connections 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 --2- 172.21.15.69:0/2202109109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3944106cc0 0x7f394407c2d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 >> 172.21.15.69:0/2202109109 conn(0x7f39441022c0 msgr2=0x7f3944102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 shutdown_connections 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:43.844+0000 7f394c422700 1 -- 172.21.15.69:0/2202109109 wait complete. 2024-05-09T09:08:44.492 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (1/15)... 2024-05-09T09:08:45.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:45.240+0000 7f4b9c6fb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T09:08:45.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:45.412+0000 7f4b9c6fb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T09:08:45.867 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:45.582+0000 7f4b9c6fb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T09:08:45.867 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:45.868+0000 7f4b9c6fb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T09:08:46.158 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:45.958+0000 7f4b9c6fb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T09:08:46.510 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:46.157+0000 7f4b9c6fb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T09:08:46.510 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:46.347+0000 7f4b9c6fb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:08:47.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2321948298' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T09:08:47.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:46.958+0000 7f4b9c6fb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T09:08:47.602 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:47.393+0000 7f4b9c6fb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T09:08:47.602 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:47.490+0000 7f4b9c6fb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T09:08:47.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:47.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:08:47.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:08:47.606 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T09:08:47.625 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T09:08:47.626 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T09:08:47.627 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T09:08:47.628 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T09:08:27.932718+0000", 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:08:47.629 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe22a632700 1 Processor -- start 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe22a632700 1 -- start start 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe22a632700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe22a632700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe22407c670 con 0x7fe2241088f0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe223fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe223fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47674/0 (socket says 172.21.15.69:47674) 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe223fff700 1 -- 172.21.15.69:0/4036185131 learned_addr learned my addr 172.21.15.69:0/4036185131 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.895+0000 7fe223fff700 1 -- 172.21.15.69:0/4036185131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22410b220 con 0x7fe2241088f0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.896+0000 7fe223fff700 1 --2- 172.21.15.69:0/4036185131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe2140098a0 tx=0x7fe214009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e533401e7aa3e18e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.896+0000 7fe222ffd700 1 -- 172.21.15.69:0/4036185131 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe214004160 con 0x7fe2241088f0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.896+0000 7fe222ffd700 1 -- 172.21.15.69:0/4036185131 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe214004300 con 0x7fe2241088f0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.896+0000 7fe222ffd700 1 -- 172.21.15.69:0/4036185131 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe214011b50 con 0x7fe2241088f0 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.896+0000 7fe22a632700 1 -- 172.21.15.69:0/4036185131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 msgr2=0x7fe22410ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 --2- 172.21.15.69:0/4036185131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe2140098a0 tx=0x7fe214009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:47.630 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 -- 172.21.15.69:0/4036185131 shutdown_connections 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 --2- 172.21.15.69:0/4036185131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22410ace0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 -- 172.21.15.69:0/4036185131 >> 172.21.15.69:0/4036185131 conn(0x7fe2241022c0 msgr2=0x7fe2241046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 -- 172.21.15.69:0/4036185131 shutdown_connections 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 -- 172.21.15.69:0/4036185131 wait complete. 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 Processor -- start 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.897+0000 7fe22a632700 1 -- start start 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe22a632700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe22a632700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe22410c120 con 0x7fe2241088f0 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe223fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe223fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47678/0 (socket says 172.21.15.69:47678) 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe223fff700 1 -- 172.21.15.69:0/2321948298 learned_addr learned my addr 172.21.15.69:0/2321948298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe223fff700 1 -- 172.21.15.69:0/2321948298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe22419c260 con 0x7fe2241088f0 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe223fff700 1 --2- 172.21.15.69:0/2321948298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe214009fb0 tx=0x7fe214011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe2140038b0 con 0x7fe2241088f0 2024-05-09T09:08:47.631 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe214011e50 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe214019420 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe22419c4f0 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.898+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe22419f1d0 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.899+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7fe214019a60 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.899+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fe214024730 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.899+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe224063980 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.901+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7fe214043dd0 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.974+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fe22419f3a0 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.974+0000 7fe2217fa700 1 -- 172.21.15.69:0/2321948298 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fe214032030 con 0x7fe2241088f0 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.975+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 msgr2=0x7fe22419bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.975+0000 7fe22a632700 1 --2- 172.21.15.69:0/2321948298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fe214009fb0 tx=0x7fe214011bf0 comp rx=0 tx=0).stop 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.976+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 shutdown_connections 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.976+0000 7fe22a632700 1 --2- 172.21.15.69:0/2321948298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe2241088f0 0x7fe22419bd20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:47.632 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.976+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 >> 172.21.15.69:0/2321948298 conn(0x7fe2241022c0 msgr2=0x7fe224102ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:47.633 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.976+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 shutdown_connections 2024-05-09T09:08:47.633 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:46.976+0000 7fe22a632700 1 -- 172.21.15.69:0/2321948298 wait complete. 2024-05-09T09:08:47.633 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (2/15)... 2024-05-09T09:08:48.010 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:47.695+0000 7f4b9c6fb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T09:08:48.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.215+0000 7f4b9c6fb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T09:08:48.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.309+0000 7f4b9c6fb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T09:08:48.802 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.562+0000 7f4b9c6fb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T09:08:48.802 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.702+0000 7f4b9c6fb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T09:08:48.803 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.803+0000 7f4b9c6fb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T09:08:49.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:48.923+0000 7f4b9c6fb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T09:08:49.510 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:49.260+0000 7f4b9c6fb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T09:08:50.117 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:49.896+0000 7f4b9c6fb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T09:08:50.117 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:50 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:50.118+0000 7f4b9c6fb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: Activating manager daemon x 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: mgrmap e2: x(active, starting, since 0.00310808s) 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T09:08:50.459 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: Manager daemon x is now available 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='mgr.14100 172.21.15.69:0/199998928' entity='mgr.x' 2024-05-09T09:08:50.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1187617985' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T09:08:50.681 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T09:08:50.682 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.683 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T09:08:50.684 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:50.684 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T09:08:50.684 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T09:08:50.684 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T09:08:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T09:08:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T09:08:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T09:08:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.692 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T09:08:50.693 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": false, 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T09:08:27.932718+0000", 2024-05-09T09:08:50.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.074+0000 7f25df0cb700 1 Processor -- start 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.074+0000 7f25df0cb700 1 -- start start 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25df0cb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25df0cb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d81076b0 con 0x7f25d8106cc0 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25dce67700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25dce67700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57764/0 (socket says 172.21.15.69:57764) 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25dce67700 1 -- 172.21.15.69:0/3008281310 learned_addr learned my addr 172.21.15.69:0/3008281310 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25dce67700 1 -- 172.21.15.69:0/3008281310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d8107ec0 con 0x7f25d8106cc0 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.075+0000 7f25dce67700 1 --2- 172.21.15.69:0/3008281310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0098a0 tx=0x7f25cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a9240714d40c441c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25d77fe700 1 -- 172.21.15.69:0/3008281310 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25cc004160 con 0x7f25d8106cc0 2024-05-09T09:08:50.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25d77fe700 1 -- 172.21.15.69:0/3008281310 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f25cc004300 con 0x7f25d8106cc0 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25d77fe700 1 -- 172.21.15.69:0/3008281310 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25cc011b50 con 0x7f25d8106cc0 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 -- 172.21.15.69:0/3008281310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 msgr2=0x7f25d81070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 --2- 172.21.15.69:0/3008281310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f25cc0098a0 tx=0x7f25cc009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 -- 172.21.15.69:0/3008281310 shutdown_connections 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 --2- 172.21.15.69:0/3008281310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d81070e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 -- 172.21.15.69:0/3008281310 >> 172.21.15.69:0/3008281310 conn(0x7f25d81022c0 msgr2=0x7f25d81046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 -- 172.21.15.69:0/3008281310 shutdown_connections 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.076+0000 7f25df0cb700 1 -- 172.21.15.69:0/3008281310 wait complete. 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.077+0000 7f25df0cb700 1 Processor -- start 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.077+0000 7f25df0cb700 1 -- start start 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.077+0000 7f25df0cb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.077+0000 7f25df0cb700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d810b270 con 0x7f25d8106cc0 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25dce67700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:50.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25dce67700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57780/0 (socket says 172.21.15.69:57780) 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25dce67700 1 -- 172.21.15.69:0/1187617985 learned_addr learned my addr 172.21.15.69:0/1187617985 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25dce67700 1 -- 172.21.15.69:0/1187617985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d819b1d0 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25dce67700 1 --2- 172.21.15.69:0/1187617985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f25cc011850 tx=0x7f25cc011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25cc003af0 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f25cc020070 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d819b460 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f25cc0195a0 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.078+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d819e150 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.079+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 1) ==== 807+0+0 (secure 0 0 0) 0x7f25cc019c30 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.079+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f25cc011e00 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.079+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d8063980 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.082+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+73141 (secure 0 0 0) 0x7f25cc003c90 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.121+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 2) ==== 46043+0+0 (secure 0 0 0) 0x7f25cc00b7c0 con 0x7f25d8106cc0 2024-05-09T09:08:50.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.158+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7f25d819e320 con 0x7f25d8106cc0 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.158+0000 7f25d5ffb700 1 -- 172.21.15.69:0/1187617985 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7f25cc02e3e0 con 0x7f25d8106cc0 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 msgr2=0x7f25d819ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 --2- 172.21.15.69:0/1187617985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f25cc011850 tx=0x7f25cc011c10 comp rx=0 tx=0).stop 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 shutdown_connections 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 --2- 172.21.15.69:0/1187617985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d8106cc0 0x7f25d819ac90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 >> 172.21.15.69:0/1187617985 conn(0x7f25d81022c0 msgr2=0x7f25d8102e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 shutdown_connections 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:50.159+0000 7f25df0cb700 1 -- 172.21.15.69:0/1187617985 wait complete. 2024-05-09T09:08:50.698 INFO:teuthology.orchestra.run.smithi069.stdout:mgr not available, waiting (3/15)... 2024-05-09T09:08:52.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:52 smithi069 ceph-mon[30123]: mgrmap e3: x(active, since 1.00675s) 2024-05-09T09:08:53.328 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:53 smithi069 ceph-mon[30123]: mgrmap e4: x(active, since 2s) 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "health": { 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-05-09T09:08:53.904 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 0 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "a" 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "monmap": { 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-05-09T09:08:53.905 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-05-09T09:08:53.906 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-05-09T09:08:53.914 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-05-09T09:08:53.914 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-05-09T09:08:53.914 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-05-09T09:08:53.914 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-05-09T09:08:53.915 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modules": [ 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "iostat", 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "nfs", 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "restful" 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ], 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "modified": "2024-05-09T09:08:27.932718+0000", 2024-05-09T09:08:53.916 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "services": {} 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout }, 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.042+0000 7fb13d2e2700 1 Processor -- start 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.042+0000 7fb13d2e2700 1 -- start start 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb13d2e2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb13d2e2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb13807d190 con 0x7fb13807e760 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57862/0 (socket says 172.21.15.69:57862) 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb136ffd700 1 -- 172.21.15.69:0/1038278797 learned_addr learned my addr 172.21.15.69:0/1038278797 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb136ffd700 1 -- 172.21.15.69:0/1038278797 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13807d310 con 0x7fb13807e760 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb136ffd700 1 --2- 172.21.15.69:0/1038278797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098a0 tx=0x7fb12c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3708d8586f448ed5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb135ffb700 1 -- 172.21.15.69:0/1038278797 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb12c004160 con 0x7fb13807e760 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb135ffb700 1 -- 172.21.15.69:0/1038278797 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb12c004300 con 0x7fb13807e760 2024-05-09T09:08:53.917 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.043+0000 7fb135ffb700 1 -- 172.21.15.69:0/1038278797 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb12c011b50 con 0x7fb13807e760 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 -- 172.21.15.69:0/1038278797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 msgr2=0x7fb13807cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/1038278797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb12c0098a0 tx=0x7fb12c009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 -- 172.21.15.69:0/1038278797 shutdown_connections 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/1038278797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13807cbc0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 -- 172.21.15.69:0/1038278797 >> 172.21.15.69:0/1038278797 conn(0x7fb1381033b0 msgr2=0x7fb1381057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 -- 172.21.15.69:0/1038278797 shutdown_connections 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.044+0000 7fb13d2e2700 1 -- 172.21.15.69:0/1038278797 wait complete. 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb13d2e2700 1 Processor -- start 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb13d2e2700 1 -- start start 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb13d2e2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb13d2e2700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb13810e370 con 0x7fb13807e760 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb136ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57866/0 (socket says 172.21.15.69:57866) 2024-05-09T09:08:53.918 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.045+0000 7fb136ffd700 1 -- 172.21.15.69:0/2739729524 learned_addr learned my addr 172.21.15.69:0/2739729524 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb136ffd700 1 -- 172.21.15.69:0/2739729524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb13819e4e0 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb136ffd700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000c00 tx=0x7fb12c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb12c003b20 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fb12c003cc0 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb13819e770 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb12c0248a0 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.046+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1381a1460 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.047+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7fb12c022020 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.047+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb138063980 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.047+0000 7fb11ffff700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 0x7fb12003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.047+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7fb12c011e00 con 0x7fb13807e760 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.047+0000 7fb1367fc700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 0x7fb12003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.048+0000 7fb1367fc700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 0x7fb12003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb128006fd0 tx=0x7fb128006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:53.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.054+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb12c02cda0 con 0x7fb13807e760 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.328+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) -- 0x7fb1381a1630 con 0x7fb13807e760 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.329+0000 7fb11ffff700 1 -- 172.21.15.69:0/2739729524 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) ==== 79+0+1239 (secure 0 0 0) 0x7fb1381a1630 con 0x7fb13807e760 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 msgr2=0x7fb12003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 0x7fb12003b510 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fb128006fd0 tx=0x7fb128006e40 comp rx=0 tx=0).stop 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 msgr2=0x7fb13819dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fb12c000c00 tx=0x7fb12c011c10 comp rx=0 tx=0).stop 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 shutdown_connections 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7fb120039050 0x7fb12003b510 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 --2- 172.21.15.69:0/2739729524 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb13807e760 0x7fb13819dfa0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 >> 172.21.15.69:0/2739729524 conn(0x7fb1381033b0 msgr2=0x7fb138103f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 shutdown_connections 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:53.332+0000 7fb13d2e2700 1 -- 172.21.15.69:0/2739729524 wait complete. 2024-05-09T09:08:53.920 INFO:teuthology.orchestra.run.smithi069.stdout:mgr is available 2024-05-09T09:08:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2739729524' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [global] 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout fsid = 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-05-09T09:08:55.081 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [mgr] 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout [osd] 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-05-09T09:08:55.082 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7c9ea700 1 Processor -- start 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7c9ea700 1 -- start start 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7c9ea700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7c9ea700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7807c6d0 con 0x7f0a781088a0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57880/0 (socket says 172.21.15.69:57880) 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7659c700 1 -- 172.21.15.69:0/371561788 learned_addr learned my addr 172.21.15.69:0/371561788 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7659c700 1 -- 172.21.15.69:0/371561788 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7810b1d0 con 0x7f0a781088a0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.323+0000 7f0a7659c700 1 --2- 172.21.15.69:0/371561788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0098a0 tx=0x7f0a6c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=363e3ad10616ae5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7559a700 1 -- 172.21.15.69:0/371561788 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a6c004160 con 0x7f0a781088a0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7559a700 1 -- 172.21.15.69:0/371561788 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a6c004300 con 0x7f0a781088a0 2024-05-09T09:08:55.083 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7559a700 1 -- 172.21.15.69:0/371561788 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a6c011b50 con 0x7f0a781088a0 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/371561788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 msgr2=0x7f0a7810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/371561788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c0098a0 tx=0x7f0a6c009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/371561788 shutdown_connections 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/371561788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7810ac90 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/371561788 >> 172.21.15.69:0/371561788 conn(0x7f0a78102350 msgr2=0x7f0a78104770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/371561788 shutdown_connections 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.324+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/371561788 wait complete. 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7c9ea700 1 Processor -- start 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7c9ea700 1 -- start start 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7c9ea700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7c9ea700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0a7810c130 con 0x7f0a781088a0 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57888/0 (socket says 172.21.15.69:57888) 2024-05-09T09:08:55.084 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.325+0000 7f0a7659c700 1 -- 172.21.15.69:0/4254271220 learned_addr learned my addr 172.21.15.69:0/4254271220 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a7659c700 1 -- 172.21.15.69:0/4254271220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0a7819c1f0 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a7659c700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c011670 tx=0x7f0a6c011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a6c0039c0 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f0a6c011e50 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0a7819c480 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0a6c01a5c0 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.326+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0a7819f170 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.327+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f0a6c01a8c0 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.327+0000 7f0a677fe700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 0x7f0a6003b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.327+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0a78063980 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.327+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f0a6c04d590 con 0x7f0a781088a0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.327+0000 7f0a75d9b700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 0x7f0a6003b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.328+0000 7f0a75d9b700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 0x7f0a6003b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006fd0 tx=0x7f0a68006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:55.085 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.332+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0a6c019c70 con 0x7f0a781088a0 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.523+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) -- 0x7f0a7819f340 con 0x7f0a781088a0 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.524+0000 7f0a677fe700 1 -- 172.21.15.69:0/4254271220 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) ==== 70+0+373 (secure 0 0 0) 0x7f0a6c032190 con 0x7f0a781088a0 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 msgr2=0x7f0a6003b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 0x7f0a6003b470 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0a68006fd0 tx=0x7f0a68006e40 comp rx=0 tx=0).stop 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 msgr2=0x7f0a7819bcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f0a6c011670 tx=0x7f0a6c011b60 comp rx=0 tx=0).stop 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 shutdown_connections 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f0a60038fb0 0x7f0a6003b470 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 --2- 172.21.15.69:0/4254271220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0a781088a0 0x7f0a7819bcb0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 >> 172.21.15.69:0/4254271220 conn(0x7f0a78102350 msgr2=0x7f0a78102e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 shutdown_connections 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:54.526+0000 7f0a7c9ea700 1 -- 172.21.15.69:0/4254271220 wait complete. 2024-05-09T09:08:55.086 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling cephadm module... 2024-05-09T09:08:55.375 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4254271220' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-05-09T09:08:56.420 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/899322350' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-05-09T09:08:56.420 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: ignoring --setuser ceph since I am not root 2024-05-09T09:08:56.420 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: ignoring --setgroup ceph since I am not root 2024-05-09T09:08:56.420 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:56.300+0000 7f385e057200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T09:08:56.687 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:56.489+0000 7f385e057200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:08:56.687 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:56.591+0000 7f385e057200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T09:08:56.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af7e2d700 1 Processor -- start 2024-05-09T09:08:56.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af7e2d700 1 -- start start 2024-05-09T09:08:56.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af7e2d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:56.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af7e2d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9af01087c0 con 0x7f9af0107dd0 2024-05-09T09:08:56.694 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af5bc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af5bc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57894/0 (socket says 172.21.15.69:57894) 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.478+0000 7f9af5bc9700 1 -- 172.21.15.69:0/3849669833 learned_addr learned my addr 172.21.15.69:0/3849669833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af5bc9700 1 -- 172.21.15.69:0/3849669833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af0108fd0 con 0x7f9af0107dd0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af5bc9700 1 --2- 172.21.15.69:0/3849669833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098a0 tx=0x7f9aec009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a54d23aa4196321 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af4bc7700 1 -- 172.21.15.69:0/3849669833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec004160 con 0x7f9af0107dd0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af4bc7700 1 -- 172.21.15.69:0/3849669833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9aec004300 con 0x7f9af0107dd0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af4bc7700 1 -- 172.21.15.69:0/3849669833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec011b50 con 0x7f9af0107dd0 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af7e2d700 1 -- 172.21.15.69:0/3849669833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 msgr2=0x7f9af01081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.479+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/3849669833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f9aec0098a0 tx=0x7f9aec009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- 172.21.15.69:0/3849669833 shutdown_connections 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/3849669833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af01081f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- 172.21.15.69:0/3849669833 >> 172.21.15.69:0/3849669833 conn(0x7f9af01033b0 msgr2=0x7f9af01057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- 172.21.15.69:0/3849669833 shutdown_connections 2024-05-09T09:08:56.695 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- 172.21.15.69:0/3849669833 wait complete. 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 Processor -- start 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- start start 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.480+0000 7f9af7e2d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9af010c380 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af5bc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af5bc9700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57900/0 (socket says 172.21.15.69:57900) 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af5bc9700 1 -- 172.21.15.69:0/899322350 learned_addr learned my addr 172.21.15.69:0/899322350 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af5bc9700 1 -- 172.21.15.69:0/899322350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9af019c270 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af5bc9700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9aec011670 tx=0x7f9aec011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec0039c0 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f9aec011e50 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9af019c500 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9aec01a5c0 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.481+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9af019f1f0 con 0x7f9af0107dd0 2024-05-09T09:08:56.696 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.482+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9af0063980 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.482+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 4) ==== 46276+0+0 (secure 0 0 0) 0x7f9aec01a8c0 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.482+0000 7f9ae6ffd700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 0x7f9adc03b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.482+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f9aec04d450 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.482+0000 7f9af53c8700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 0x7f9adc03b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.483+0000 7f9af53c8700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 0x7f9adc03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0006fd0 tx=0x7f9ae0006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.487+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9aec019c70 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:55.737+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) -- 0x7f9af019f830 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.132+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) ==== 86+0+0 (secure 0 0 0) 0x7f9aec02cc00 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.134+0000 7f9ae6ffd700 1 -- 172.21.15.69:0/899322350 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f9aec02d910 con 0x7f9af0107dd0 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 msgr2=0x7f9adc03b470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 0x7f9adc03b470 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9ae0006fd0 tx=0x7f9ae0006e40 comp rx=0 tx=0).stop 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 msgr2=0x7f9af019bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9aec011670 tx=0x7f9aec011b60 comp rx=0 tx=0).stop 2024-05-09T09:08:56.697 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 shutdown_connections 2024-05-09T09:08:56.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f9adc038fb0 0x7f9adc03b470 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:56.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 --2- 172.21.15.69:0/899322350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9af0107dd0 0x7f9af019bd30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:56.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 >> 172.21.15.69:0/899322350 conn(0x7f9af01033b0 msgr2=0x7f9af0103f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:56.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 shutdown_connections 2024-05-09T09:08:56.698 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:56.135+0000 7f9af7e2d700 1 -- 172.21.15.69:0/899322350 wait complete. 2024-05-09T09:08:56.994 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:56.909+0000 7f385e057200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T09:08:56.994 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:56.995+0000 7f385e057200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T09:08:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:57 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/899322350' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-05-09T09:08:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:57 smithi069 ceph-mon[30123]: mgrmap e5: x(active, since 6s) 2024-05-09T09:08:57.970 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:08:57.970 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f02070fa700 1 Processor -- start 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f02070fa700 1 -- start start 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f02070fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f02070fa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f020007c6f0 con 0x7f0200108810 2024-05-09T09:08:57.971 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f0204e96700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f0204e96700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57924/0 (socket says 172.21.15.69:57924) 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.109+0000 7f0204e96700 1 -- 172.21.15.69:0/3338446324 learned_addr learned my addr 172.21.15.69:0/3338446324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.110+0000 7f0204e96700 1 -- 172.21.15.69:0/3338446324 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f020010b140 con 0x7f0200108810 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.110+0000 7f0204e96700 1 --2- 172.21.15.69:0/3338446324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0098a0 tx=0x7f01fc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2bb8492ff0f4633 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.110+0000 7f01f77fe700 1 -- 172.21.15.69:0/3338446324 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01fc004160 con 0x7f0200108810 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.110+0000 7f01f77fe700 1 -- 172.21.15.69:0/3338446324 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f01fc004300 con 0x7f0200108810 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.110+0000 7f01f77fe700 1 -- 172.21.15.69:0/3338446324 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01fc011b50 con 0x7f0200108810 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- 172.21.15.69:0/3338446324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 msgr2=0x7f020010ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 --2- 172.21.15.69:0/3338446324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f01fc0098a0 tx=0x7f01fc009bb0 comp rx=0 tx=0).stop 2024-05-09T09:08:57.972 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- 172.21.15.69:0/3338446324 shutdown_connections 2024-05-09T09:08:57.973 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 --2- 172.21.15.69:0/3338446324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f020010ac00 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:57.973 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- 172.21.15.69:0/3338446324 >> 172.21.15.69:0/3338446324 conn(0x7f0200102200 msgr2=0x7f0200104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:57.973 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- 172.21.15.69:0/3338446324 shutdown_connections 2024-05-09T09:08:57.973 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- 172.21.15.69:0/3338446324 wait complete. 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 Processor -- start 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- start start 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.111+0000 7f02070fa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f020010c0f0 con 0x7f0200108810 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f0204e96700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f0204e96700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57938/0 (socket says 172.21.15.69:57938) 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f0204e96700 1 -- 172.21.15.69:0/3695370687 learned_addr learned my addr 172.21.15.69:0/3695370687 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f0204e96700 1 -- 172.21.15.69:0/3695370687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0200108580 con 0x7f0200108810 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f0204e96700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f01fc011670 tx=0x7f01fc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:08:57.974 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01fc0039c0 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f02001067c0 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f01fc011e50 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f01fc01a5c0 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.112+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0200106ca0 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.113+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7f01fc01a870 con 0x7f0200108810 2024-05-09T09:08:57.975 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.113+0000 7f01f5ffb700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 0x7f01e803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.113+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7f01fc04d450 con 0x7f0200108810 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.113+0000 7f01f7fff700 1 -- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 msgr2=0x7f01e803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.113+0000 7f01f7fff700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 0x7f01e803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.114+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0200063980 con 0x7f0200108810 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.119+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f01fc019c70 con 0x7f0200108810 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.314+0000 7f01f7fff700 1 -- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 msgr2=0x7f01e803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:08:57.976 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.314+0000 7f01f7fff700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 0x7f01e803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.414+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f0200107350 con 0x7f0200108810 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.418+0000 7f01f5ffb700 1 -- 172.21.15.69:0/3695370687 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) ==== 56+0+88 (secure 0 0 0) 0x7f01fc02c9c0 con 0x7f0200108810 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.421+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 msgr2=0x7f01e803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.421+0000 7f02070fa700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 0x7f01e803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.421+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 msgr2=0x7f0200108040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.421+0000 7f02070fa700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f01fc011670 tx=0x7f01fc011b60 comp rx=0 tx=0).stop 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.422+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 shutdown_connections 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.422+0000 7f02070fa700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7f01e8039050 0x7f01e803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:57.977 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.422+0000 7f02070fa700 1 --2- 172.21.15.69:0/3695370687 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0200108810 0x7f0200108040 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:08:57.978 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.422+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 >> 172.21.15.69:0/3695370687 conn(0x7f0200102200 msgr2=0x7f0200102d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:08:57.978 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.424+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 shutdown_connections 2024-05-09T09:08:57.978 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:57.424+0000 7f02070fa700 1 -- 172.21.15.69:0/3695370687 wait complete. 2024-05-09T09:08:57.978 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2024-05-09T09:08:57.978 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 5... 2024-05-09T09:08:58.229 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:08:58 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3695370687' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T09:08:58.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:58.231+0000 7f385e057200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T09:08:58.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:58.407+0000 7f385e057200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T09:08:58.863 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:58.577+0000 7f385e057200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T09:08:58.864 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:58.865+0000 7f385e057200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T09:08:59.161 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:58.956+0000 7f385e057200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T09:08:59.161 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:59.162+0000 7f385e057200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T09:08:59.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:59.355+0000 7f385e057200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:09:00.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:08:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:08:59.962+0000 7f385e057200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T09:09:00.681 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:00.387+0000 7f385e057200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T09:09:00.681 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:00.483+0000 7f385e057200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T09:09:00.681 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:00.682+0000 7f385e057200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T09:09:01.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.195+0000 7f385e057200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T09:09:01.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.286+0000 7f385e057200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T09:09:01.859 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.524+0000 7f385e057200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T09:09:01.859 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.654+0000 7f385e057200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T09:09:01.860 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.747+0000 7f385e057200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T09:09:01.860 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:01.861+0000 7f385e057200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T09:09:02.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:02.156+0000 7f385e057200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T09:09:03.009 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:02.747+0000 7f385e057200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T09:09:03.009 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:02.959+0000 7f385e057200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T09:09:03.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: Active manager daemon x restarted 2024-05-09T09:09:03.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: Activating manager daemon x 2024-05-09T09:09:03.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: osdmap e2: 0 total, 0 up, 0 in 2024-05-09T09:09:03.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: mgrmap e6: x(active, starting, since 0.00350676s) 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: Manager daemon x is now available 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:03 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T09:09:04.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: Found migration_current of "None". Setting to last migration. 2024-05-09T09:09:04.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T09:09:04.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:04.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:04.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:04.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:04 smithi069 ceph-mon[30123]: mgrmap e7: x(active, since 1.00572s) 2024-05-09T09:09:04.570 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:09:04.570 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-05-09T09:09:04.570 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.374+0000 7ff58400e700 1 Processor -- start 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.374+0000 7ff58400e700 1 -- start start 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff58400e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff58400e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c07c6b0 con 0x7ff57c106640 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff581daa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff581daa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47924/0 (socket says 172.21.15.69:47924) 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff581daa700 1 -- 172.21.15.69:0/950368649 learned_addr learned my addr 172.21.15.69:0/950368649 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff581daa700 1 -- 172.21.15.69:0/950368649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c106fa0 con 0x7ff57c106640 2024-05-09T09:09:04.571 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff581daa700 1 --2- 172.21.15.69:0/950368649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5780098a0 tx=0x7ff578009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a93fcfdc93f3377 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff580da8700 1 -- 172.21.15.69:0/950368649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff578004160 con 0x7ff57c106640 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff580da8700 1 -- 172.21.15.69:0/950368649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff578004300 con 0x7ff57c106640 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.375+0000 7ff580da8700 1 -- 172.21.15.69:0/950368649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff578011b50 con 0x7ff57c106640 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 -- 172.21.15.69:0/950368649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 msgr2=0x7ff57c106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 --2- 172.21.15.69:0/950368649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7ff5780098a0 tx=0x7ff578009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 -- 172.21.15.69:0/950368649 shutdown_connections 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 --2- 172.21.15.69:0/950368649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c106a60 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 -- 172.21.15.69:0/950368649 >> 172.21.15.69:0/950368649 conn(0x7ff57c102200 msgr2=0x7ff57c104640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 -- 172.21.15.69:0/950368649 shutdown_connections 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.376+0000 7ff58400e700 1 -- 172.21.15.69:0/950368649 wait complete. 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff58400e700 1 Processor -- start 2024-05-09T09:09:04.572 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff58400e700 1 -- start start 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff58400e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff58400e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff57c107fc0 con 0x7ff57c106640 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff581daa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff581daa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47928/0 (socket says 172.21.15.69:47928) 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff581daa700 1 -- 172.21.15.69:0/3694730153 learned_addr learned my addr 172.21.15.69:0/3694730153 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.377+0000 7ff581daa700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff57c19e420 con 0x7ff57c106640 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff581daa700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff578011670 tx=0x7ff578011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff5780039c0 con 0x7ff57c106640 2024-05-09T09:09:04.573 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7ff578011e50 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff57c19e6b0 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7ff57801a5c0 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.378+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff57c1a13a0 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 5) ==== 46287+0+0 (secure 0 0 0) 0x7ff57801a870 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff572ffd700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff56803bc20 con 0x7ff568039050 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(1..1 src has 1..1) ==== 725+0+0 (secure 0 0 0) 0x7ff57804d270 con 0x7ff57c106640 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff5815a9700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.379+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.580+0000 7ff5815a9700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.580+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T09:09:04.574 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.981+0000 7ff5815a9700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:58.981+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:59.782+0000 7ff5815a9700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:08:59.782+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:01.383+0000 7ff5815a9700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/2869747387 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:01.383+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:02.962+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 6) ==== 46054+0+0 (secure 0 0 0) 0x7ff57801abb0 con 0x7ff57c106640 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:02.963+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 msgr2=0x7ff56803b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:09:04.575 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:02.963+0000 7ff572ffd700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.964+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7ff57804e910 con 0x7ff57c106640 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.964+0000 7ff572ffd700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 0x7ff56803ed70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.964+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7ff578016e70 con 0x7ff56803c980 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.964+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 0x7ff56803ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.965+0000 7ff5815a9700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 0x7ff56803ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff56c003a10 tx=0x7ff56c0092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.966+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7ff578016e70 con 0x7ff56803c980 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7ff57c063980 con 0x7ff56803c980 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff572ffd700 1 -- 172.21.15.69:0/3694730153 <== mgr.14120 v2:172.21.15.69:6800/936285351 2 ==== command_reply(tid 1: 0 ) ==== 8+0+51 (secure 0 0 0) 0x7ff57c063980 con 0x7ff56803c980 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 msgr2=0x7ff56803ed70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 0x7ff56803ed70 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff56c003a10 tx=0x7ff56c0092b0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.576 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 msgr2=0x7ff57c19dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7ff578011670 tx=0x7ff578011b60 comp rx=0 tx=0).stop 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 shutdown_connections 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7ff56803c980 0x7ff56803ed70 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff57c106640 0x7ff57c19dee0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 --2- 172.21.15.69:0/3694730153 >> [v2:172.21.15.69:6800/2869747387,v1:172.21.15.69:6801/2869747387] conn(0x7ff568039050 0x7ff56803b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 >> 172.21.15.69:0/3694730153 conn(0x7ff57c102200 msgr2=0x7ff57c194b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 shutdown_connections 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:03.973+0000 7ff58400e700 1 -- 172.21.15.69:0/3694730153 wait complete. 2024-05-09T09:09:04.577 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 5 is available 2024-05-09T09:09:04.578 INFO:teuthology.orchestra.run.smithi069.stdout:Setting orchestrator backend to cephadm... 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: [09/May/2024:09:09:03] ENGINE Bus STARTING 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: [09/May/2024:09:09:03] ENGINE Serving on http://172.21.15.69:8765 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: [09/May/2024:09:09:03] ENGINE Serving on https://172.21.15.69:7150 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: [09/May/2024:09:09:03] ENGINE Bus STARTED 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T09:09:05.226 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:05 smithi069 ceph-mon[30123]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T09:09:05.777 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.997+0000 7fcbd35e8700 1 Processor -- start 2024-05-09T09:09:05.777 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.997+0000 7fcbd35e8700 1 -- start start 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.997+0000 7fcbd35e8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.997+0000 7fcbd35e8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc07c690 con 0x7fcbcc1066d0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbd1384700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbd1384700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47986/0 (socket says 172.21.15.69:47986) 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbd1384700 1 -- 172.21.15.69:0/1451544737 learned_addr learned my addr 172.21.15.69:0/1451544737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbd1384700 1 -- 172.21.15.69:0/1451544737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc107030 con 0x7fcbcc1066d0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbd1384700 1 --2- 172.21.15.69:0/1451544737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009e80 tx=0x7fcbc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db4ee9304626aaf7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbc3fff700 1 -- 172.21.15.69:0/1451544737 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbc8017070 con 0x7fcbcc1066d0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbc3fff700 1 -- 172.21.15.69:0/1451544737 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcbc800b8c0 con 0x7fcbcc1066d0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.998+0000 7fcbc3fff700 1 -- 172.21.15.69:0/1451544737 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbc80045c0 con 0x7fcbcc1066d0 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- 172.21.15.69:0/1451544737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 msgr2=0x7fcbcc106af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/1451544737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcbc8009e80 tx=0x7fcbc800b330 comp rx=0 tx=0).stop 2024-05-09T09:09:05.778 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- 172.21.15.69:0/1451544737 shutdown_connections 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/1451544737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc106af0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- 172.21.15.69:0/1451544737 >> 172.21.15.69:0/1451544737 conn(0x7fcbcc1022d0 msgr2=0x7fcbcc1046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- 172.21.15.69:0/1451544737 shutdown_connections 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- 172.21.15.69:0/1451544737 wait complete. 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 Processor -- start 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:04.999+0000 7fcbd35e8700 1 -- start start 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd35e8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:05.779 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd35e8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcbcc108050 con 0x7fcbcc1066d0 2024-05-09T09:09:05.782 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd1384700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:05.782 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd1384700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48002/0 (socket says 172.21.15.69:48002) 2024-05-09T09:09:05.782 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd1384700 1 -- 172.21.15.69:0/745842416 learned_addr learned my addr 172.21.15.69:0/745842416 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:05.782 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd1384700 1 -- 172.21.15.69:0/745842416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcbcc19e3f0 con 0x7fcbcc1066d0 2024-05-09T09:09:05.782 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd1384700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80072a0 tx=0x7fcbc800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbc8020020 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcbcc19e680 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcbc8003b50 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcbc8017040 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.000+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcbcc1a1370 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.001+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 7) ==== 46181+0+0 (secure 0 0 0) 0x7fcbc8023020 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.001+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcbcc063980 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.002+0000 7fcbc27fc700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 0x7fcbb803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.002+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcbc800f050 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.002+0000 7fcbd0b83700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 0x7fcbb803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.002+0000 7fcbd0b83700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 0x7fcbb803b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc006fd0 tx=0x7fcbbc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.007+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcbc802a020 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.018+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcbc8011390 con 0x7fcbcc1066d0 2024-05-09T09:09:05.783 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.220+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) -- 0x7fcbcc02df30 con 0x7fcbb8038ed0 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.227+0000 7fcbc27fc700 1 -- 172.21.15.69:0/745842416 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcbcc02df30 con 0x7fcbb8038ed0 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 msgr2=0x7fcbb803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 0x7fcbb803b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcbbc006fd0 tx=0x7fcbbc006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 msgr2=0x7fcbcc19deb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcbc80072a0 tx=0x7fcbc800bd50 comp rx=0 tx=0).stop 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 shutdown_connections 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcbb8038ed0 0x7fcbb803b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 --2- 172.21.15.69:0/745842416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcbcc1066d0 0x7fcbcc19deb0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 >> 172.21.15.69:0/745842416 conn(0x7fcbcc1022d0 msgr2=0x7fcbcc194af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 shutdown_connections 2024-05-09T09:09:05.784 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:05.230+0000 7fcbd35e8700 1 -- 172.21.15.69:0/745842416 wait complete. 2024-05-09T09:09:06.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:06 smithi069 ceph-mon[30123]: mgrmap e8: x(active, since 2s) 2024-05-09T09:09:06.158 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:06 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:06.159 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:06 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout value unchanged 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80ab588700 1 Processor -- start 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80ab588700 1 -- start start 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80ab588700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80ab588700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80a407d170 con 0x7f80a407e740 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80a9324700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80a9324700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48014/0 (socket says 172.21.15.69:48014) 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.160+0000 7f80a9324700 1 -- 172.21.15.69:0/739690838 learned_addr learned my addr 172.21.15.69:0/739690838 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:06.870 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.161+0000 7f80a9324700 1 -- 172.21.15.69:0/739690838 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a407d2f0 con 0x7f80a407e740 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.161+0000 7f80a9324700 1 --2- 172.21.15.69:0/739690838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80a00098a0 tx=0x7f80a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3348158635dac448 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.161+0000 7f809bfff700 1 -- 172.21.15.69:0/739690838 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80a0004160 con 0x7f80a407e740 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.161+0000 7f809bfff700 1 -- 172.21.15.69:0/739690838 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f80a0004300 con 0x7f80a407e740 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.161+0000 7f809bfff700 1 -- 172.21.15.69:0/739690838 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80a0011b50 con 0x7f80a407e740 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- 172.21.15.69:0/739690838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 msgr2=0x7f80a407cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 --2- 172.21.15.69:0/739690838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f80a00098a0 tx=0x7f80a0009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- 172.21.15.69:0/739690838 shutdown_connections 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 --2- 172.21.15.69:0/739690838 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a407cba0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- 172.21.15.69:0/739690838 >> 172.21.15.69:0/739690838 conn(0x7f80a41033b0 msgr2=0x7f80a41057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:06.871 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- 172.21.15.69:0/739690838 shutdown_connections 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- 172.21.15.69:0/739690838 wait complete. 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 Processor -- start 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.162+0000 7f80ab588700 1 -- start start 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80ab588700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80ab588700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80a410e360 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80a9324700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80a9324700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48030/0 (socket says 172.21.15.69:48030) 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80a9324700 1 -- 172.21.15.69:0/4236298113 learned_addr learned my addr 172.21.15.69:0/4236298113 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80a9324700 1 -- 172.21.15.69:0/4236298113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a419e450 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80a9324700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f80a0011670 tx=0x7f80a0011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80a00039c0 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f80a0011e50 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a419e6e0 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f80a001a5c0 con 0x7f80a407e740 2024-05-09T09:09:06.872 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.163+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a41a13d0 con 0x7f80a407e740 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.164+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80a4063980 con 0x7f80a407e740 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.164+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f80a001a8c0 con 0x7f80a407e740 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.164+0000 7f809a7fc700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 0x7f809003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.165+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f80a004d610 con 0x7f80a407e740 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.165+0000 7f80a8b23700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 0x7f809003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.165+0000 7f80a8b23700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 0x7f809003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8094006fd0 tx=0x7f8094006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.169+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f80a0019c70 con 0x7f80a407e740 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.368+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) -- 0x7f80a41045d0 con 0x7f8090039000 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.368+0000 7f809a7fc700 1 -- 172.21.15.69:0/4236298113 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+16 (secure 0 0 0) 0x7f80a41045d0 con 0x7f8090039000 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 msgr2=0x7f809003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 0x7f809003b4c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f8094006fd0 tx=0x7f8094006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 msgr2=0x7f80a419df10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f80a0011670 tx=0x7f80a0011b60 comp rx=0 tx=0).stop 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 shutdown_connections 2024-05-09T09:09:06.873 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f8090039000 0x7f809003b4c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.370+0000 7f80ab588700 1 --2- 172.21.15.69:0/4236298113 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a407e740 0x7f80a419df10 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.371+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 >> 172.21.15.69:0/4236298113 conn(0x7f80a41033b0 msgr2=0x7f80a4103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.371+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 shutdown_connections 2024-05-09T09:09:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:06.371+0000 7f80ab588700 1 -- 172.21.15.69:0/4236298113 wait complete. 2024-05-09T09:09:06.874 INFO:teuthology.orchestra.run.smithi069.stdout:Generating ssh key... 2024-05-09T09:09:07.255 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:07 smithi069 ceph-mon[30123]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: Generating public/private rsa key pair. 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: Your identification has been saved in /tmp/tmpp9pjhn39/key. 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: Your public key has been saved in /tmp/tmpp9pjhn39/key.pub. 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: The key fingerprint is: 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: SHA256:DaMDULUfad1Ll2n9h3Jq84vE00vihQgAH9iX7AsSne8 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: The key's randomart image is: 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: +---[RSA 3072]----+ 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | ..o=oo . | 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | .oo+o+o . + | 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | ..++* . o = .| 2024-05-09T09:09:07.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | ...=o= . + ..| 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | .ooS.. o o o| 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | .E. o * .| 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | . X + | 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | + O . | 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: | o +. | 2024-05-09T09:09:07.961 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: +----[SHA256]-----+ 2024-05-09T09:09:08.235 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:08 smithi069 ceph-mon[30123]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:08.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:08 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:08.236 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:08 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.256+0000 7fcdddf24700 1 Processor -- start 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.256+0000 7fcdddf24700 1 -- start start 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.256+0000 7fcdddf24700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.256+0000 7fcdddf24700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd807c680 con 0x7fcdd81088a0 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48044/0 (socket says 172.21.15.69:48044) 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd77fe700 1 -- 172.21.15.69:0/420410103 learned_addr learned my addr 172.21.15.69:0/420410103 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd77fe700 1 -- 172.21.15.69:0/420410103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd810b1d0 con 0x7fcdd81088a0 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd77fe700 1 --2- 172.21.15.69:0/420410103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009e80 tx=0x7fcdc800b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=601495a0ce860cb1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd67fc700 1 -- 172.21.15.69:0/420410103 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc8017070 con 0x7fcdd81088a0 2024-05-09T09:09:08.240 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd67fc700 1 -- 172.21.15.69:0/420410103 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcdc800b8c0 con 0x7fcdd81088a0 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.257+0000 7fcdd67fc700 1 -- 172.21.15.69:0/420410103 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc80045c0 con 0x7fcdd81088a0 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 -- 172.21.15.69:0/420410103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 msgr2=0x7fcdd810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 --2- 172.21.15.69:0/420410103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8009e80 tx=0x7fcdc800b330 comp rx=0 tx=0).stop 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 -- 172.21.15.69:0/420410103 shutdown_connections 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 --2- 172.21.15.69:0/420410103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd810ac90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 -- 172.21.15.69:0/420410103 >> 172.21.15.69:0/420410103 conn(0x7fcdd8102290 msgr2=0x7fcdd81046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 -- 172.21.15.69:0/420410103 shutdown_connections 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 -- 172.21.15.69:0/420410103 wait complete. 2024-05-09T09:09:08.241 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.258+0000 7fcdddf24700 1 Processor -- start 2024-05-09T09:09:08.244 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdddf24700 1 -- start start 2024-05-09T09:09:08.244 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdddf24700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdddf24700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdd810c120 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdd77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48046/0 (socket says 172.21.15.69:48046) 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdd77fe700 1 -- 172.21.15.69:0/1091980233 learned_addr learned my addr 172.21.15.69:0/1091980233 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdd77fe700 1 -- 172.21.15.69:0/1091980233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdd819c190 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.259+0000 7fcdd77fe700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8004350 tx=0x7fcdc800bd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.260+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc8020020 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.260+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fcdc8003c10 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.260+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdd819c420 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.260+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fcdc8017040 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.260+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdd819f110 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fcdc8023020 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdd8063980 con 0x7fcdd81088a0 2024-05-09T09:09:08.245 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdd4ff9700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 0x7fcdc003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fcdc800f050 con 0x7fcdd81088a0 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdd6ffd700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 0x7fcdc003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.261+0000 7fcdd6ffd700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 0x7fcdc003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.266+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdc802a020 con 0x7fcdd81088a0 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.462+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) -- 0x7fcdd8103500 con 0x7fcdc0039050 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.712+0000 7fcdd4ff9700 1 -- 172.21.15.69:0/1091980233 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7fcdd8103500 con 0x7fcdc0039050 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 msgr2=0x7fcdc003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 0x7fcdc003b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc006fd0 tx=0x7fcdcc006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 msgr2=0x7fcdd819bc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fcdc8004350 tx=0x7fcdc800bd50 comp rx=0 tx=0).stop 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 shutdown_connections 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fcdc0039050 0x7fcdc003b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 --2- 172.21.15.69:0/1091980233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdd81088a0 0x7fcdd819bc50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:08.246 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 >> 172.21.15.69:0/1091980233 conn(0x7fcdd8102290 msgr2=0x7fcdd8102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:08.247 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 shutdown_connections 2024-05-09T09:09:08.247 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:07.714+0000 7fcdddf24700 1 -- 172.21.15.69:0/1091980233 wait complete. 2024-05-09T09:09:09.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:09 smithi069 ceph-mon[30123]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:09.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:09 smithi069 ceph-mon[30123]: Generating ssh key... 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.655+0000 7fe699753700 1 Processor -- start 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.655+0000 7fe699753700 1 -- start start 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.655+0000 7fe699753700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.655+0000 7fe699753700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe69407c710 con 0x7fe694106640 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe692ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe692ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55284/0 (socket says 172.21.15.69:55284) 2024-05-09T09:09:09.363 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe692ffd700 1 -- 172.21.15.69:0/4186266253 learned_addr learned my addr 172.21.15.69:0/4186266253 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe692ffd700 1 -- 172.21.15.69:0/4186266253 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe694108fb0 con 0x7fe694106640 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe692ffd700 1 --2- 172.21.15.69:0/4186266253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098a0 tx=0x7fe684009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a07f29c4f724521e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe6927fc700 1 -- 172.21.15.69:0/4186266253 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe684004160 con 0x7fe694106640 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe6927fc700 1 -- 172.21.15.69:0/4186266253 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe684004300 con 0x7fe694106640 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.656+0000 7fe6927fc700 1 -- 172.21.15.69:0/4186266253 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe684011b50 con 0x7fe694106640 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 -- 172.21.15.69:0/4186266253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 msgr2=0x7fe694108a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 --2- 172.21.15.69:0/4186266253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fe6840098a0 tx=0x7fe684009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 -- 172.21.15.69:0/4186266253 shutdown_connections 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 --2- 172.21.15.69:0/4186266253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe694108a70 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 -- 172.21.15.69:0/4186266253 >> 172.21.15.69:0/4186266253 conn(0x7fe6941022c0 msgr2=0x7fe6941046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 -- 172.21.15.69:0/4186266253 shutdown_connections 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 -- 172.21.15.69:0/4186266253 wait complete. 2024-05-09T09:09:09.364 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.657+0000 7fe699753700 1 Processor -- start 2024-05-09T09:09:09.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe699753700 1 -- start start 2024-05-09T09:09:09.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe699753700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:09.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe699753700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe694109fd0 con 0x7fe694106640 2024-05-09T09:09:09.368 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe692ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe692ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55298/0 (socket says 172.21.15.69:55298) 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe692ffd700 1 -- 172.21.15.69:0/4194534645 learned_addr learned my addr 172.21.15.69:0/4194534645 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe692ffd700 1 -- 172.21.15.69:0/4194534645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe6941a28b0 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe692ffd700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe684009fb0 tx=0x7fe684011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe6840038b0 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fe684011e50 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe6941a2b40 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fe684019420 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.658+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe6941a5830 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.659+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fe6840196d0 con 0x7fe694106640 2024-05-09T09:09:09.369 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.659+0000 7fe690ff9700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 0x7fe68003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.659+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fe68404d2b0 con 0x7fe694106640 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.659+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe694063980 con 0x7fe694106640 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.660+0000 7fe68bfff700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 0x7fe68003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.660+0000 7fe68bfff700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 0x7fe68003b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe67c006fd0 tx=0x7fe67c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.664+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe68400bc70 con 0x7fe694106640 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.858+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) -- 0x7fe69402de20 con 0x7fe680039050 2024-05-09T09:09:09.370 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.859+0000 7fe690ff9700 1 -- 172.21.15.69:0/4194534645 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+595 (secure 0 0 0) 0x7fe69402de20 con 0x7fe680039050 2024-05-09T09:09:09.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 msgr2=0x7fe68003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:09.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 0x7fe68003b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe67c006fd0 tx=0x7fe67c006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:09.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 msgr2=0x7fe6941a2370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:09.374 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fe684009fb0 tx=0x7fe684011bf0 comp rx=0 tx=0).stop 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 shutdown_connections 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fe680039050 0x7fe68003b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 --2- 172.21.15.69:0/4194534645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe694106640 0x7fe6941a2370 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 >> 172.21.15.69:0/4194534645 conn(0x7fe6941022c0 msgr2=0x7fe6941046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 shutdown_connections 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:08.861+0000 7fe699753700 1 -- 172.21.15.69:0/4194534645 wait complete. 2024-05-09T09:09:09.375 INFO:teuthology.orchestra.run.smithi069.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:09:09.376 INFO:teuthology.orchestra.run.smithi069.stdout:Adding key to root@localhost authorized_keys... 2024-05-09T09:09:09.376 INFO:teuthology.orchestra.run.smithi069.stdout:Adding host smithi069... 2024-05-09T09:09:10.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:10 smithi069 ceph-mon[30123]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:11.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:11 smithi069 ceph-mon[30123]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:12 smithi069 ceph-mon[30123]: Deploying cephadm binary to smithi069 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Added host 'smithi069' with addr '172.21.15.69' 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f741b35b700 1 Processor -- start 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f741b35b700 1 -- start start 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f741b35b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f741b35b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f741407c6f0 con 0x7f7414108850 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f74190f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:14.667 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f74190f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55312/0 (socket says 172.21.15.69:55312) 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.778+0000 7f74190f7700 1 -- 172.21.15.69:0/2325704428 learned_addr learned my addr 172.21.15.69:0/2325704428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f74190f7700 1 -- 172.21.15.69:0/2325704428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741410b180 con 0x7f7414108850 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f74190f7700 1 --2- 172.21.15.69:0/2325704428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74100098a0 tx=0x7f7410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=312875cfe9fc33d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f740bfff700 1 -- 172.21.15.69:0/2325704428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7410004160 con 0x7f7414108850 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f740bfff700 1 -- 172.21.15.69:0/2325704428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7410004300 con 0x7f7414108850 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f740bfff700 1 -- 172.21.15.69:0/2325704428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7410011b50 con 0x7f7414108850 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f741b35b700 1 -- 172.21.15.69:0/2325704428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 msgr2=0x7f741410ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f741b35b700 1 --2- 172.21.15.69:0/2325704428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f74100098a0 tx=0x7f7410009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f741b35b700 1 -- 172.21.15.69:0/2325704428 shutdown_connections 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f741b35b700 1 --2- 172.21.15.69:0/2325704428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741410ac40 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.779+0000 7f741b35b700 1 -- 172.21.15.69:0/2325704428 >> 172.21.15.69:0/2325704428 conn(0x7f7414102200 msgr2=0x7f7414104680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 -- 172.21.15.69:0/2325704428 shutdown_connections 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 -- 172.21.15.69:0/2325704428 wait complete. 2024-05-09T09:09:14.668 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 Processor -- start 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 -- start start 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f741b35b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f741410c130 con 0x7f7414108850 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f74190f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f74190f7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55328/0 (socket says 172.21.15.69:55328) 2024-05-09T09:09:14.669 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.780+0000 7f74190f7700 1 -- 172.21.15.69:0/677049252 learned_addr learned my addr 172.21.15.69:0/677049252 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f74190f7700 1 -- 172.21.15.69:0/677049252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f741419c200 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f74190f7700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7410011670 tx=0x7f7410011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f74100039c0 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7f7410011e50 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f741419c490 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f741001a5c0 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.781+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f741419f180 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.782+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f741001a870 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.782+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7414063980 con 0x7f7414108850 2024-05-09T09:09:14.670 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.782+0000 7f740a7fc700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 0x7f740003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.782+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f741004d540 con 0x7f7414108850 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.782+0000 7f74188f6700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 0x7f740003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.783+0000 7f74188f6700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 0x7f740003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.787+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7410019c70 con 0x7f7414108850 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:09.984+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi069", "addr": "172.21.15.69", "target": ["mon-mgr", ""]}) -- 0x7f74141034b0 con 0x7f7400039050 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.119+0000 7f740a7fc700 1 -- 172.21.15.69:0/677049252 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+48 (secure 0 0 0) 0x7f74141034b0 con 0x7f7400039050 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 msgr2=0x7f740003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 0x7f740003b510 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7404006fd0 tx=0x7f7404006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 msgr2=0x7f741419bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7410011670 tx=0x7f7410011b60 comp rx=0 tx=0).stop 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 shutdown_connections 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f7400039050 0x7f740003b510 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 --2- 172.21.15.69:0/677049252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7414108850 0x7f741419bcc0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:14.671 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 >> 172.21.15.69:0/677049252 conn(0x7f7414102200 msgr2=0x7f7414102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:14.672 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 shutdown_connections 2024-05-09T09:09:14.672 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:14.122+0000 7f741b35b700 1 -- 172.21.15.69:0/677049252 wait complete. 2024-05-09T09:09:14.672 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mon service... 2024-05-09T09:09:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:15 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:15 smithi069 ceph-mon[30123]: Added host smithi069 2024-05-09T09:09:15.247 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:15 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:16.614 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:16 smithi069 ceph-mon[30123]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:16.614 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:16 smithi069 ceph-mon[30123]: Saving service mon spec with placement count:5 2024-05-09T09:09:16.614 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:16 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:16.897 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-05-09T09:09:16.897 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.072+0000 7fa7aa7c8700 1 Processor -- start 2024-05-09T09:09:16.897 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.072+0000 7fa7aa7c8700 1 -- start start 2024-05-09T09:09:16.897 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7aa7c8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7aa7c8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a4106a90 con 0x7fa7a41060a0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55336/0 (socket says 172.21.15.69:55336) 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a3fff700 1 -- 172.21.15.69:0/884194867 learned_addr learned my addr 172.21.15.69:0/884194867 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a3fff700 1 -- 172.21.15.69:0/884194867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a4106c10 con 0x7fa7a41060a0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a3fff700 1 --2- 172.21.15.69:0/884194867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7940098a0 tx=0x7fa794009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f586fc090c014396 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a37fe700 1 -- 172.21.15.69:0/884194867 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa794004160 con 0x7fa7a41060a0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a37fe700 1 -- 172.21.15.69:0/884194867 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa794004300 con 0x7fa7a41060a0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.073+0000 7fa7a37fe700 1 -- 172.21.15.69:0/884194867 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa794011b50 con 0x7fa7a41060a0 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/884194867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 msgr2=0x7fa7a41064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:16.898 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/884194867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa7940098a0 tx=0x7fa794009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/884194867 shutdown_connections 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/884194867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a41064c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/884194867 >> 172.21.15.69:0/884194867 conn(0x7fa7a4101a20 msgr2=0x7fa7a4103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/884194867 shutdown_connections 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.074+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/884194867 wait complete. 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7aa7c8700 1 Processor -- start 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7aa7c8700 1 -- start start 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7aa7c8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7aa7c8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a4109ef0 con 0x7fa7a41060a0 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55340/0 (socket says 172.21.15.69:55340) 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7a3fff700 1 -- 172.21.15.69:0/1783925945 learned_addr learned my addr 172.21.15.69:0/1783925945 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:16.899 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7a3fff700 1 -- 172.21.15.69:0/1783925945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a419c230 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.075+0000 7fa7a3fff700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa794011670 tx=0x7fa794011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa7940039c0 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a419c520 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1109+0+0 (secure 0 0 0) 0x7fa794011e50 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fa79401a5c0 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a419f1a0 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.076+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fa794024430 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.077+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7a410c350 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.077+0000 7fa7a1ffb700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 0x7fa79003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.077+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fa79404d4c0 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.077+0000 7fa79bfff700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 0x7fa79003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.077+0000 7fa79bfff700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 0x7fa79003b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa78c006fd0 tx=0x7fa78c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.082+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa79402d910 con 0x7fa7a41060a0 2024-05-09T09:09:16.900 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.303+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fa7a4102cc0 con 0x7fa790039000 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.308+0000 7fa7a1ffb700 1 -- 172.21.15.69:0/1783925945 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fa7a4102cc0 con 0x7fa790039000 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 msgr2=0x7fa79003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 0x7fa79003b4c0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fa78c006fd0 tx=0x7fa78c006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 msgr2=0x7fa7a419bcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa794011670 tx=0x7fa794011b60 comp rx=0 tx=0).stop 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 shutdown_connections 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fa790039000 0x7fa79003b4c0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 --2- 172.21.15.69:0/1783925945 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a41060a0 0x7fa7a419bcf0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 >> 172.21.15.69:0/1783925945 conn(0x7fa7a4101a20 msgr2=0x7fa7a41025b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 shutdown_connections 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:15.310+0000 7fa7aa7c8700 1 -- 172.21.15.69:0/1783925945 wait complete. 2024-05-09T09:09:16.901 INFO:teuthology.orchestra.run.smithi069.stdout:Deploying unmanaged mgr service... 2024-05-09T09:09:17.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:17 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:17.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:17 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:18.148 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb29ea61700 1 Processor -- start 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb29ea61700 1 -- start start 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb29ea61700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb29ea61700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb298106a70 con 0x7fb298106080 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb297fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:18.149 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb297fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55356/0 (socket says 172.21.15.69:55356) 2024-05-09T09:09:18.150 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb297fff700 1 -- 172.21.15.69:0/1954295068 learned_addr learned my addr 172.21.15.69:0/1954295068 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:18.150 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.350+0000 7fb297fff700 1 -- 172.21.15.69:0/1954295068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb298106bf0 con 0x7fb298106080 2024-05-09T09:09:18.150 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.351+0000 7fb297fff700 1 --2- 172.21.15.69:0/1954295068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2880098a0 tx=0x7fb288009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed2bf9e80f9bba7a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.351+0000 7fb296ffd700 1 -- 172.21.15.69:0/1954295068 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb288004160 con 0x7fb298106080 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.351+0000 7fb296ffd700 1 -- 172.21.15.69:0/1954295068 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb288004300 con 0x7fb298106080 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.351+0000 7fb296ffd700 1 -- 172.21.15.69:0/1954295068 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb288011b20 con 0x7fb298106080 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 -- 172.21.15.69:0/1954295068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 msgr2=0x7fb2981064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 --2- 172.21.15.69:0/1954295068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fb2880098a0 tx=0x7fb288009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 -- 172.21.15.69:0/1954295068 shutdown_connections 2024-05-09T09:09:18.154 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 --2- 172.21.15.69:0/1954295068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb2981064a0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 -- 172.21.15.69:0/1954295068 >> 172.21.15.69:0/1954295068 conn(0x7fb298101a20 msgr2=0x7fb298103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 -- 172.21.15.69:0/1954295068 shutdown_connections 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.352+0000 7fb29ea61700 1 -- 172.21.15.69:0/1954295068 wait complete. 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb29ea61700 1 Processor -- start 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb29ea61700 1 -- start start 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb29ea61700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb29ea61700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb29810c330 con 0x7fb298106080 2024-05-09T09:09:18.155 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb297fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb297fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55368/0 (socket says 172.21.15.69:55368) 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.353+0000 7fb297fff700 1 -- 172.21.15.69:0/3167707677 learned_addr learned my addr 172.21.15.69:0/3167707677 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb297fff700 1 -- 172.21.15.69:0/3167707677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb29819c220 con 0x7fb298106080 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb297fff700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2880117e0 tx=0x7fb288011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb288003bf0 con 0x7fb298106080 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb28801a910 con 0x7fb298106080 2024-05-09T09:09:18.156 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb29819c4b0 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb2880249a0 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.354+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb29819c910 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.355+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb298063980 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.355+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fb288022020 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.356+0000 7fb2957fa700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 0x7fb28003b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.356+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fb28804d380 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.356+0000 7fb2977fe700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 0x7fb28003b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.356+0000 7fb2977fe700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 0x7fb28003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006fd0 tx=0x7fb28c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.361+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb288011e20 con 0x7fb298106080 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.581+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) -- 0x7fb298102bd0 con 0x7fb280039000 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.589+0000 7fb2957fa700 1 -- 172.21.15.69:0/3167707677 <== mgr.14120 v2:172.21.15.69:6800/936285351 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fb298102bd0 con 0x7fb280039000 2024-05-09T09:09:18.157 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 msgr2=0x7fb28003b4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:18.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 0x7fb28003b4c0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb28c006fd0 tx=0x7fb28c006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:18.158 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 msgr2=0x7fb29819bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fb2880117e0 tx=0x7fb288011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 shutdown_connections 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fb280039000 0x7fb28003b4c0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 --2- 172.21.15.69:0/3167707677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb298106080 0x7fb29819bce0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 >> 172.21.15.69:0/3167707677 conn(0x7fb298101a20 msgr2=0x7fb2981024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 shutdown_connections 2024-05-09T09:09:18.159 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:17.592+0000 7fb29ea61700 1 -- 172.21.15.69:0/3167707677 wait complete. 2024-05-09T09:09:18.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:18 smithi069 ceph-mon[30123]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:18.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:18 smithi069 ceph-mon[30123]: Saving service mgr spec with placement count:2 2024-05-09T09:09:18.807 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:18 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:19.847 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.576+0000 7f98da440700 1 Processor -- start 2024-05-09T09:09:19.847 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.576+0000 7f98da440700 1 -- start start 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98da440700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98da440700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98d407be60 con 0x7f98d4108860 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53414/0 (socket says 172.21.15.69:53414) 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98d3fff700 1 -- 172.21.15.69:0/2737384824 learned_addr learned my addr 172.21.15.69:0/2737384824 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98d3fff700 1 -- 172.21.15.69:0/2737384824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d410b190 con 0x7f98d4108860 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.577+0000 7f98d3fff700 1 --2- 172.21.15.69:0/2737384824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98c40098a0 tx=0x7f98c4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c84846c3611ddcf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98d2ffd700 1 -- 172.21.15.69:0/2737384824 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98c4004160 con 0x7f98d4108860 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98d2ffd700 1 -- 172.21.15.69:0/2737384824 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98c4004300 con 0x7f98d4108860 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98d2ffd700 1 -- 172.21.15.69:0/2737384824 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98c4011b20 con 0x7f98d4108860 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 -- 172.21.15.69:0/2737384824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 msgr2=0x7f98d410ac50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:19.848 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 --2- 172.21.15.69:0/2737384824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f98c40098a0 tx=0x7f98c4009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 -- 172.21.15.69:0/2737384824 shutdown_connections 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 --2- 172.21.15.69:0/2737384824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d410ac50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 -- 172.21.15.69:0/2737384824 >> 172.21.15.69:0/2737384824 conn(0x7f98d4102290 msgr2=0x7f98d41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 -- 172.21.15.69:0/2737384824 shutdown_connections 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.578+0000 7f98da440700 1 -- 172.21.15.69:0/2737384824 wait complete. 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98da440700 1 Processor -- start 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98da440700 1 -- start start 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98da440700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98da440700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98d410c1f0 con 0x7f98d4108860 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98d3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53420/0 (socket says 172.21.15.69:53420) 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.579+0000 7f98d3fff700 1 -- 172.21.15.69:0/619034035 learned_addr learned my addr 172.21.15.69:0/619034035 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98d3fff700 1 -- 172.21.15.69:0/619034035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98d419e420 con 0x7f98d4108860 2024-05-09T09:09:19.849 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98d3fff700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98c40117e0 tx=0x7f98c4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98c4003ba0 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98c401a910 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f98c40248f0 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98d419e6b0 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.580+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98d419eb10 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.581+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f98c4022020 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.581+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f98d4063980 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.581+0000 7f98d17fa700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 0x7f98c003b560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.581+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f98c404d470 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.581+0000 7f98d37fe700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 0x7f98c003b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.582+0000 7f98d37fe700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 0x7f98c003b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006fd0 tx=0x7f98bc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.586+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98c400f2e0 con 0x7f98d4108860 2024-05-09T09:09:19.850 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.800+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) -- 0x7f98d41a13a0 con 0x7f98d4108860 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.805+0000 7f98d17fa700 1 -- 172.21.15.69:0/619034035 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) ==== 142+0+0 (secure 0 0 0) 0x7f98c402d3f0 con 0x7f98d4108860 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 msgr2=0x7f98c003b560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 0x7f98c003b560 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f98bc006fd0 tx=0x7f98bc006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 msgr2=0x7f98d419dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f98c40117e0 tx=0x7f98c4011ba0 comp rx=0 tx=0).stop 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 shutdown_connections 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f98c00390a0 0x7f98c003b560 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 --2- 172.21.15.69:0/619034035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98d4108860 0x7f98d419dee0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 >> 172.21.15.69:0/619034035 conn(0x7f98d4102290 msgr2=0x7f98d4102ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 shutdown_connections 2024-05-09T09:09:19.851 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:18.808+0000 7f98da440700 1 -- 172.21.15.69:0/619034035 wait complete. 2024-05-09T09:09:20.101 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:19 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/619034035' entity='client.admin' 2024-05-09T09:09:21.062 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f3431ba4700 1 Processor -- start 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f3431ba4700 1 -- start start 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f3431ba4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f3431ba4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f342c07be60 con 0x7f342c106650 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f342b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f342b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53428/0 (socket says 172.21.15.69:53428) 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f342b7fe700 1 -- 172.21.15.69:0/805385103 learned_addr learned my addr 172.21.15.69:0/805385103 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f342b7fe700 1 -- 172.21.15.69:0/805385103 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c106fb0 con 0x7f342c106650 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.290+0000 7f342b7fe700 1 --2- 172.21.15.69:0/805385103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f341c0098a0 tx=0x7f341c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b92fc620fa46e72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.291+0000 7f342a7fc700 1 -- 172.21.15.69:0/805385103 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f341c004160 con 0x7f342c106650 2024-05-09T09:09:21.063 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.291+0000 7f342a7fc700 1 -- 172.21.15.69:0/805385103 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341c004300 con 0x7f342c106650 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.291+0000 7f342a7fc700 1 -- 172.21.15.69:0/805385103 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f341c011b20 con 0x7f342c106650 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.291+0000 7f3431ba4700 1 -- 172.21.15.69:0/805385103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 msgr2=0x7f342c106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.291+0000 7f3431ba4700 1 --2- 172.21.15.69:0/805385103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f341c0098a0 tx=0x7f341c009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- 172.21.15.69:0/805385103 shutdown_connections 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 --2- 172.21.15.69:0/805385103 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c106a70 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- 172.21.15.69:0/805385103 >> 172.21.15.69:0/805385103 conn(0x7f342c102290 msgr2=0x7f342c1046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- 172.21.15.69:0/805385103 shutdown_connections 2024-05-09T09:09:21.064 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- 172.21.15.69:0/805385103 wait complete. 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 Processor -- start 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- start start 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.292+0000 7f3431ba4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f342c108080 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f342b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f342b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53434/0 (socket says 172.21.15.69:53434) 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f342b7fe700 1 -- 172.21.15.69:0/3623538929 learned_addr learned my addr 172.21.15.69:0/3623538929 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f342b7fe700 1 -- 172.21.15.69:0/3623538929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f342c19e3e0 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f342b7fe700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f341c0117e0 tx=0x7f341c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f341c003bf0 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f341c01a910 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f341c024970 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f342c19e670 con 0x7f342c106650 2024-05-09T09:09:21.065 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.293+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f342c19ead0 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.294+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7f341c022020 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.294+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f342c063980 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.294+0000 7f3428ff9700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 0x7f341403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.294+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f341c04d380 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.295+0000 7f342affd700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 0x7f341403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.295+0000 7f342affd700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 0x7f341403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3420006fd0 tx=0x7f3420006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.299+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f341c011e20 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.536+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) -- 0x7f342c062f30 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.538+0000 7f3428ff9700 1 -- 172.21.15.69:0/3623538929 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) ==== 130+0+0 (secure 0 0 0) 0x7f341c0172b0 con 0x7f342c106650 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 msgr2=0x7f341403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 0x7f341403b510 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f3420006fd0 tx=0x7f3420006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:21.066 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 msgr2=0x7f342c19dea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f341c0117e0 tx=0x7f341c011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 shutdown_connections 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3414039050 0x7f341403b510 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 --2- 172.21.15.69:0/3623538929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f342c106650 0x7f342c19dea0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 >> 172.21.15.69:0/3623538929 conn(0x7f342c102290 msgr2=0x7f342c194ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 shutdown_connections 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:20.541+0000 7f3431ba4700 1 -- 172.21.15.69:0/3623538929 wait complete. 2024-05-09T09:09:21.067 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling the dashboard module... 2024-05-09T09:09:21.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:20 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:21.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:20 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3623538929' entity='client.admin' 2024-05-09T09:09:22.470 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:22 smithi069 ceph-mon[30123]: from='mgr.14120 172.21.15.69:0/1021909657' entity='mgr.x' 2024-05-09T09:09:22.470 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:22 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1732381087' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-05-09T09:09:22.471 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: ignoring --setuser ceph since I am not root 2024-05-09T09:09:22.471 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: ignoring --setgroup ceph since I am not root 2024-05-09T09:09:22.471 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:22.284+0000 7fb252bfb200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T09:09:22.471 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:22.471+0000 7fb252bfb200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:09:22.732 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:22.574+0000 7fb252bfb200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T09:09:22.740 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.462+0000 7fee322ed700 1 Processor -- start 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.462+0000 7fee322ed700 1 -- start start 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.462+0000 7fee322ed700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.462+0000 7fee322ed700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c106aa0 con 0x7fee2c1060b0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53446/0 (socket says 172.21.15.69:53446) 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2bfff700 1 -- 172.21.15.69:0/3329955526 learned_addr learned my addr 172.21.15.69:0/3329955526 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2bfff700 1 -- 172.21.15.69:0/3329955526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c109040 con 0x7fee2c1060b0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2bfff700 1 --2- 172.21.15.69:0/3329955526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee200098a0 tx=0x7fee20009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=caf97734541fb35e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2affd700 1 -- 172.21.15.69:0/3329955526 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee20004160 con 0x7fee2c1060b0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2affd700 1 -- 172.21.15.69:0/3329955526 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee20004300 con 0x7fee2c1060b0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.463+0000 7fee2affd700 1 -- 172.21.15.69:0/3329955526 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee20011b20 con 0x7fee2c1060b0 2024-05-09T09:09:22.741 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 -- 172.21.15.69:0/3329955526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 msgr2=0x7fee2c1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 --2- 172.21.15.69:0/3329955526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fee200098a0 tx=0x7fee20009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 -- 172.21.15.69:0/3329955526 shutdown_connections 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 --2- 172.21.15.69:0/3329955526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c1064d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 -- 172.21.15.69:0/3329955526 >> 172.21.15.69:0/3329955526 conn(0x7fee2c101a20 msgr2=0x7fee2c103e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 -- 172.21.15.69:0/3329955526 shutdown_connections 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.464+0000 7fee322ed700 1 -- 172.21.15.69:0/3329955526 wait complete. 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee322ed700 1 Processor -- start 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee322ed700 1 -- start start 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee322ed700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee322ed700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fee2c10c3f0 con 0x7fee2c1060b0 2024-05-09T09:09:22.742 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53462/0 (socket says 172.21.15.69:53462) 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee2bfff700 1 -- 172.21.15.69:0/1732381087 learned_addr learned my addr 172.21.15.69:0/1732381087 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.465+0000 7fee2bfff700 1 -- 172.21.15.69:0/1732381087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fee2c19c200 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee2bfff700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee200117e0 tx=0x7fee20011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee20003bf0 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fee2001a910 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fee200249a0 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fee2c19c490 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.466+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fee2c19c8f0 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.467+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 8) ==== 46287+0+0 (secure 0 0 0) 0x7fee20022020 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.467+0000 7fee297fa700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 0x7fee1403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.467+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7fee2004d3a0 con 0x7fee2c1060b0 2024-05-09T09:09:22.743 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.467+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fee2c063980 con 0x7fee2c1060b0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.467+0000 7fee2b7fe700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 0x7fee1403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.468+0000 7fee2b7fe700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 0x7fee1403b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.477+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fee20011e20 con 0x7fee2c1060b0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:21.754+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) -- 0x7fee2c19f5b0 con 0x7fee2c1060b0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.107+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) ==== 88+0+0 (secure 0 0 0) 0x7fee20019c70 con 0x7fee2c1060b0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.108+0000 7fee297fa700 1 -- 172.21.15.69:0/1732381087 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7fee2002d960 con 0x7fee2c1060b0 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.109+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 msgr2=0x7fee1403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.109+0000 7fee322ed700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 0x7fee1403b510 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fee1c006fd0 tx=0x7fee1c006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.109+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 msgr2=0x7fee2c19bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.109+0000 7fee322ed700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fee200117e0 tx=0x7fee20011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 shutdown_connections 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7fee14039050 0x7fee1403b510 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 --2- 172.21.15.69:0/1732381087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fee2c1060b0 0x7fee2c19bcc0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:22.744 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 >> 172.21.15.69:0/1732381087 conn(0x7fee2c101a20 msgr2=0x7fee2c1024c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:22.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 shutdown_connections 2024-05-09T09:09:22.745 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:22.110+0000 7fee322ed700 1 -- 172.21.15.69:0/1732381087 wait complete. 2024-05-09T09:09:22.991 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:22.902+0000 7fb252bfb200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T09:09:22.991 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:22.992+0000 7fb252bfb200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T09:09:23.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:23 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1732381087' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-05-09T09:09:23.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:23 smithi069 ceph-mon[30123]: mgrmap e9: x(active, since 19s) 2024-05-09T09:09:24.087 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "available": true, 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.163+0000 7f3cb6f33700 1 Processor -- start 2024-05-09T09:09:24.088 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb6f33700 1 -- start start 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb6f33700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb6f33700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb007e040 con 0x7f3cb007c310 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb4ccf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb4ccf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53508/0 (socket says 172.21.15.69:53508) 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb4ccf700 1 -- 172.21.15.69:0/2582519164 learned_addr learned my addr 172.21.15.69:0/2582519164 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb4ccf700 1 -- 172.21.15.69:0/2582519164 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb007cc70 con 0x7f3cb007c310 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.164+0000 7f3cb4ccf700 1 --2- 172.21.15.69:0/2582519164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40098a0 tx=0x7f3ca4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68ac46fe6a2274ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:24.089 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3caf7fe700 1 -- 172.21.15.69:0/2582519164 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ca4004160 con 0x7f3cb007c310 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3caf7fe700 1 -- 172.21.15.69:0/2582519164 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ca4004300 con 0x7f3cb007c310 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3caf7fe700 1 -- 172.21.15.69:0/2582519164 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ca4011b20 con 0x7f3cb007c310 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 -- 172.21.15.69:0/2582519164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 msgr2=0x7f3cb007c730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/2582519164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40098a0 tx=0x7f3ca4009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 -- 172.21.15.69:0/2582519164 shutdown_connections 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/2582519164 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb007c730 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 -- 172.21.15.69:0/2582519164 >> 172.21.15.69:0/2582519164 conn(0x7f3cb0103380 msgr2=0x7f3cb01057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:24.090 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 -- 172.21.15.69:0/2582519164 shutdown_connections 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.165+0000 7f3cb6f33700 1 -- 172.21.15.69:0/2582519164 wait complete. 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb6f33700 1 Processor -- start 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb6f33700 1 -- start start 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb6f33700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb6f33700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3cb010e4c0 con 0x7f3cb007c310 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb4ccf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb4ccf700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53510/0 (socket says 172.21.15.69:53510) 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.166+0000 7f3cb4ccf700 1 -- 172.21.15.69:0/3173674207 learned_addr learned my addr 172.21.15.69:0/3173674207 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cb4ccf700 1 -- 172.21.15.69:0/3173674207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3cb019e4f0 con 0x7f3cb007c310 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cb4ccf700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40117e0 tx=0x7f3ca4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:24.091 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ca4003ba0 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3ca401a910 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3cb019e780 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f3ca40248f0 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.167+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3cb019ebe0 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f3ca4022020 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3cb0063980 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3cadffb700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 0x7f3ca003b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f3ca404d4c0 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3caffff700 1 -- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 msgr2=0x7f3ca003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.168+0000 7f3caffff700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 0x7f3ca003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.173+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3ca400b3e0 con 0x7f3cb007c310 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.369+0000 7f3caffff700 1 -- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 msgr2=0x7f3ca003b5c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.369+0000 7f3caffff700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 0x7f3ca003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T09:09:24.092 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.471+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) -- 0x7f3cb01a1920 con 0x7f3cb007c310 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.472+0000 7f3cadffb700 1 -- 172.21.15.69:0/3173674207 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) ==== 56+0+88 (secure 0 0 0) 0x7f3ca4020c10 con 0x7f3cb007c310 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 msgr2=0x7f3ca003b5c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 0x7f3ca003b5c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 msgr2=0x7f3cb019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f3ca40117e0 tx=0x7f3ca4011ba0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 shutdown_connections 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f3ca0039100 0x7f3ca003b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 --2- 172.21.15.69:0/3173674207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3cb007c310 0x7f3cb019dfb0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 >> 172.21.15.69:0/3173674207 conn(0x7f3cb0103380 msgr2=0x7f3cb0103e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 shutdown_connections 2024-05-09T09:09:24.093 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:23.474+0000 7f3cb6f33700 1 -- 172.21.15.69:0/3173674207 wait complete. 2024-05-09T09:09:24.094 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for the mgr to restart... 2024-05-09T09:09:24.094 INFO:teuthology.orchestra.run.smithi069.stdout:Waiting for mgr epoch 9... 2024-05-09T09:09:24.378 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:24 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3173674207' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-05-09T09:09:24.379 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:24.205+0000 7fb252bfb200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T09:09:24.689 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:24.384+0000 7fb252bfb200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T09:09:24.689 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:24.562+0000 7fb252bfb200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T09:09:24.947 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:24.854+0000 7fb252bfb200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T09:09:24.947 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:24.948+0000 7fb252bfb200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T09:09:25.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:25.152+0000 7fb252bfb200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T09:09:25.759 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:25.347+0000 7fb252bfb200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:09:26.259 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:25.949+0000 7fb252bfb200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T09:09:26.677 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:26.375+0000 7fb252bfb200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T09:09:26.677 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:26.474+0000 7fb252bfb200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T09:09:26.677 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:26.677+0000 7fb252bfb200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T09:09:27.493 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.200+0000 7fb252bfb200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T09:09:27.494 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.293+0000 7fb252bfb200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T09:09:27.759 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.537+0000 7fb252bfb200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T09:09:27.759 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.671+0000 7fb252bfb200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T09:09:28.189 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.768+0000 7fb252bfb200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T09:09:28.189 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:27 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:27.887+0000 7fb252bfb200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T09:09:28.189 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:28 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:28.189+0000 7fb252bfb200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T09:09:29.069 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:28 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:28.799+0000 7fb252bfb200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T09:09:29.069 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:09:29 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:09:29.017+0000 7fb252bfb200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T09:09:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: Active manager daemon x restarted 2024-05-09T09:09:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: Activating manager daemon x 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: osdmap e3: 0 total, 0 up, 0 in 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: mgrmap e10: x(active, starting, since 0.00343979s) 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: Manager daemon x is now available 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-05-09T09:09:30.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-05-09T09:09:30.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:30 smithi069 ceph-mon[30123]: mgrmap e11: x(active, since 1.00583s) 2024-05-09T09:09:30.919 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout { 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout "initialized": true 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout } 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.484+0000 7f9efaa22700 1 Processor -- start 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9efaa22700 1 -- start start 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9efaa22700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9efaa22700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef407be40 con 0x7f9ef4108880 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53512/0 (socket says 172.21.15.69:53512) 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef3fff700 1 -- 172.21.15.69:0/2039605633 learned_addr learned my addr 172.21.15.69:0/2039605633 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef3fff700 1 -- 172.21.15.69:0/2039605633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef410b1b0 con 0x7f9ef4108880 2024-05-09T09:09:30.920 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef3fff700 1 --2- 172.21.15.69:0/2039605633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40098a0 tx=0x7f9ee4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8105786218314237 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef2ffd700 1 -- 172.21.15.69:0/2039605633 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ee4004160 con 0x7f9ef4108880 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef2ffd700 1 -- 172.21.15.69:0/2039605633 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee4004300 con 0x7f9ef4108880 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.485+0000 7f9ef2ffd700 1 -- 172.21.15.69:0/2039605633 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ee4011b20 con 0x7f9ef4108880 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 -- 172.21.15.69:0/2039605633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 msgr2=0x7f9ef410ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2039605633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40098a0 tx=0x7f9ee4009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 -- 172.21.15.69:0/2039605633 shutdown_connections 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2039605633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef410ac70 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 -- 172.21.15.69:0/2039605633 >> 172.21.15.69:0/2039605633 conn(0x7f9ef4102290 msgr2=0x7f9ef41046b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 -- 172.21.15.69:0/2039605633 shutdown_connections 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.486+0000 7f9efaa22700 1 -- 172.21.15.69:0/2039605633 wait complete. 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9efaa22700 1 Processor -- start 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9efaa22700 1 -- start start 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9efaa22700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:30.921 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9efaa22700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9ef410c1c0 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9ef3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53516/0 (socket says 172.21.15.69:53516) 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9ef3fff700 1 -- 172.21.15.69:0/2543764054 learned_addr learned my addr 172.21.15.69:0/2543764054 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9ef3fff700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9ef419c1b0 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.487+0000 7f9ef3fff700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40117e0 tx=0x7f9ee4011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.488+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ee4003bf0 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.488+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ee401a910 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.488+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f9ee4024970 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.488+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9ef419c440 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.488+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9ef419c8a0 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 9) ==== 46300+0+0 (secure 0 0 0) 0x7f9ee4022020 con 0x7f9ef4108880 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef17fa700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:30.922 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9edc03bc20 con 0x7f9edc039050 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(2..2 src has 1..2) ==== 940+0+0 (secure 0 0 0) 0x7f9ee404d590 con 0x7f9ef4108880 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef37fe700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.489+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.690+0000 7f9ef37fe700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:24.690+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:25.091+0000 7f9ef37fe700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:25.091+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:25.892+0000 7f9ef37fe700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:25.892+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:27.493+0000 7f9ef37fe700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.69:6800/936285351 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:27.493+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-05-09T09:09:30.923 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:29.020+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mgrmap(e 10) ==== 46067+0+0 (secure 0 0 0) 0x7f9ee4024c20 con 0x7f9ef4108880 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:29.020+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 msgr2=0x7f9edc03b510 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:29.020+0000 7f9ef17fa700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.022+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7f9ee404f2e0 con 0x7f9ef4108880 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.022+0000 7f9ef17fa700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 0x7f9edc03ef00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.022+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9ee4016e70 con 0x7f9edc03cb10 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.022+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 0x7f9edc03ef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.023+0000 7f9ef37fe700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 0x7f9edc03ef00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8003a10 tx=0x7f9ee80092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.024+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== command_reply(tid 0: 0 ) ==== 8+0+7774 (secure 0 0 0) 0x7f9ee4016e70 con 0x7f9edc03cb10 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.031+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- command(tid 1: {"prefix": "mgr_status"}) -- 0x7f9ef4063980 con 0x7f9edc03cb10 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9ef17fa700 1 -- 172.21.15.69:0/2543764054 <== mgr.14152 v2:172.21.15.69:6800/3550836912 2 ==== command_reply(tid 1: 0 ) ==== 8+0+52 (secure 0 0 0) 0x7f9ef4063980 con 0x7f9edc03cb10 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 msgr2=0x7f9edc03ef00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 0x7f9edc03ef00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f9ee8003a10 tx=0x7f9ee80092b0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.924 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 msgr2=0x7f9ef419bc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f9ee40117e0 tx=0x7f9ee4011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 shutdown_connections 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9ef4108880 0x7f9ef419bc70 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9edc03cb10 0x7f9edc03ef00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 --2- 172.21.15.69:0/2543764054 >> [v2:172.21.15.69:6800/936285351,v1:172.21.15.69:6801/936285351] conn(0x7f9edc039050 0x7f9edc03b510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 >> 172.21.15.69:0/2543764054 conn(0x7f9ef4102290 msgr2=0x7f9ef4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 shutdown_connections 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:30.032+0000 7f9efaa22700 1 -- 172.21.15.69:0/2543764054 wait complete. 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:mgr epoch 9 is available 2024-05-09T09:09:30.925 INFO:teuthology.orchestra.run.smithi069.stdout:Generating a dashboard self-signed certificate... 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: [09/May/2024:09:09:29] ENGINE Bus STARTING 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: [09/May/2024:09:09:29] ENGINE Serving on https://172.21.15.69:7150 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: [09/May/2024:09:09:29] ENGINE Serving on http://172.21.15.69:8765 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: [09/May/2024:09:09:29] ENGINE Bus STARTED 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-05-09T09:09:31.238 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:32.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-05-09T09:09:32.182 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.341+0000 7fd7f8a83700 1 Processor -- start 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.341+0000 7fd7f8a83700 1 -- start start 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.341+0000 7fd7f8a83700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f8a83700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f41076b0 con 0x7fd7f4106cc0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42430/0 (socket says 172.21.15.69:42430) 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f259c700 1 -- 172.21.15.69:0/1790598826 learned_addr learned my addr 172.21.15.69:0/1790598826 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f259c700 1 -- 172.21.15.69:0/1790598826 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f4107ec0 con 0x7fd7f4106cc0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f259c700 1 --2- 172.21.15.69:0/1790598826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400bb40 tx=0x7fd7e400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2ede66935ac7d483 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f159a700 1 -- 172.21.15.69:0/1790598826 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7e400dbe0 con 0x7fd7f4106cc0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f159a700 1 -- 172.21.15.69:0/1790598826 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e4004510 con 0x7fd7f4106cc0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.342+0000 7fd7f159a700 1 -- 172.21.15.69:0/1790598826 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7e4010470 con 0x7fd7f4106cc0 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 -- 172.21.15.69:0/1790598826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 msgr2=0x7fd7f41070e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/1790598826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fd7e400bb40 tx=0x7fd7e400be50 comp rx=0 tx=0).stop 2024-05-09T09:09:32.183 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 -- 172.21.15.69:0/1790598826 shutdown_connections 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/1790598826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f41070e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 -- 172.21.15.69:0/1790598826 >> 172.21.15.69:0/1790598826 conn(0x7fd7f41022c0 msgr2=0x7fd7f41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 -- 172.21.15.69:0/1790598826 shutdown_connections 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.343+0000 7fd7f8a83700 1 -- 172.21.15.69:0/1790598826 wait complete. 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f8a83700 1 Processor -- start 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f8a83700 1 -- start start 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f8a83700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f8a83700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd7f407b850 con 0x7fd7f4106cc0 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42436/0 (socket says 172.21.15.69:42436) 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f259c700 1 -- 172.21.15.69:0/3813842938 learned_addr learned my addr 172.21.15.69:0/3813842938 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f259c700 1 -- 172.21.15.69:0/3813842938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd7f419a0e0 con 0x7fd7f4106cc0 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7f259c700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40046e0 tx=0x7fd7e400ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:32.184 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.344+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7e4027070 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.345+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd7f419a370 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.345+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd7e4004350 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.345+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd7e4022a60 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.345+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7f419d060 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7f4063980 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 11) ==== 46194+0+0 (secure 0 0 0) 0x7fd7e402a070 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7e37fe700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 0x7fd7dc03b440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd7e401f070 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7f1d9b700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 0x7fd7dc03b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.346+0000 7fd7f1d9b700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 0x7fd7dc03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8006fd0 tx=0x7fd7e8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.351+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd7e4015070 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.429+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7fd7e4010aa0 con 0x7fd7f4106cc0 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.559+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) -- 0x7fd7f4062760 con 0x7fd7dc038f80 2024-05-09T09:09:32.185 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.655+0000 7fd7e37fe700 1 -- 172.21.15.69:0/3813842938 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+32 (secure 0 0 0) 0x7fd7f4062760 con 0x7fd7dc038f80 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 msgr2=0x7fd7dc03b440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 0x7fd7dc03b440 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd7e8006fd0 tx=0x7fd7e8006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 msgr2=0x7fd7f4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fd7e40046e0 tx=0x7fd7e400ded0 comp rx=0 tx=0).stop 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 shutdown_connections 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7dc038f80 0x7fd7dc03b440 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 --2- 172.21.15.69:0/3813842938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd7f4106cc0 0x7fd7f4199ba0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 >> 172.21.15.69:0/3813842938 conn(0x7fd7f41022c0 msgr2=0x7fd7f41901e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.657+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 shutdown_connections 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:31.658+0000 7fd7f8a83700 1 -- 172.21.15.69:0/3813842938 wait complete. 2024-05-09T09:09:32.186 INFO:teuthology.orchestra.run.smithi069.stdout:Creating initial admin user... 2024-05-09T09:09:32.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:32 smithi069 ceph-mon[30123]: mgrmap e12: x(active, since 2s) 2024-05-09T09:09:32.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:32 smithi069 ceph-mon[30123]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:32.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:32.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:32.556 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$pQKtb5uakA2rY0SImGnl5uFS96WkWiUzfjNqm3Ty7Lg1rkhwQWFTC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1715245773, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cbe0e4700 1 Processor -- start 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cbe0e4700 1 -- start start 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cbe0e4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cbe0e4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb807c6d0 con 0x7f0cb81088a0 2024-05-09T09:09:34.203 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cb77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cb77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42440/0 (socket says 172.21.15.69:42440) 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.547+0000 7f0cb77fe700 1 -- 172.21.15.69:0/90387091 learned_addr learned my addr 172.21.15.69:0/90387091 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cb77fe700 1 -- 172.21.15.69:0/90387091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb810b1d0 con 0x7f0cb81088a0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cb77fe700 1 --2- 172.21.15.69:0/90387091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80098a0 tx=0x7f0ca8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a617be6a29572e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cb67fc700 1 -- 172.21.15.69:0/90387091 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ca8004160 con 0x7f0cb81088a0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cb67fc700 1 -- 172.21.15.69:0/90387091 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca8004300 con 0x7f0cb81088a0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cb67fc700 1 -- 172.21.15.69:0/90387091 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ca8011b20 con 0x7f0cb81088a0 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/90387091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 msgr2=0x7f0cb810ac90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/90387091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80098a0 tx=0x7f0ca8009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.548+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/90387091 shutdown_connections 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/90387091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb810ac90 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/90387091 >> 172.21.15.69:0/90387091 conn(0x7f0cb81022d0 msgr2=0x7f0cb81046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/90387091 shutdown_connections 2024-05-09T09:09:34.204 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/90387091 wait complete. 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 Processor -- start 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.549+0000 7f0cbe0e4700 1 -- start start 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cbe0e4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cbe0e4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0cb810c1e0 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb77fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42444/0 (socket says 172.21.15.69:42444) 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb77fe700 1 -- 172.21.15.69:0/562042574 learned_addr learned my addr 172.21.15.69:0/562042574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb77fe700 1 -- 172.21.15.69:0/562042574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0cb819c1d0 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb77fe700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80117e0 tx=0x7f0ca8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ca8003bf0 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0ca801a910 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0cb819c460 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f0ca8024970 con 0x7f0cb81088a0 2024-05-09T09:09:34.205 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.550+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0cb819c8c0 con 0x7f0cb81088a0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.551+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f0ca8022020 con 0x7f0cb81088a0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.551+0000 7f0cb4ff9700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 0x7f0ca003b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.551+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f0ca804d4a0 con 0x7f0cb81088a0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.551+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0cb8063980 con 0x7f0cb81088a0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.551+0000 7f0cb6ffd700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 0x7f0ca003b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.552+0000 7f0cb6ffd700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 0x7f0ca003b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cac006fd0 tx=0x7f0cac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.556+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f0ca8011e20 con 0x7f0cb81088a0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:32.791+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) -- 0x7f0cb819f210 con 0x7f0ca00390b0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.016+0000 7f0cb4ff9700 1 -- 172.21.15.69:0/562042574 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+252 (secure 0 0 0) 0x7f0cb819f210 con 0x7f0ca00390b0 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 msgr2=0x7f0ca003b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:34.206 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 0x7f0ca003b570 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f0cac006fd0 tx=0x7f0cac006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 msgr2=0x7f0cb819bc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f0ca80117e0 tx=0x7f0ca8011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 shutdown_connections 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0ca00390b0 0x7f0ca003b570 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 --2- 172.21.15.69:0/562042574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0cb81088a0 0x7f0cb819bc90 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 >> 172.21.15.69:0/562042574 conn(0x7f0cb81022d0 msgr2=0x7f0cb8102da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 shutdown_connections 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:33.019+0000 7f0cbe0e4700 1 -- 172.21.15.69:0/562042574 wait complete. 2024-05-09T09:09:34.207 INFO:teuthology.orchestra.run.smithi069.stdout:Fetching dashboard port number... 2024-05-09T09:09:34.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:34 smithi069 ceph-mon[30123]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:34.321 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:35.401 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:35 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2153425279' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stdout 8443 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.623+0000 7f2aa0955700 1 Processor -- start 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.623+0000 7f2aa0955700 1 -- start start 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2aa0955700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2aa0955700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c07be60 con 0x7f2a9c108870 2024-05-09T09:09:35.409 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42448/0 (socket says 172.21.15.69:42448) 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9a59c700 1 -- 172.21.15.69:0/3244998941 learned_addr learned my addr 172.21.15.69:0/3244998941 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9a59c700 1 -- 172.21.15.69:0/3244998941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c10b1a0 con 0x7f2a9c108870 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9a59c700 1 --2- 172.21.15.69:0/3244998941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0098a0 tx=0x7f2a8c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6da4ff8f20b47e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9959a700 1 -- 172.21.15.69:0/3244998941 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a8c004160 con 0x7f2a9c108870 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.624+0000 7f2a9959a700 1 -- 172.21.15.69:0/3244998941 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c004300 con 0x7f2a9c108870 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2a9959a700 1 -- 172.21.15.69:0/3244998941 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a8c011b20 con 0x7f2a9c108870 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 -- 172.21.15.69:0/3244998941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 msgr2=0x7f2a9c10ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 --2- 172.21.15.69:0/3244998941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0098a0 tx=0x7f2a8c009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 -- 172.21.15.69:0/3244998941 shutdown_connections 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 --2- 172.21.15.69:0/3244998941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c10ac60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 -- 172.21.15.69:0/3244998941 >> 172.21.15.69:0/3244998941 conn(0x7f2a9c1022a0 msgr2=0x7f2a9c1046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:35.410 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 -- 172.21.15.69:0/3244998941 shutdown_connections 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.625+0000 7f2aa0955700 1 -- 172.21.15.69:0/3244998941 wait complete. 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2aa0955700 1 Processor -- start 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2aa0955700 1 -- start start 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2aa0955700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2aa0955700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a9c10c200 con 0x7f2a9c108870 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2a9a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2a9a59c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42454/0 (socket says 172.21.15.69:42454) 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2a9a59c700 1 -- 172.21.15.69:0/2153425279 learned_addr learned my addr 172.21.15.69:0/2153425279 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.626+0000 7f2a9a59c700 1 -- 172.21.15.69:0/2153425279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a9c19c1c0 con 0x7f2a9c108870 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2a9a59c700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0117e0 tx=0x7f2a8c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a8c003bf0 con 0x7f2a9c108870 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2a8c01a910 con 0x7f2a9c108870 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a9c19c450 con 0x7f2a9c108870 2024-05-09T09:09:35.411 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f2a8c024970 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.627+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a9c19c8b0 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f2a8c022020 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2a8b7fe700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 0x7f2a8403b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f2a8c04d690 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a9c063980 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2a99d9b700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 0x7f2a8403b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.628+0000 7f2a99d9b700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 0x7f2a8403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2a90006fd0 tx=0x7f2a90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.633+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2a8c011e20 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.829+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) -- 0x7f2a9c19f300 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.829+0000 7f2a8b7fe700 1 -- 172.21.15.69:0/2153425279 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) ==== 112+0+5 (secure 0 0 0) 0x7f2a8c00ba00 con 0x7f2a9c108870 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 msgr2=0x7f2a8403b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 0x7f2a8403b510 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f2a90006fd0 tx=0x7f2a90006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 msgr2=0x7f2a9c19bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f2a8c0117e0 tx=0x7f2a8c011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:35.412 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 shutdown_connections 2024-05-09T09:09:35.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2a84039050 0x7f2a8403b510 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:35.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 --2- 172.21.15.69:0/2153425279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a9c108870 0x7f2a9c19bc80 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:35.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 >> 172.21.15.69:0/2153425279 conn(0x7f2a9c1022a0 msgr2=0x7f2a9c102d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:35.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 shutdown_connections 2024-05-09T09:09:35.413 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:34.831+0000 7f2aa0955700 1 -- 172.21.15.69:0/2153425279 wait complete. 2024-05-09T09:09:35.418 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-05-09T09:09:35.418 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout disabled 2024-05-09T09:09:35.426 INFO:teuthology.orchestra.run.smithi069.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-05-09T09:09:35.426 INFO:teuthology.orchestra.run.smithi069.stdout:systemctl: stdout inactive 2024-05-09T09:09:35.426 INFO:teuthology.orchestra.run.smithi069.stdout:firewalld.service is not enabled 2024-05-09T09:09:35.426 INFO:teuthology.orchestra.run.smithi069.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout:Ceph Dashboard is now available at: 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout: URL: https://smithi069.front.sepia.ceph.com:8443/ 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout: User: admin 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout: Password: 6hn9ohthtz 2024-05-09T09:09:35.428 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:35.429 INFO:teuthology.orchestra.run.smithi069.stdout:Saving cluster configuration to /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config directory 2024-05-09T09:09:35.429 INFO:teuthology.orchestra.run.smithi069.stdout:Enabling autotune for osd_memory_target 2024-05-09T09:09:36.615 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766b272700 1 Processor -- start 2024-05-09T09:09:36.615 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766b272700 1 -- start start 2024-05-09T09:09:36.615 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766b272700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:36.615 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766b272700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f766407c710 con 0x7f7664108840 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766900e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766900e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42456/0 (socket says 172.21.15.69:42456) 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.840+0000 7f766900e700 1 -- 172.21.15.69:0/2164172194 learned_addr learned my addr 172.21.15.69:0/2164172194 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.841+0000 7f766900e700 1 -- 172.21.15.69:0/2164172194 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f766410b170 con 0x7f7664108840 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.841+0000 7f766900e700 1 --2- 172.21.15.69:0/2164172194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f76600098a0 tx=0x7f7660009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7cd9ac5d7a82e7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.841+0000 7f765bfff700 1 -- 172.21.15.69:0/2164172194 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7660004160 con 0x7f7664108840 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.841+0000 7f765bfff700 1 -- 172.21.15.69:0/2164172194 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7660004300 con 0x7f7664108840 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.841+0000 7f765bfff700 1 -- 172.21.15.69:0/2164172194 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7660011b20 con 0x7f7664108840 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- 172.21.15.69:0/2164172194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 msgr2=0x7f766410ac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 --2- 172.21.15.69:0/2164172194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f76600098a0 tx=0x7f7660009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- 172.21.15.69:0/2164172194 shutdown_connections 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 --2- 172.21.15.69:0/2164172194 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f766410ac30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:36.616 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- 172.21.15.69:0/2164172194 >> 172.21.15.69:0/2164172194 conn(0x7f76641022d0 msgr2=0x7f76641046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- 172.21.15.69:0/2164172194 shutdown_connections 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- 172.21.15.69:0/2164172194 wait complete. 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 Processor -- start 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- start start 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.842+0000 7f766b272700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f766410c170 con 0x7f7664108840 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766900e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766900e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42458/0 (socket says 172.21.15.69:42458) 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766900e700 1 -- 172.21.15.69:0/809276501 learned_addr learned my addr 172.21.15.69:0/809276501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766900e700 1 -- 172.21.15.69:0/809276501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7664108600 con 0x7f7664108840 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766900e700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f76600117e0 tx=0x7f7660011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7660003bf0 con 0x7f7664108840 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f766001a910 con 0x7f7664108840 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f7660024970 con 0x7f7664108840 2024-05-09T09:09:36.617 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76641067f0 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.843+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7664106cb0 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.844+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 12) ==== 46300+0+0 (secure 0 0 0) 0x7f7660022020 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.844+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7664063980 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.844+0000 7f765a7fc700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 0x7f765003b510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.844+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f766004d600 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.845+0000 7f766880d700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 0x7f765003b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.845+0000 7f766880d700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 0x7f765003b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7654006fd0 tx=0x7f7654006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:35.849+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7660011e20 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.018+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f7660024cd0 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.047+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) -- 0x7f7664107690 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.047+0000 7f765a7fc700 1 -- 172.21.15.69:0/809276501 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) ==== 127+0+0 (secure 0 0 0) 0x7f766002d3f0 con 0x7f7664108840 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 msgr2=0x7f765003b510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:36.618 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 0x7f765003b510 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f7654006fd0 tx=0x7f7654006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 msgr2=0x7f76641080c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f76600117e0 tx=0x7f7660011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 shutdown_connections 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7650039050 0x7f765003b510 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 --2- 172.21.15.69:0/809276501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7664108840 0x7f76641080c0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.049+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 >> 172.21.15.69:0/809276501 conn(0x7f76641022d0 msgr2=0x7f7664102dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.050+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 shutdown_connections 2024-05-09T09:09:36.619 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.050+0000 7f766b272700 1 -- 172.21.15.69:0/809276501 wait complete. 2024-05-09T09:09:37.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:37 smithi069 ceph-mon[30123]: mgrmap e13: x(active, since 7s) 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d9ddb1700 1 Processor -- start 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d9ddb1700 1 -- start start 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d9ddb1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d9ddb1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d981087c0 con 0x7f1d98107dd0 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42462/0 (socket says 172.21.15.69:42462) 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d977fe700 1 -- 172.21.15.69:0/2355542135 learned_addr learned my addr 172.21.15.69:0/2355542135 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d977fe700 1 -- 172.21.15.69:0/2355542135 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d98108fd0 con 0x7f1d98107dd0 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.963+0000 7f1d977fe700 1 --2- 172.21.15.69:0/2355542135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d800098a0 tx=0x7f1d80009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16fa52cd3cbb7120 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:37.822 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d967fc700 1 -- 172.21.15.69:0/2355542135 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d80004160 con 0x7f1d98107dd0 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d967fc700 1 -- 172.21.15.69:0/2355542135 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d80004300 con 0x7f1d98107dd0 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d967fc700 1 -- 172.21.15.69:0/2355542135 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d80011b20 con 0x7f1d98107dd0 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/2355542135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 msgr2=0x7f1d981081f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/2355542135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f1d800098a0 tx=0x7f1d80009bb0 comp rx=0 tx=0).stop 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/2355542135 shutdown_connections 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/2355542135 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d981081f0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/2355542135 >> 172.21.15.69:0/2355542135 conn(0x7f1d981033b0 msgr2=0x7f1d981057d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/2355542135 shutdown_connections 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.964+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/2355542135 wait complete. 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d9ddb1700 1 Processor -- start 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d9ddb1700 1 -- start start 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d9ddb1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d9ddb1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1d9810c380 con 0x7f1d98107dd0 2024-05-09T09:09:37.823 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d977fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42474/0 (socket says 172.21.15.69:42474) 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.965+0000 7f1d977fe700 1 -- 172.21.15.69:0/750418439 learned_addr learned my addr 172.21.15.69:0/750418439 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d977fe700 1 -- 172.21.15.69:0/750418439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1d9819c1e0 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d977fe700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d800117e0 tx=0x7f1d80011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d80003bf0 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d8001a910 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1d9819c470 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1d800249a0 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.966+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1d9819c8d0 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.967+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1d80022020 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.967+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1d98063980 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.967+0000 7f1d94ff9700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 0x7f1d8403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.967+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1d8004d770 con 0x7f1d98107dd0 2024-05-09T09:09:37.824 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.967+0000 7f1d96ffd700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 0x7f1d8403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.968+0000 7f1d96ffd700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 0x7f1d8403b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d88006fd0 tx=0x7f1d88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:36.972+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1d80011e20 con 0x7f1d98107dd0 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.281+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) -- 0x7f1d9819f400 con 0x7f1d98107dd0 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.283+0000 7f1d94ff9700 1 -- 172.21.15.69:0/750418439 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) ==== 153+0+0 (secure 0 0 0) 0x7f1d8000ba00 con 0x7f1d98107dd0 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 msgr2=0x7f1d8403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 0x7f1d8403b5b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f1d88006fd0 tx=0x7f1d88006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 msgr2=0x7f1d9819bca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1d800117e0 tx=0x7f1d80011c10 comp rx=0 tx=0).stop 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 shutdown_connections 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d840390f0 0x7f1d8403b5b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 --2- 172.21.15.69:0/750418439 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1d98107dd0 0x7f1d9819bca0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:37.825 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 >> 172.21.15.69:0/750418439 conn(0x7f1d981033b0 msgr2=0x7f1d98103e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 shutdown_connections 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr 2024-05-09T09:09:37.286+0000 7f1d9ddb1700 1 -- 172.21.15.69:0/750418439 wait complete. 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:Or, if you are only running a single cluster on this host: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout:Please consider enabling telemetry to help improve Ceph: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.826 INFO:teuthology.orchestra.run.smithi069.stdout: ceph telemetry on 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout:For more information see: 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:37.827 INFO:teuthology.orchestra.run.smithi069.stdout:Bootstrap complete. 2024-05-09T09:09:37.855 INFO:tasks.cephadm:Fetching config... 2024-05-09T09:09:37.856 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:09:37.856 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-05-09T09:09:37.872 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-05-09T09:09:37.872 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:09:37.873 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-05-09T09:09:37.929 INFO:tasks.cephadm:Fetching mon keyring... 2024-05-09T09:09:37.929 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:09:37.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/keyring of=/dev/stdout 2024-05-09T09:09:37.994 INFO:tasks.cephadm:Fetching pub ssh key... 2024-05-09T09:09:37.994 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:09:37.995 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-05-09T09:09:38.050 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-05-09T09:09:38.050 DEBUG:teuthology.orchestra.run.smithi069:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T09:09:38.125 INFO:teuthology.orchestra.run.smithi069.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:09:38.138 DEBUG:teuthology.orchestra.run.smithi103:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T09:09:38.180 INFO:teuthology.orchestra.run.smithi103.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:09:38.191 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-05-09T09:09:38.232 INFO:teuthology.orchestra.run.smithi163.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC3BPXpp+qOiQsqCLDIPxc3d2zGg7AJgRX05+1uhzRbSg9xcmV+pu6gZTPN5BeUBf40gcLzT9pFrA+eYEA5gPRkNJUsvG5rKNSrMMgsSIsLmH1RDa2ixPgsqzXD6CqSifs3Q5V0sdnqMi5Zp/jAHDmAg56RIF37vkeSfp+AWZGpBOKTelzu4rbxL/bxciyZumTbQa3YO2znjXi6yHbOj+rAVMUWX4sYxn4k5a5dQatY5YboWPlWLKY6YYSvZZS2BRrPmSw4zds20/4CoI375kWkqtAsPmW/EbRcJuxt1VgI+4+qhudPIzBXnWtpp2CcXIzK0EUB4w4FA+ovtq1j9c4Ivrc+i/Xkrh5Z5KkqhgtcQUXV23RCCSRL9GwvnGNjMBwZ7RfJx+5/55X56UUhipSCqlGoRrR5u/qcmTI1fiH/SaXkyhOkMA/H351mQx6hcA6VSrIWiW3NoESleJJ4/cjHOBafrn0dsqogsaxKRhspMqNqQfTJC92IOMHfPKhU5jM= ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:09:38.244 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-05-09T09:09:38.500 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:09:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/750418439' entity='client.admin' 2024-05-09T09:09:40.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 -- 172.21.15.69:0/1602408792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 msgr2=0x7fb9501010c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:40.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 --2- 172.21.15.69:0/1602408792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501010c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb940009910 tx=0x7fb940009c20 comp rx=0 tx=0).stop 2024-05-09T09:09:40.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 -- 172.21.15.69:0/1602408792 shutdown_connections 2024-05-09T09:09:40.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 --2- 172.21.15.69:0/1602408792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501010c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 -- 172.21.15.69:0/1602408792 >> 172.21.15.69:0/1602408792 conn(0x7fb9500fc260 msgr2=0x7fb9500fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 -- 172.21.15.69:0/1602408792 shutdown_connections 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 -- 172.21.15.69:0/1602408792 wait complete. 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.882+0000 7fb955afa700 1 Processor -- start 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb955afa700 1 -- start start 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb955afa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb955afa700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb950106d30 con 0x7fb950100ca0 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94f7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57336/0 (socket says 172.21.15.69:57336) 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94f7fe700 1 -- 172.21.15.69:0/543907813 learned_addr learned my addr 172.21.15.69:0/543907813 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94f7fe700 1 -- 172.21.15.69:0/543907813 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb950197a00 con 0x7fb950100ca0 2024-05-09T09:09:40.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94f7fe700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb940006010 tx=0x7fb940005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:40.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb940027070 con 0x7fb950100ca0 2024-05-09T09:09:40.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb94000be40 con 0x7fb950100ca0 2024-05-09T09:09:40.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb950197c90 con 0x7fb950100ca0 2024-05-09T09:09:40.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.883+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb94000f460 con 0x7fb950100ca0 2024-05-09T09:09:40.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.884+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb95019a980 con 0x7fb950100ca0 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.884+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb94002a070 con 0x7fb950100ca0 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.884+0000 7fb94cff9700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 0x7fb93803b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.885+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb94001e030 con 0x7fb950100ca0 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.885+0000 7fb94effd700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 0x7fb93803b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.885+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb9501074b0 con 0x7fb950100ca0 2024-05-09T09:09:40.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.885+0000 7fb94effd700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 0x7fb93803b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb944006fd0 tx=0x7fb944006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:40.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:40.890+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb940031070 con 0x7fb950100ca0 2024-05-09T09:09:41.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:41.042 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:41.043 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi069", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:09:41.043 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:41.043 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:41.043 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:41.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.092+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) -- 0x7fb95002dbf0 con 0x7fb950100ca0 2024-05-09T09:09:41.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.094+0000 7fb94cff9700 1 -- 172.21.15.69:0/543907813 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) ==== 125+0+0 (secure 0 0 0) 0x7fb940013070 con 0x7fb950100ca0 2024-05-09T09:09:41.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 msgr2=0x7fb93803b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 0x7fb93803b610 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb944006fd0 tx=0x7fb944006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 msgr2=0x7fb9501974c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fb940006010 tx=0x7fb940005050 comp rx=0 tx=0).stop 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 shutdown_connections 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb938039150 0x7fb93803b610 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 --2- 172.21.15.69:0/543907813 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb950100ca0 0x7fb9501974c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 >> 172.21.15.69:0/543907813 conn(0x7fb9500fc260 msgr2=0x7fb95006a610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 shutdown_connections 2024-05-09T09:09:41.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:41.097+0000 7fb955afa700 1 -- 172.21.15.69:0/543907813 wait complete. 2024-05-09T09:09:41.782 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-05-09T09:09:41.783 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-05-09T09:09:42.354 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:09:42.364 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:42 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/543907813' entity='client.admin' 2024-05-09T09:09:42.364 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:42.365 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:43.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- 172.21.15.69:0/972362927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 msgr2=0x7f734c104c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:43.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 --2- 172.21.15.69:0/972362927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c104c20 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f733c009910 tx=0x7f733c009c20 comp rx=0 tx=0).stop 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- 172.21.15.69:0/972362927 shutdown_connections 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 --2- 172.21.15.69:0/972362927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c104c20 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- 172.21.15.69:0/972362927 >> 172.21.15.69:0/972362927 conn(0x7f734c0fc260 msgr2=0x7f734c0fe680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- 172.21.15.69:0/972362927 shutdown_connections 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- 172.21.15.69:0/972362927 wait complete. 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 Processor -- start 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.716+0000 7f7352725700 1 -- start start 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f7352725700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f7352725700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f734c107dd0 con 0x7f734c102830 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f734bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f734bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57346/0 (socket says 172.21.15.69:57346) 2024-05-09T09:09:43.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f734bfff700 1 -- 172.21.15.69:0/504937691 learned_addr learned my addr 172.21.15.69:0/504937691 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:43.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f734bfff700 1 -- 172.21.15.69:0/504937691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f734c1025f0 con 0x7f734c102830 2024-05-09T09:09:43.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f734bfff700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f733c005b40 tx=0x7f733c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f733c027070 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f733c00be40 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f734c1007e0 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.717+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f733c00f460 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.718+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f734c100d20 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.718+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f733c02a070 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.718+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f734c108550 con 0x7f734c102830 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.719+0000 7f73497fa700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 0x7f733403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:43.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.719+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f733c01e030 con 0x7f734c102830 2024-05-09T09:09:43.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.719+0000 7f734b7fe700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 0x7f733403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:43.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.719+0000 7f734b7fe700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 0x7f733403b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7340006fd0 tx=0x7f7340006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:43.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.723+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f733c013070 con 0x7f734c102830 2024-05-09T09:09:43.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.919+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) -- 0x7f734c190970 con 0x7f73340390f0 2024-05-09T09:09:43.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.923+0000 7f73497fa700 1 -- 172.21.15.69:0/504937691 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f734c190970 con 0x7f73340390f0 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 msgr2=0x7f733403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 0x7f733403b5b0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f7340006fd0 tx=0x7f7340006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 msgr2=0x7f734c1020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f733c005b40 tx=0x7f733c005030 comp rx=0 tx=0).stop 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 shutdown_connections 2024-05-09T09:09:43.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f73340390f0 0x7f733403b5b0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:43.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 --2- 172.21.15.69:0/504937691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f734c102830 0x7f734c1020b0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:43.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 >> 172.21.15.69:0/504937691 conn(0x7f734c0fc260 msgr2=0x7f734c0fccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:43.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 shutdown_connections 2024-05-09T09:09:43.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:43.925+0000 7f7352725700 1 -- 172.21.15.69:0/504937691 wait complete. 2024-05-09T09:09:44.662 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi103 2024-05-09T09:09:44.662 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:09:44.662 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.conf 2024-05-09T09:09:44.681 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:09:44.681 DEBUG:teuthology.orchestra.run.smithi103:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:09:44.738 INFO:tasks.cephadm:Adding host smithi103 to orchestrator... 2024-05-09T09:09:44.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch host add smithi103 2024-05-09T09:09:44.987 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: Updating smithi069:/etc/ceph/ceph.conf 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:45.216 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:45.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:45 smithi069 ceph-mon[30123]: Updating smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:09:45.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:45 smithi069 ceph-mon[30123]: Updating smithi069:/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:09:45.928 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:45 smithi069 ceph-mon[30123]: Updating smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T09:09:46.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.452+0000 7fc4602bc700 1 -- 172.21.15.69:0/2575047527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 msgr2=0x7fc458107130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:46.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.452+0000 7fc4602bc700 1 --2- 172.21.15.69:0/2575047527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc458107130 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc44c009910 tx=0x7fc44c009c20 comp rx=0 tx=0).stop 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- 172.21.15.69:0/2575047527 shutdown_connections 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 --2- 172.21.15.69:0/2575047527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc458107130 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- 172.21.15.69:0/2575047527 >> 172.21.15.69:0/2575047527 conn(0x7fc4580fba90 msgr2=0x7fc4580fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- 172.21.15.69:0/2575047527 shutdown_connections 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- 172.21.15.69:0/2575047527 wait complete. 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 Processor -- start 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- start start 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.453+0000 7fc4602bc700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc45810c3a0 con 0x7fc458106d50 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc45e058700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc45e058700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57358/0 (socket says 172.21.15.69:57358) 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc45e058700 1 -- 172.21.15.69:0/3758752027 learned_addr learned my addr 172.21.15.69:0/3758752027 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc45e058700 1 -- 172.21.15.69:0/3758752027 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc45819dae0 con 0x7fc458106d50 2024-05-09T09:09:46.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc45e058700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005850 tx=0x7fc44c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:46.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc44c026070 con 0x7fc458106d50 2024-05-09T09:09:46.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc44c00bc30 con 0x7fc458106d50 2024-05-09T09:09:46.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc44c003a20 con 0x7fc458106d50 2024-05-09T09:09:46.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc45819dd70 con 0x7fc458106d50 2024-05-09T09:09:46.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.454+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc458197670 con 0x7fc458106d50 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.455+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc44c029070 con 0x7fc458106d50 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.455+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc458101250 con 0x7fc458106d50 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.455+0000 7fc44b7fe700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 0x7fc44403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.455+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc44c01e070 con 0x7fc458106d50 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.455+0000 7fc45d857700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 0x7fc44403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:46.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.456+0000 7fc45d857700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 0x7fc44403b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc454006fd0 tx=0x7fc454006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:46.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.460+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc44c013070 con 0x7fc458106d50 2024-05-09T09:09:46.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:46.657+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}) -- 0x7fc458198ba0 con 0x7fc444039150 2024-05-09T09:09:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:47 smithi069 ceph-mon[30123]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi103", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:48.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:48 smithi069 ceph-mon[30123]: Deploying cephadm binary to smithi103 2024-05-09T09:09:50.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:50 smithi069 ceph-mon[30123]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:09:50.834 INFO:teuthology.orchestra.run.smithi069.stdout:Added host 'smithi103' with addr '172.21.15.103' 2024-05-09T09:09:50.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.834+0000 7fc44b7fe700 1 -- 172.21.15.69:0/3758752027 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fc458198ba0 con 0x7fc444039150 2024-05-09T09:09:50.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 msgr2=0x7fc44403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:50.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 0x7fc44403b610 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fc454006fd0 tx=0x7fc454006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:50.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 msgr2=0x7fc45819d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc44c005850 tx=0x7fc44c005030 comp rx=0 tx=0).stop 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 shutdown_connections 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc444039150 0x7fc44403b610 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 --2- 172.21.15.69:0/3758752027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc458106d50 0x7fc45819d5a0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 >> 172.21.15.69:0/3758752027 conn(0x7fc4580fba90 msgr2=0x7fc4580fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 shutdown_connections 2024-05-09T09:09:50.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:50.837+0000 7fc4602bc700 1 -- 172.21.15.69:0/3758752027 wait complete. 2024-05-09T09:09:51.525 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T09:09:51.762 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:09:52.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:52.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:51 smithi069 ceph-mon[30123]: Added host smithi103 2024-05-09T09:09:52.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:09:52.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:51 smithi069 ceph-mon[30123]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:09:52.096 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:53.102 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:53.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 -- 172.21.15.69:0/2503160746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 msgr2=0x7f1b94101270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/2503160746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94101270 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f1b84009910 tx=0x7f1b84009c20 comp rx=0 tx=0).stop 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 -- 172.21.15.69:0/2503160746 shutdown_connections 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/2503160746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94101270 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 -- 172.21.15.69:0/2503160746 >> 172.21.15.69:0/2503160746 conn(0x7f1b940fc740 msgr2=0x7f1b940feb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 -- 172.21.15.69:0/2503160746 shutdown_connections 2024-05-09T09:09:53.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.107+0000 7f1b99d7c700 1 -- 172.21.15.69:0/2503160746 wait complete. 2024-05-09T09:09:53.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b99d7c700 1 Processor -- start 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b99d7c700 1 -- start start 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b99d7c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b99d7c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b9410b260 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b937fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b937fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38840/0 (socket says 172.21.15.69:38840) 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.108+0000 7f1b937fe700 1 -- 172.21.15.69:0/1499056281 learned_addr learned my addr 172.21.15.69:0/1499056281 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b937fe700 1 -- 172.21.15.69:0/1499056281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b94198cb0 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b937fe700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005850 tx=0x7f1b84005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b84026070 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b8400bbd0 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b94198fa0 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7f1b84003970 con 0x7f1b94100e90 2024-05-09T09:09:53.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.109+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b94199460 con 0x7f1b94100e90 2024-05-09T09:09:53.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.110+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f1b84029070 con 0x7f1b94100e90 2024-05-09T09:09:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.110+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b94102370 con 0x7f1b94100e90 2024-05-09T09:09:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.110+0000 7f1b90ff9700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 0x7f1b7c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.110+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7f1b8401e070 con 0x7f1b94100e90 2024-05-09T09:09:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.111+0000 7f1b92ffd700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 0x7f1b7c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:53.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.111+0000 7f1b92ffd700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 0x7f1b7c03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1b88006fd0 tx=0x7f1b88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:53.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.115+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b84013070 con 0x7f1b94100e90 2024-05-09T09:09:53.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.316+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b94063980 con 0x7f1b7c039150 2024-05-09T09:09:53.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.317+0000 7f1b90ff9700 1 -- 172.21.15.69:0/1499056281 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+160 (secure 0 0 0) 0x7f1b94063980 con 0x7f1b7c039150 2024-05-09T09:09:53.315 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:09:53.315 INFO:teuthology.orchestra.run.smithi069.stdout:[{"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}] 2024-05-09T09:09:53.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 msgr2=0x7f1b7c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 0x7f1b7c03b610 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1b88006fd0 tx=0x7f1b88006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 msgr2=0x7f1b94198770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1b84005850 tx=0x7f1b84005030 comp rx=0 tx=0).stop 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 shutdown_connections 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b7c039150 0x7f1b7c03b610 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 --2- 172.21.15.69:0/1499056281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b94100e90 0x7f1b94198770 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 >> 172.21.15.69:0/1499056281 conn(0x7f1b940fc740 msgr2=0x7f1b94103cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 shutdown_connections 2024-05-09T09:09:53.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:53.319+0000 7f1b99d7c700 1 -- 172.21.15.69:0/1499056281 wait complete. 2024-05-09T09:09:53.816 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi163 2024-05-09T09:09:53.816 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:09:53.816 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.conf 2024-05-09T09:09:53.835 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:09:53.835 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:09:53.893 INFO:tasks.cephadm:Adding host smithi163 to orchestrator... 2024-05-09T09:09:53.893 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch host add smithi163 2024-05-09T09:09:54.129 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:09:54.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:53 smithi069 ceph-mon[30123]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:09:54.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:55.228 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:54 smithi069 ceph-mon[30123]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 -- 172.21.15.69:0/1813789381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 msgr2=0x7fb430102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 --2- 172.21.15.69:0/1813789381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb430102370 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fb42c00b0a0 tx=0x7fb42c00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 -- 172.21.15.69:0/1813789381 shutdown_connections 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 --2- 172.21.15.69:0/1813789381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb430102370 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 -- 172.21.15.69:0/1813789381 >> 172.21.15.69:0/1813789381 conn(0x7fb4300fd860 msgr2=0x7fb4300ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 -- 172.21.15.69:0/1813789381 shutdown_connections 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.527+0000 7fb437903700 1 -- 172.21.15.69:0/1813789381 wait complete. 2024-05-09T09:09:55.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb437903700 1 Processor -- start 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb437903700 1 -- start start 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb437903700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb437903700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb43010c360 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb43569f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb43569f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38852/0 (socket says 172.21.15.69:38852) 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb43569f700 1 -- 172.21.15.69:0/2771016362 learned_addr learned my addr 172.21.15.69:0/2771016362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb43569f700 1 -- 172.21.15.69:0/2771016362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb43019d940 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.528+0000 7fb43569f700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb42c015040 tx=0x7fb42c003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb42c017040 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb42c022070 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fb42c01d5d0 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb43019dbd0 con 0x7fb430101f90 2024-05-09T09:09:55.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4301975c0 con 0x7fb430101f90 2024-05-09T09:09:55.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.529+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fb42c01d880 con 0x7fb430101f90 2024-05-09T09:09:55.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.530+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb430103470 con 0x7fb430101f90 2024-05-09T09:09:55.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.530+0000 7fb4267fc700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 0x7fb41c03b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:09:55.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.530+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fb42c00f050 con 0x7fb430101f90 2024-05-09T09:09:55.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.530+0000 7fb434e9e700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 0x7fb41c03b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:09:55.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.530+0000 7fb434e9e700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 0x7fb41c03b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:09:55.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.535+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb42c007e20 con 0x7fb430101f90 2024-05-09T09:09:55.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:55.735+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}) -- 0x7fb430198af0 con 0x7fb41c039150 2024-05-09T09:09:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:55 smithi069 ceph-mon[30123]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:09:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:09:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:09:58.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:57 smithi069 ceph-mon[30123]: Deploying cephadm binary to smithi163 2024-05-09T09:09:58.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:57 smithi069 ceph-mon[30123]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-09T09:09:58.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:57 smithi069 ceph-mon[30123]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:09:58.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:09:57 smithi069 ceph-mon[30123]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:09:59.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.865+0000 7fb4267fc700 1 -- 172.21.15.69:0/2771016362 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+49 (secure 0 0 0) 0x7fb430198af0 con 0x7fb41c039150 2024-05-09T09:09:59.865 INFO:teuthology.orchestra.run.smithi069.stdout:Added host 'smithi163' with addr '172.21.15.163' 2024-05-09T09:09:59.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.867+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 msgr2=0x7fb41c03b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.867+0000 7fb437903700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 0x7fb41c03b610 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb420006fd0 tx=0x7fb420006e40 comp rx=0 tx=0).stop 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.867+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 msgr2=0x7fb43019d400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb42c015040 tx=0x7fb42c003d50 comp rx=0 tx=0).stop 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 shutdown_connections 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb41c039150 0x7fb41c03b610 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 --2- 172.21.15.69:0/2771016362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb430101f90 0x7fb43019d400 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 >> 172.21.15.69:0/2771016362 conn(0x7fb4300fd860 msgr2=0x7fb430103960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 shutdown_connections 2024-05-09T09:09:59.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:09:59.868+0000 7fb437903700 1 -- 172.21.15.69:0/2771016362 wait complete. 2024-05-09T09:10:00.189 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:00.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:00.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:00.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:00.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:00.190 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:00.569 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch host ls --format=json 2024-05-09T09:10:00.806 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:10:01.186 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:01 smithi069 ceph-mon[30123]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:01.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:01 smithi069 ceph-mon[30123]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T09:10:01.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:01 smithi069 ceph-mon[30123]: Added host smithi163 2024-05-09T09:10:01.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:01.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:02.343 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:02 smithi069 ceph-mon[30123]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 -- 172.21.15.69:0/3772218612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 msgr2=0x7fd788100910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 --2- 172.21.15.69:0/3772218612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd788100910 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd778009910 tx=0x7fd778009c20 comp rx=0 tx=0).stop 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 -- 172.21.15.69:0/3772218612 shutdown_connections 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 --2- 172.21.15.69:0/3772218612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd788100910 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 -- 172.21.15.69:0/3772218612 >> 172.21.15.69:0/3772218612 conn(0x7fd7880773a0 msgr2=0x7fd7880777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 -- 172.21.15.69:0/3772218612 shutdown_connections 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.363+0000 7fd78d74f700 1 -- 172.21.15.69:0/3772218612 wait complete. 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd78d74f700 1 Processor -- start 2024-05-09T09:10:02.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd78d74f700 1 -- start start 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd78d74f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd78d74f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd78810c390 con 0x7fd788100530 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd786ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd786ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38942/0 (socket says 172.21.15.69:38942) 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd786ffd700 1 -- 172.21.15.69:0/1477339358 learned_addr learned my addr 172.21.15.69:0/1477339358 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:10:02.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd786ffd700 1 -- 172.21.15.69:0/1477339358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd78819d930 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd786ffd700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd778006010 tx=0x7fd778005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd778026070 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd77800bbd0 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.364+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd78819dc20 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.365+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fd778003910 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.365+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd7881975b0 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.365+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fd778029070 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.365+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd7881035f0 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.366+0000 7fd784ff9700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 0x7fd77403b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.366+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fd77801e070 con 0x7fd788100530 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.366+0000 7fd77ffff700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 0x7fd77403b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:02.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.366+0000 7fd77ffff700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 0x7fd77403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:02.370 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.371+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd778013070 con 0x7fd788100530 2024-05-09T09:10:02.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.570+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fd788198ae0 con 0x7fd7740390f0 2024-05-09T09:10:02.571 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.572+0000 7fd784ff9700 1 -- 172.21.15.69:0/1477339358 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+240 (secure 0 0 0) 0x7fd788198ae0 con 0x7fd7740390f0 2024-05-09T09:10:02.571 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:10:02.571 INFO:teuthology.orchestra.run.smithi069.stdout:[{"addr": "172.21.15.69", "hostname": "smithi069", "labels": [], "status": ""}, {"addr": "172.21.15.103", "hostname": "smithi103", "labels": [], "status": ""}, {"addr": "172.21.15.163", "hostname": "smithi163", "labels": [], "status": ""}] 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 msgr2=0x7fd77403b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 0x7fd77403b5b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd770006fd0 tx=0x7fd770006e40 comp rx=0 tx=0).stop 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 msgr2=0x7fd78819d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fd778006010 tx=0x7fd778005030 comp rx=0 tx=0).stop 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 shutdown_connections 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd7740390f0 0x7fd77403b5b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 --2- 172.21.15.69:0/1477339358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd788100530 0x7fd78819d3f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 >> 172.21.15.69:0/1477339358 conn(0x7fd7880773a0 msgr2=0x7fd7880fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 shutdown_connections 2024-05-09T09:10:02.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:02.574+0000 7fd78d74f700 1 -- 172.21.15.69:0/1477339358 wait complete. 2024-05-09T09:10:03.088 INFO:tasks.cephadm:Setting crush tunables to default 2024-05-09T09:10:03.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd crush tunables default 2024-05-09T09:10:03.323 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:10:03.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:03.707 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:03 smithi069 ceph-mon[30123]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:10:03.708 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:03 smithi069 ceph-mon[30123]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 -- 172.21.15.69:0/4091043643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 msgr2=0x7fc1981081a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 --2- 172.21.15.69:0/4091043643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc1981081a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fc19400b0a0 tx=0x7fc19400b3b0 comp rx=0 tx=0).stop 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 -- 172.21.15.69:0/4091043643 shutdown_connections 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 --2- 172.21.15.69:0/4091043643 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc1981081a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 -- 172.21.15.69:0/4091043643 >> 172.21.15.69:0/4091043643 conn(0x7fc198076780 msgr2=0x7fc198076b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 -- 172.21.15.69:0/4091043643 shutdown_connections 2024-05-09T09:10:04.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.791+0000 7fc19fc71700 1 -- 172.21.15.69:0/4091043643 wait complete. 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19fc71700 1 Processor -- start 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19fc71700 1 -- start start 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19fc71700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19fc71700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc19810c350 con 0x7fc198107dc0 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19da0d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19da0d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38956/0 (socket says 172.21.15.69:38956) 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19da0d700 1 -- 172.21.15.69:0/3637236514 learned_addr learned my addr 172.21.15.69:0/3637236514 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:10:04.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19da0d700 1 -- 172.21.15.69:0/3637236514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc19819d950 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.792+0000 7fc19da0d700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc194015040 tx=0x7fc194003d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.793+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc194017040 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.793+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc194022070 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.793+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc19819dbe0 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.793+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fc19401d5d0 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.793+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc198197620 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fc194004000 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc198103470 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc18effd700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 0x7fc18403b610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(3..3 src has 1..3) ==== 1155+0+0 (secure 0 0 0) 0x7fc19400f050 con 0x7fc198107dc0 2024-05-09T09:10:04.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc19d20c700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 0x7fc18403b610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:04.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.794+0000 7fc19d20c700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 0x7fc18403b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc188006fd0 tx=0x7fc188006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:04.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.799+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc194007e20 con 0x7fc198107dc0 2024-05-09T09:10:04.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:04.996+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) -- 0x7fc198063980 con 0x7fc198107dc0 2024-05-09T09:10:05.047 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.048+0000 7fc18effd700 1 -- 172.21.15.69:0/3637236514 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) ==== 124+0+0 (secure 0 0 0) 0x7fc194004440 con 0x7fc198107dc0 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 msgr2=0x7fc18403b610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 0x7fc18403b610 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc188006fd0 tx=0x7fc188006e40 comp rx=0 tx=0).stop 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 msgr2=0x7fc19819d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fc194015040 tx=0x7fc194003d50 comp rx=0 tx=0).stop 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 shutdown_connections 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc184039150 0x7fc18403b610 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 --2- 172.21.15.69:0/3637236514 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc198107dc0 0x7fc19819d410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 >> 172.21.15.69:0/3637236514 conn(0x7fc198076780 msgr2=0x7fc1980fe0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 shutdown_connections 2024-05-09T09:10:05.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:05.050+0000 7fc19fc71700 1 -- 172.21.15.69:0/3637236514 wait complete. 2024-05-09T09:10:05.051 INFO:teuthology.orchestra.run.smithi069.stderr:adjusted tunables profile to default 2024-05-09T09:10:05.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:05 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3637236514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T09:10:05.661 INFO:tasks.cephadm:Adding mon.a on smithi069 2024-05-09T09:10:05.661 INFO:tasks.cephadm:Adding mon.b on smithi103 2024-05-09T09:10:05.661 INFO:tasks.cephadm:Adding mon.c on smithi163 2024-05-09T09:10:05.661 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch apply mon '3;smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c' 2024-05-09T09:10:05.897 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3637236514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T09:10:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.383+0000 7fefb5de0700 1 -- 172.21.15.163:0/531647833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 msgr2=0x7fefb0102370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.383+0000 7fefb5de0700 1 --2- 172.21.15.163:0/531647833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb0102370 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fefa0009910 tx=0x7fefa0009c20 comp rx=0 tx=0).stop 2024-05-09T09:10:07.383 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 -- 172.21.15.163:0/531647833 shutdown_connections 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 --2- 172.21.15.163:0/531647833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb0102370 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 -- 172.21.15.163:0/531647833 >> 172.21.15.163:0/531647833 conn(0x7fefb00fd860 msgr2=0x7fefb00ffc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 -- 172.21.15.163:0/531647833 shutdown_connections 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 -- 172.21.15.163:0/531647833 wait complete. 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 Processor -- start 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 -- start start 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.384+0000 7fefb5de0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefb5de0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefb010c360 con 0x7fefb0101f90 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefaf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefaf7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.163:39150/0 (socket says 172.21.15.163:39150) 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefaf7fe700 1 -- 172.21.15.163:0/1271746257 learned_addr learned my addr 172.21.15.163:0/1271746257 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefaf7fe700 1 -- 172.21.15.163:0/1271746257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefb019fbf0 con 0x7fefb0101f90 2024-05-09T09:10:07.384 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefaf7fe700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fefa0005850 tx=0x7fefa0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefa0026070 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefa000bbd0 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefb019fee0 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7fefa0003970 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.385+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefb0199830 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.386+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fefa0029070 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.386+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefb0103470 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.387+0000 7fefad7fa700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 0x7fef9c03b5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.387+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fefa001e070 con 0x7fefb0101f90 2024-05-09T09:10:07.386 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.387+0000 7fefa6ffd700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 0x7fef9c03b5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:07.387 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.387+0000 7fefa6ffd700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 0x7fef9c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fef98006fd0 tx=0x7fef98006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:07.390 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.392+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefa0013070 con 0x7fefb0101f90 2024-05-09T09:10:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-09T09:10:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T09:10:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:07.596 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.597+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c", "target": ["mon-mgr", ""]}) -- 0x7fefb019ae20 con 0x7fef9c0390f0 2024-05-09T09:10:07.603 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.604+0000 7fefad7fa700 1 -- 172.21.15.163:0/1271746257 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fefb019ae20 con 0x7fef9c0390f0 2024-05-09T09:10:07.603 INFO:teuthology.orchestra.run.smithi163.stdout:Scheduled mon update... 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 msgr2=0x7fef9c03b5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 0x7fef9c03b5b0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fef98006fd0 tx=0x7fef98006e40 comp rx=0 tx=0).stop 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 msgr2=0x7fefb019f6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fefa0005850 tx=0x7fefa0005030 comp rx=0 tx=0).stop 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 shutdown_connections 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef9c0390f0 0x7fef9c03b5b0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.606+0000 7fefb5de0700 1 --2- 172.21.15.163:0/1271746257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefb0101f90 0x7fefb019f6b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.607+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 >> 172.21.15.163:0/1271746257 conn(0x7fefb00fd860 msgr2=0x7fefb0076030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.607+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 shutdown_connections 2024-05-09T09:10:07.605 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:07.607+0000 7fefb5de0700 1 -- 172.21.15.163:0/1271746257 wait complete. 2024-05-09T09:10:08.224 DEBUG:teuthology.orchestra.run.smithi103:mon.b> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.b.service 2024-05-09T09:10:08.227 DEBUG:teuthology.orchestra.run.smithi163:mon.c> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.c.service 2024-05-09T09:10:08.229 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T09:10:08.229 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T09:10:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:09.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:09 smithi069 ceph-mon[30123]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:09.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:09 smithi069 ceph-mon[30123]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c;count:3 2024-05-09T09:10:09.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:09 smithi069 ceph-mon[30123]: Deploying daemon mon.c on smithi163 2024-05-09T09:10:10.291 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:10:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:10 smithi069 ceph-mon[30123]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:11.512 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:11 smithi163 systemd[1]: Starting Ceph mon.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:10:12.061 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- 172.21.15.163:0/1302342170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 msgr2=0x7feb641033d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/1302342170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb641033d0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7feb5c009910 tx=0x7feb5c009c20 comp rx=0 tx=0).stop 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- 172.21.15.163:0/1302342170 shutdown_connections 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/1302342170 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb641033d0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- 172.21.15.163:0/1302342170 >> 172.21.15.163:0/1302342170 conn(0x7feb640fc860 msgr2=0x7feb640fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- 172.21.15.163:0/1302342170 shutdown_connections 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- 172.21.15.163:0/1302342170 wait complete. 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 Processor -- start 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.062+0000 7feb6c6a7700 1 -- start start 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6c6a7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6c6a7700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7feb6410c4e0 con 0x7feb64102ff0 2024-05-09T09:10:12.062 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6a443700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6a443700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.163:39168/0 (socket says 172.21.15.163:39168) 2024-05-09T09:10:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6a443700 1 -- 172.21.15.163:0/3134935488 learned_addr learned my addr 172.21.15.163:0/3134935488 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:10:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6a443700 1 -- 172.21.15.163:0/3134935488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7feb6419fda0 con 0x7feb64102ff0 2024-05-09T09:10:12.063 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.063+0000 7feb6a443700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006010 tx=0x7feb5c005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.066+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb5c026070 con 0x7feb64102ff0 2024-05-09T09:10:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.066+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7feb5c00bc30 con 0x7feb64102ff0 2024-05-09T09:10:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.066+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7feb641a0030 con 0x7feb64102ff0 2024-05-09T09:10:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.066+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 205+0+0 (secure 0 0 0) 0x7feb5c003ab0 con 0x7feb64102ff0 2024-05-09T09:10:12.065 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.067+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7feb64199a20 con 0x7feb64102ff0 2024-05-09T09:10:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.067+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7feb6406ae10 con 0x7feb64102ff0 2024-05-09T09:10:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.067+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7feb5c029070 con 0x7feb64102ff0 2024-05-09T09:10:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.068+0000 7feb5b7fe700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 0x7feb5003b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.068+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7feb5c01e070 con 0x7feb64102ff0 2024-05-09T09:10:12.066 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.068+0000 7feb69c42700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 0x7feb5003b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:12.067 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.068+0000 7feb69c42700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 0x7feb5003b660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb54006fd0 tx=0x7feb54006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:12.071 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.073+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7feb5c013070 con 0x7feb64102ff0 2024-05-09T09:10:12.336 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 podman[34314]: 2024-05-09 09:10:12.035444046 +0000 UTC m=+0.197546257 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:10:12.336 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 podman[34314]: 2024-05-09 09:10:12.143391171 +0000 UTC m=+0.305493383 container create 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, RELEASE=squid-96805dc) 2024-05-09T09:10:12.336 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 podman[34314]: 2024-05-09 09:10:12.337725294 +0000 UTC m=+0.499827521 container init 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-09T09:10:12.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.362+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7feb64063980 con 0x7feb64102ff0 2024-05-09T09:10:12.361 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.362+0000 7feb5b7fe700 1 -- 172.21.15.163:0/3134935488 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) ==== 95+0+745 (secure 0 0 0) 0x7feb5c030020 con 0x7feb64102ff0 2024-05-09T09:10:12.361 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:10:12.361 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":1,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","modified":"2024-05-09T09:08:24.964355Z","created":"2024-05-09T09:08:24.964355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-05-09T09:10:12.363 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 msgr2=0x7feb5003b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 0x7feb5003b660 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feb54006fd0 tx=0x7feb54006e40 comp rx=0 tx=0).stop 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 msgr2=0x7feb6419f860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7feb5c006010 tx=0x7feb5c005050 comp rx=0 tx=0).stop 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 shutdown_connections 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feb500391a0 0x7feb5003b660 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 --2- 172.21.15.163:0/3134935488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7feb64102ff0 0x7feb6419f860 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 >> 172.21.15.163:0/3134935488 conn(0x7feb640fc860 msgr2=0x7feb641081b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 shutdown_connections 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:12.365+0000 7feb6c6a7700 1 -- 172.21.15.163:0/3134935488 wait complete. 2024-05-09T09:10:12.364 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 1 2024-05-09T09:10:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:12 smithi069 ceph-mon[30123]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 podman[34314]: 2024-05-09 09:10:12.341019172 +0000 UTC m=+0.503121389 container start 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: set uid:gid to 167:167 (ceph:ceph) 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev), process ceph-mon, pid 2 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pidfile_write: ignore empty --pid-file 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: load: jerasure load: lrc 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: RocksDB version: 7.9.2 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Git sha 0 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Compile date 2024-05-07 19:19:25 2024-05-09T09:10:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: DB SUMMARY 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: DB Session ID: 7S2FO0KOPSZUJW4PYAUJ 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: CURRENT file: CURRENT 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: IDENTITY file: IDENTITY 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.error_if_exists: 0 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.create_if_missing: 0 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.paranoid_checks: 1 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.flush_verify_memtable_count: 1 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.env: 0x559aa77fbc60 2024-05-09T09:10:12.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.fs: PosixFileSystem 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.info_log: 0x559aa807bac0 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_file_opening_threads: 16 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.statistics: (nil) 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.use_fsync: 0 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_log_file_size: 0 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.log_file_time_to_roll: 0 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.keep_log_file_num: 1000 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.recycle_log_file_num: 0 2024-05-09T09:10:12.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_fallocate: 1 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_mmap_reads: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_mmap_writes: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.use_direct_reads: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.create_missing_column_families: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.db_log_dir: 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.wal_dir: 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.table_cache_numshardbits: 6 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.WAL_ttl_seconds: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.WAL_size_limit_MB: 0 2024-05-09T09:10:12.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.is_fd_close_on_exec: 1 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.advise_random_on_open: 1 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.db_write_buffer_size: 0 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.write_buffer_manager: 0x559aa804e1e0 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-05-09T09:10:12.810 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.use_adaptive_mutex: 0 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.rate_limiter: (nil) 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.wal_recovery_mode: 2 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enable_thread_tracking: 0 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enable_pipelined_write: 0 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.unordered_write: 0 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-05-09T09:10:12.814 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.row_cache: None 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.wal_filter: None 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_ingest_behind: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.two_write_queues: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.manual_wal_flush: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.wal_compression: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.atomic_flush: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.persist_stats_to_disk: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.write_dbid_to_manifest: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.log_readahead_size: 0 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-05-09T09:10:12.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.best_efforts_recovery: 0 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.allow_data_in_errors: 0 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.db_host_id: __hostname__ 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enforce_single_del_contracts: true 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_background_jobs: 2 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_background_compactions: -1 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_subcompactions: 1 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.delayed_write_rate : 16777216 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_total_wal_size: 0 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.stats_dump_period_sec: 600 2024-05-09T09:10:12.816 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.stats_persist_period_sec: 600 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_open_files: -1 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bytes_per_sync: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.wal_bytes_per_sync: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.strict_bytes_per_sync: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_readahead_size: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_background_flushes: -1 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Compression algorithms supported: 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kZSTD supported: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kXpressCompression supported: 0 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kLZ4HCCompression supported: 1 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kZlibCompression supported: 1 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kSnappyCompression supported: 1 2024-05-09T09:10:12.817 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kLZ4Compression supported: 1 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: kBZip2Compression supported: 0 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Fast CRC32 supported: Supported on x86 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: DMutex implementation: pthread_mutex_t 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.merge_operator: 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_filter: None 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_filter_factory: None 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.sst_partitioner_factory: None 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.memtable_factory: SkipListFactory 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.table_factory: BlockBasedTable 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x559aa807bbc0) 2024-05-09T09:10:12.818 INFO:journalctl@ceph.mon.c.smithi163.stdout: cache_index_and_filter_blocks: 1 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: pin_top_level_index_and_filter: 1 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_type: 0 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: data_block_index_type: 0 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_shortening: 1 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: data_block_hash_table_util_ratio: 0.750000 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: checksum: 4 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: no_block_cache: 0 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache: 0x559aa807f090 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_name: BinnedLRUCache 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_options: 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: capacity : 536870912 2024-05-09T09:10:12.819 INFO:journalctl@ceph.mon.c.smithi163.stdout: num_shard_bits : 4 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: strict_capacity_limit : 0 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: high_pri_pool_ratio: 0.000 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_cache_compressed: (nil) 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: persistent_cache: (nil) 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_size: 4096 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_size_deviation: 10 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_restart_interval: 16 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: index_block_restart_interval: 1 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: metadata_block_size: 4096 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: partition_filters: 0 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: use_delta_encoding: 1 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: filter_policy: bloomfilter 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: whole_key_filtering: 1 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: verify_compression: 0 2024-05-09T09:10:12.820 INFO:journalctl@ceph.mon.c.smithi163.stdout: read_amp_bytes_per_bit: 0 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: format_version: 5 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: enable_index_compression: 1 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: block_align: 0 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: max_auto_readahead_size: 262144 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: prepopulate_block_cache: 0 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: initial_auto_readahead_size: 8192 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout: num_file_reads_for_auto_readahead: 2 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.write_buffer_size: 33554432 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_write_buffer_number: 2 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression: NoCompression 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression: Disabled 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.prefix_extractor: nullptr 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.num_levels: 7 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-05-09T09:10:12.821 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.window_bits: -14 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.level: 32767 2024-05-09T09:10:12.822 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.strategy: 0 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.enabled: false 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.target_file_size_base: 67108864 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.target_file_size_multiplier: 1 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-05-09T09:10:12.823 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.arena_block_size: 1048576 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.disable_auto_compactions: 0 2024-05-09T09:10:12.824 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.table_properties_collectors: 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.inplace_update_support: 0 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.inplace_update_num_locks: 10000 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-05-09T09:10:12.825 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.memtable_huge_page_size: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.bloom_locality: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.max_successive_merges: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.optimize_filters_for_hits: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.paranoid_file_checks: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.force_consistency_checks: 1 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.report_bg_io_stats: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.ttl: 2592000 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.periodic_compaction_seconds: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enable_blob_files: false 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.min_blob_size: 0 2024-05-09T09:10:12.826 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_file_size: 268435456 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_compression_type: NoCompression 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.enable_blob_garbage_collection: false 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.blob_file_starting_level: 0 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 060d2c7f-0287-4d50-82e8-78a83acd94d9 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245812371545, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245812371988, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1715245812, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "060d2c7f-0287-4d50-82e8-78a83acd94d9", "db_session_id": "7S2FO0KOPSZUJW4PYAUJ", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: EVENT_LOG_v1 {"time_micros": 1715245812372072, "job": 1, "event": "recovery_finished"} 2024-05-09T09:10:12.827 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x559aa8170000 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: DB pointer 0x559aa815a000 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** DB Stats ** 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.828 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** Compaction Stats [default] ** 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** Compaction Stats [default] ** 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.7 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-05-09T09:10:12.829 INFO:journalctl@ceph.mon.c.smithi163.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: AddFile(Keys): cumulative 0, interval 0 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: Cumulative compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: Interval compaction: 0.00 GB write, 0.60 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: Block cache BinnedLRUCache@0x559aa807f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout: ** File Read Latency Histogram By Level [default] ** 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-05-09T09:10:12.830 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: using public_addr v2:172.21.15.163:0/0 -> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: starting mon.c rank -1 at public addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] at bind addrs [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(???) e0 preinit fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).mds e1 new map 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).mds e1 print_map 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: e1 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: legacy client fscid: -1 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout: No filesystems configured 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-05-09T09:10:12.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Added host smithi103 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:10:12.832 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi163", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi103", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Deploying cephadm binary to smithi163 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.833 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi103:/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Added host smithi163 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3637236514' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.834 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3637236514' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi163", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi163:/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.client.admin.keyring 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.835 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Saving service mon spec with placement smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c;count:3 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: Deploying daemon mon.c on smithi163 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:12.836 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 ceph-mon[34354]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-05-09T09:10:12.837 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 bash[34314]: 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 2024-05-09T09:10:12.837 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:12 smithi163 systemd[1]: Started Ceph mon.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:10:14.109 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T09:10:14.110 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T09:10:14.354 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: mon.a calling monitor election 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: mon.c calling monitor election 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: monmap epoch 2 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: last_changed 2024-05-09T09:10:12.396999+0000 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: min_mon_release 19 (squid) 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: election_strategy: 1 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.c 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: fsmap 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: mgrmap e13: x(active, since 48s) 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: overall HEALTH_OK 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:17.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: mon.a calling monitor election 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: mon.c calling monitor election 2024-05-09T09:10:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: monmap epoch 2 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: last_changed 2024-05-09T09:10:12.396999+0000 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: min_mon_release 19 (squid) 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: election_strategy: 1 2024-05-09T09:10:17.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.c 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: fsmap 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: mgrmap e13: x(active, since 48s) 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: overall HEALTH_OK 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:17.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:18.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:18 smithi069 ceph-mon[30123]: Deploying daemon mon.b on smithi103 2024-05-09T09:10:18.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:18.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.803+0000 7fab88944700 1 -- 172.21.15.163:0/2423073361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 msgr2=0x7fab6c005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:18.802 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 --2- 172.21.15.163:0/2423073361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab6c005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fab8406a610 tx=0x7fab7400feb0 comp rx=0 tx=0).stop 2024-05-09T09:10:18.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:18 smithi163 ceph-mon[34354]: Deploying daemon mon.b on smithi103 2024-05-09T09:10:18.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:18.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 -- 172.21.15.163:0/2423073361 shutdown_connections 2024-05-09T09:10:18.803 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 --2- 172.21.15.163:0/2423073361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab6c005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 -- 172.21.15.163:0/2423073361 >> 172.21.15.163:0/2423073361 conn(0x7fab840fc860 msgr2=0x7fab840fec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 -- 172.21.15.163:0/2423073361 shutdown_connections 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.804+0000 7fab88944700 1 -- 172.21.15.163:0/2423073361 wait complete. 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 Processor -- start 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 -- start start 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab84109ed0 con 0x7fab6c0035c0 2024-05-09T09:10:18.804 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab88944700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab84109d50 con 0x7fab84101190 2024-05-09T09:10:18.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.805+0000 7fab81d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:18.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:18.806 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.163:45612/0 (socket says 172.21.15.163:45612) 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab81d9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:59556/0 (socket says 172.21.15.163:59556) 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 -- 172.21.15.163:0/181577996 learned_addr learned my addr 172.21.15.163:0/181577996 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab81d9b700 1 -- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 msgr2=0x7fab84101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab81d9b700 1 -- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 msgr2=0x7fab84101610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab81d9b700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab81d9b700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 -- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 msgr2=0x7fab84101610 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 -- 172.21.15.163:0/181577996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab84102b70 con 0x7fab6c0035c0 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab8259c700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fab74001eb0 tx=0x7fab740057d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fab74004ac0 con 0x7fab6c0035c0 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab74013e40 con 0x7fab6c0035c0 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab841a3c80 con 0x7fab6c0035c0 2024-05-09T09:10:18.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.806+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 309+0+0 (secure 0 0 0) 0x7fab74011aa0 con 0x7fab6c0035c0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.807+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab841a41c0 con 0x7fab6c0035c0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.807+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fab7402c070 con 0x7fab6c0035c0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.808+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fab8406ae10 con 0x7fab6c0035c0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.808+0000 7fab737fe700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 0x7fab6403b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.808+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fab7402a070 con 0x7fab6c0035c0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.808+0000 7fab81d9b700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 0x7fab6403b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:18.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.809+0000 7fab81d9b700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 0x7fab6403b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab78009850 tx=0x7fab78008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:18.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:18.813+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fab74011e20 con 0x7fab6c0035c0 2024-05-09T09:10:19.090 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.092+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7fab8402db80 con 0x7fab6c0035c0 2024-05-09T09:10:19.091 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.093+0000 7fab737fe700 1 -- 172.21.15.163:0/181577996 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) ==== 95+0+1015 (secure 0 0 0) 0x7fab740222c0 con 0x7fab6c0035c0 2024-05-09T09:10:19.091 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:10:19.091 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":2,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","modified":"2024-05-09T09:10:12.396999Z","created":"2024-05-09T09:08:24.964355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 msgr2=0x7fab6403b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 0x7fab6403b3a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fab78009850 tx=0x7fab78008040 comp rx=0 tx=0).stop 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 msgr2=0x7fab84100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fab74001eb0 tx=0x7fab740057d0 comp rx=0 tx=0).stop 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 shutdown_connections 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fab64038ee0 0x7fab6403b3a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab6c0035c0 0x7fab84100c50 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 --2- 172.21.15.163:0/181577996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab84101190 0x7fab84101610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 >> 172.21.15.163:0/181577996 conn(0x7fab840fc860 msgr2=0x7fab840fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 shutdown_connections 2024-05-09T09:10:19.094 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:19.095+0000 7fab88944700 1 -- 172.21.15.163:0/181577996 wait complete. 2024-05-09T09:10:19.095 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 2 2024-05-09T09:10:19.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:19 smithi163 ceph-mon[34354]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:19.662 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:19 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/181577996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T09:10:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:19 smithi069 ceph-mon[30123]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:19 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/181577996' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T09:10:20.688 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-05-09T09:10:20.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph mon dump -f json 2024-05-09T09:10:20.946 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 -- 172.21.15.163:0/1870362316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 msgr2=0x7f8c14005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1870362316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c14005a50 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8c2c101d90 tx=0x7f8c1c00fe60 comp rx=0 tx=0).stop 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 -- 172.21.15.163:0/1870362316 shutdown_connections 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1870362316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c14005a50 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 -- 172.21.15.163:0/1870362316 >> 172.21.15.163:0/1870362316 conn(0x7f8c2c0fd910 msgr2=0x7f8c2c0ffd30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 -- 172.21.15.163:0/1870362316 shutdown_connections 2024-05-09T09:10:26.701 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.701+0000 7f8c31c60700 1 -- 172.21.15.163:0/1870362316 wait complete. 2024-05-09T09:10:26.703 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.702+0000 7f8c31c60700 1 Processor -- start 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.702+0000 7f8c31c60700 1 -- start start 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c2c074b60 0x7f8c2c074fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c109ea0 con 0x7f8c140035c0 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c109d20 con 0x7f8c2c074b60 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c31c60700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8c2c10a020 con 0x7f8c2c1a3cf0 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.163:59134/0 (socket says 172.21.15.163:59134) 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 -- 172.21.15.163:0/1818006685 learned_addr learned my addr 172.21.15.163:0/1818006685 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2bfff700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:26.704 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2affd700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c2c074b60 0x7f8c2c074fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2bfff700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 msgr2=0x7f8c2c1a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 14 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2bfff700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 msgr2=0x7f8c2c1a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2bfff700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 msgr2=0x7f8c2c1a60e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2bfff700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c2c074b60 msgr2=0x7f8c2c074fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c2c074b60 0x7f8c2c074fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:26.705 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.703+0000 7f8c2b7fe700 1 -- 172.21.15.163:0/1818006685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8c2c1a67b0 con 0x7f8c140035c0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c2b7fe700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c003a50 tx=0x7f8c1c005790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c02b020 con 0x7f8c140035c0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8c1c004180 con 0x7f8c140035c0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8c2c1a6a40 con 0x7f8c140035c0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8c1c022040 con 0x7f8c140035c0 2024-05-09T09:10:26.706 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.704+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8c2c1a6fd0 con 0x7f8c140035c0 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.705+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f8c1c02e020 con 0x7f8c140035c0 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.705+0000 7f8c28ff9700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 0x7f8c1803ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.705+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f8c1c04ec90 con 0x7f8c140035c0 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.705+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8c2c063980 con 0x7f8c140035c0 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.706+0000 7f8c2affd700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 0x7f8c1803ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:26.707 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.706+0000 7f8c2affd700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 0x7f8c1803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009850 tx=0x7f8c20008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:26.710 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.711+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8c1c035020 con 0x7f8c140035c0 2024-05-09T09:10:26.997 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.997+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) -- 0x7f8c2c075b00 con 0x7f8c140035c0 2024-05-09T09:10:26.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:10:26.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: mon.a calling monitor election 2024-05-09T09:10:26.997 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: mon.c calling monitor election 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: mon.b calling monitor election 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: monmap epoch 3 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:26.998 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: last_changed 2024-05-09T09:10:21.556075+0000 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: min_mon_release 19 (squid) 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: election_strategy: 1 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.c 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: fsmap 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: mgrmap e13: x(active, since 57s) 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: overall HEALTH_OK 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:26.999 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.000 INFO:teuthology.orchestra.run.smithi163.stdout: 2024-05-09T09:10:27.000 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:26.998+0000 7f8c28ff9700 1 -- 172.21.15.163:0/1818006685 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) ==== 95+0+1285 (secure 0 0 0) 0x7f8c1c004330 con 0x7f8c140035c0 2024-05-09T09:10:27.000 INFO:teuthology.orchestra.run.smithi163.stdout:{"epoch":3,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","modified":"2024-05-09T09:10:21.556075Z","created":"2024-05-09T09:08:24.964355Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:3300","nonce":0},{"type":"v1","addr":"172.21.15.69:6789","nonce":0}]},"addr":"172.21.15.69:6789/0","public_addr":"172.21.15.69:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:3300","nonce":0},{"type":"v1","addr":"172.21.15.163:6789","nonce":0}]},"addr":"172.21.15.163:6789/0","public_addr":"172.21.15.163:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:3300","nonce":0},{"type":"v1","addr":"172.21.15.103:6789","nonce":0}]},"addr":"172.21.15.103:6789/0","public_addr":"172.21.15.103:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 msgr2=0x7f8c1803ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 0x7f8c1803ba40 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f8c20009850 tx=0x7f8c20008040 comp rx=0 tx=0).stop 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 msgr2=0x7f8c2c074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8c1c003a50 tx=0x7f8c1c005790 comp rx=0 tx=0).stop 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 shutdown_connections 2024-05-09T09:10:27.001 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8c2c1a3cf0 0x7f8c2c1a60e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8c140035c0 0x7f8c2c074620 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8c18039580 0x7f8c1803ba40 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 --2- 172.21.15.163:0/1818006685 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8c2c074b60 0x7f8c2c074fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 >> 172.21.15.163:0/1818006685 conn(0x7f8c2c0fd910 msgr2=0x7f8c2c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 shutdown_connections 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:27.001+0000 7f8c31c60700 1 -- 172.21.15.163:0/1818006685 wait complete. 2024-05-09T09:10:27.002 INFO:teuthology.orchestra.run.smithi163.stderr:dumped monmap epoch 3 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: mon.a calling monitor election 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: mon.c calling monitor election 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:10:27.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: mon.b calling monitor election 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: monmap epoch 3 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: last_changed 2024-05-09T09:10:21.556075+0000 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: created 2024-05-09T09:08:24.964355+0000 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: min_mon_release 19 (squid) 2024-05-09T09:10:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: election_strategy: 1 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: 0: [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] mon.a 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: 1: [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] mon.c 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: 2: [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] mon.b 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: fsmap 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: osdmap e4: 0 total, 0 up, 0 in 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: mgrmap e13: x(active, since 57s) 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: overall HEALTH_OK 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:27.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:27.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:27 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/1818006685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T09:10:27.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:27 smithi163 ceph-mon[34354]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:27.725 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:27.753 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-05-09T09:10:27.754 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph config generate-minimal-conf 2024-05-09T09:10:27.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:27.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:27 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/1818006685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T09:10:27.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:27 smithi103 ceph-mon[34421]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:27.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:28.001 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:10:28.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:28.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:27 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/1818006685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-05-09T09:10:28.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:27 smithi069 ceph-mon[30123]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:28.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:10:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:29.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:29.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.550+0000 7f85ea9ab700 1 -- 172.21.15.69:0/3532841885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4100e70 msgr2=0x7f85e4101250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:29.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.550+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/3532841885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4100e70 0x7f85e4101250 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009910 tx=0x7f85d4009c20 comp rx=0 tx=0).stop 2024-05-09T09:10:29.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.550+0000 7f85ea9ab700 1 -- 172.21.15.69:0/3532841885 shutdown_connections 2024-05-09T09:10:29.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.550+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/3532841885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4100e70 0x7f85e4101250 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.550+0000 7f85ea9ab700 1 -- 172.21.15.69:0/3532841885 >> 172.21.15.69:0/3532841885 conn(0x7f85e40fc720 msgr2=0x7f85e40feb40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.551+0000 7f85ea9ab700 1 -- 172.21.15.69:0/3532841885 shutdown_connections 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.551+0000 7f85ea9ab700 1 -- 172.21.15.69:0/3532841885 wait complete. 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.551+0000 7f85ea9ab700 1 Processor -- start 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.551+0000 7f85ea9ab700 1 -- start start 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.551+0000 7f85ea9ab700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85e4100e70 0x7f85e419d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85ea9ab700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 0x7f85e41978b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85ea9ab700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85ea9ab700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410b200 con 0x7f85e4197df0 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85ea9ab700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410b080 con 0x7f85e419dc30 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85ea9ab700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f85e410b380 con 0x7f85e4100e70 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51160/0 (socket says 172.21.15.69:51160) 2024-05-09T09:10:29.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 -- 172.21.15.69:0/1101052584 learned_addr learned my addr 172.21.15.69:0/1101052584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e3fff700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85e4100e70 0x7f85e419d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e37fe700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 0x7f85e41978b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 -- 172.21.15.69:0/1101052584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85e4100e70 msgr2=0x7f85e419d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85e4100e70 0x7f85e419d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 -- 172.21.15.69:0/1101052584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 msgr2=0x7f85e41978b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 0x7f85e41978b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 -- 172.21.15.69:0/1101052584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f85e41a45c0 con 0x7f85e4197df0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e37fe700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 0x7f85e41978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e8f48700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00ba50 tx=0x7f85dc00be10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc019070 con 0x7f85e4197df0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.552+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f85dc014440 con 0x7f85e4197df0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.553+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f85e41a47f0 con 0x7f85e4197df0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.553+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f85dc007590 con 0x7f85e4197df0 2024-05-09T09:10:29.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.553+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f85e41a4ce0 con 0x7f85e4197df0 2024-05-09T09:10:29.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f85dc01c070 con 0x7f85e4197df0 2024-05-09T09:10:29.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f85e4063980 con 0x7f85e4197df0 2024-05-09T09:10:29.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85e17fa700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 0x7f85cc03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:29.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f85dc04b940 con 0x7f85e4197df0 2024-05-09T09:10:29.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85e3fff700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 0x7f85cc03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:29.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.554+0000 7f85e3fff700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 0x7f85cc03ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009750 tx=0x7f85d400b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:29.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.559+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f85dc014910 con 0x7f85e4197df0 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.764+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) -- 0x7f85e41a5000 con 0x7f85e4197df0 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.764+0000 7f85e17fa700 1 -- 172.21.15.69:0/1101052584 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) ==== 76+0+275 (secure 0 0 0) 0x7f85dc00e090 con 0x7f85e4197df0 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stdout:# minimal ceph.conf for 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stdout:[global] 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stdout: fsid = 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:10:29.763 INFO:teuthology.orchestra.run.smithi069.stdout: mon_host = [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] 2024-05-09T09:10:29.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.766+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 msgr2=0x7f85cc03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:29.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 0x7f85cc03ba40 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f85d4009750 tx=0x7f85d400b540 comp rx=0 tx=0).stop 2024-05-09T09:10:29.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 msgr2=0x7f85e41982a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f85dc00ba50 tx=0x7f85dc00be10 comp rx=0 tx=0).stop 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 shutdown_connections 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f85e4197df0 0x7f85e41982a0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f85e4100e70 0x7f85e419d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f85cc039580 0x7f85cc03ba40 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 --2- 172.21.15.69:0/1101052584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f85e419dc30 0x7f85e41978b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 >> 172.21.15.69:0/1101052584 conn(0x7f85e40fc720 msgr2=0x7f85e4069fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 shutdown_connections 2024-05-09T09:10:29.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:29.767+0000 7f85ea9ab700 1 -- 172.21.15.69:0/1101052584 wait complete. 2024-05-09T09:10:30.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi069:/etc/ceph/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.301 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:30 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1101052584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi069:/etc/ceph/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:30 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1101052584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi069:/etc/ceph/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi103:/etc/ceph/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi163:/etc/ceph/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: Updating smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/config/ceph.conf 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:30 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1101052584' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:30.970 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-05-09T09:10:30.970 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:10:30.970 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T09:10:30.997 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:10:30.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:31.062 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:10:31.062 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T09:10:31.094 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:10:31.095 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:31.160 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:10:31.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.conf 2024-05-09T09:10:31.192 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:10:31.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-05-09T09:10:31.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:31 smithi069 ceph-mon[30123]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T09:10:31.254 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:31 smithi069 ceph-mon[30123]: Reconfiguring daemon mon.a on smithi069 2024-05-09T09:10:31.260 INFO:tasks.cephadm:Adding mgr.x on smithi069 2024-05-09T09:10:31.260 INFO:tasks.cephadm:Adding mgr.y on smithi103 2024-05-09T09:10:31.260 INFO:tasks.cephadm:Adding mgr.z on smithi163 2024-05-09T09:10:31.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch apply mgr '3;smithi069=x;smithi103=y;smithi163=z' 2024-05-09T09:10:31.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:31 smithi103 ceph-mon[34421]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T09:10:31.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:31 smithi103 ceph-mon[34421]: Reconfiguring daemon mon.a on smithi069 2024-05-09T09:10:31.538 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:10:31.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:31 smithi163 ceph-mon[34354]: Reconfiguring mon.a (unknown last config time)... 2024-05-09T09:10:31.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:31 smithi163 ceph-mon[34354]: Reconfiguring daemon mon.a on smithi069 2024-05-09T09:10:32.277 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:32.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:32.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:32.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:32.990 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 -- 172.21.15.163:0/1297291601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff0100dd0 msgr2=0x7fcff01011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 --2- 172.21.15.163:0/1297291601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff0100dd0 0x7fcff01011f0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fcfe800b0a0 tx=0x7fcfe800b3b0 comp rx=0 tx=0).stop 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 -- 172.21.15.163:0/1297291601 shutdown_connections 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 --2- 172.21.15.163:0/1297291601 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff0100dd0 0x7fcff01011f0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 -- 172.21.15.163:0/1297291601 >> 172.21.15.163:0/1297291601 conn(0x7fcff00fc390 msgr2=0x7fcff00fe7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 -- 172.21.15.163:0/1297291601 shutdown_connections 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.990+0000 7fcff5847700 1 -- 172.21.15.163:0/1297291601 wait complete. 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 Processor -- start 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 -- start start 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff0100dd0 0x7fcff0197730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:32.991 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:32.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0107f10 con 0x7fcff01980f0 2024-05-09T09:10:32.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0107d90 con 0x7fcff0100dd0 2024-05-09T09:10:32.992 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.991+0000 7fcff5847700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcff0108090 con 0x7fcff0197c70 2024-05-09T09:10:32.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfef7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:32.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:32.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfef7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.163:59152/0 (socket says 172.21.15.163:59152) 2024-05-09T09:10:32.993 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfef7fe700 1 -- 172.21.15.163:0/1810021509 learned_addr learned my addr 172.21.15.163:0/1810021509 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:33748/0 (socket says 172.21.15.163:33748) 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfeeffd700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff0100dd0 0x7fcff0197730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 -- 172.21.15.163:0/1810021509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff0100dd0 msgr2=0x7fcff0197730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff0100dd0 0x7fcff0197730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 -- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 msgr2=0x7fcff019f270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 -- 172.21.15.163:0/1810021509 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcff019fb40 con 0x7fcff0197c70 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfef7fe700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcfee7fc700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b550 tx=0x7fcfd800b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.992+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8004230 con 0x7fcff0197c70 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.993+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfd80043d0 con 0x7fcff0197c70 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.993+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfd8011b20 con 0x7fcff0197c70 2024-05-09T09:10:32.994 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.993+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcff019fe30 con 0x7fcff0197c70 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.993+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcff01a03b0 con 0x7fcff0197c70 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.994+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcff0063980 con 0x7fcff0197c70 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.994+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7fcfd8011d00 con 0x7fcff0197c70 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.994+0000 7fcff4845700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 0x7fcfdc03ba40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.994+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7fcfd804bbc0 con 0x7fcff0197c70 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.994+0000 7fcfeeffd700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 0x7fcfdc03ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:32.995 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.995+0000 7fcfeeffd700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 0x7fcfdc03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80076a0 tx=0x7fcfe8005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:32.998 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:32.999+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcfd8018350 con 0x7fcff0197c70 2024-05-09T09:10:33.212 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.213+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi069=x;smithi103=y;smithi163=z", "target": ["mon-mgr", ""]}) -- 0x7fcff00627f0 con 0x7fcfdc039580 2024-05-09T09:10:33.218 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.219+0000 7fcff4845700 1 -- 172.21.15.163:0/1810021509 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+24 (secure 0 0 0) 0x7fcff00627f0 con 0x7fcfdc039580 2024-05-09T09:10:33.219 INFO:teuthology.orchestra.run.smithi163.stdout:Scheduled mgr update... 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 msgr2=0x7fcfdc03ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 0x7fcfdc03ba40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fcfe80076a0 tx=0x7fcfe8005040 comp rx=0 tx=0).stop 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 msgr2=0x7fcff019cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fcfd800b550 tx=0x7fcfd800b860 comp rx=0 tx=0).stop 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 shutdown_connections 2024-05-09T09:10:33.221 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcff01980f0 0x7fcff019f270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfdc039580 0x7fcfdc03ba40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.221+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcff0100dd0 0x7fcff0197730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.222+0000 7fcff5847700 1 --2- 172.21.15.163:0/1810021509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcff0197c70 0x7fcff019cce0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.222+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 >> 172.21.15.163:0/1810021509 conn(0x7fcff00fc390 msgr2=0x7fcff00fcfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.222+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 shutdown_connections 2024-05-09T09:10:33.222 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:10:33.222+0000 7fcff5847700 1 -- 172.21.15.163:0/1810021509 wait complete. 2024-05-09T09:10:33.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: Reconfiguring mon.b (monmap changed)... 2024-05-09T09:10:33.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: Reconfiguring daemon mon.b on smithi103 2024-05-09T09:10:33.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: Reconfiguring mon.b (monmap changed)... 2024-05-09T09:10:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: Reconfiguring daemon mon.b on smithi103 2024-05-09T09:10:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:33.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: Reconfiguring mon.b (monmap changed)... 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: Reconfiguring daemon mon.b on smithi103 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-05-09T09:10:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:33.869 DEBUG:teuthology.orchestra.run.smithi103:mgr.y> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y.service 2024-05-09T09:10:33.872 DEBUG:teuthology.orchestra.run.smithi163:mgr.z> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.z.service 2024-05-09T09:10:33.875 INFO:tasks.cephadm:Deploying OSDs... 2024-05-09T09:10:33.875 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:10:33.875 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T09:10:33.895 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T09:10:33.895 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_1 2024-05-09T09:10:33.955 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T09:10:33.955 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:33.955 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-09T09:10:33.955 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:33.955 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:33.956 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-05-09 09:09:41.473388017 +0000 2024-05-09T09:10:33.956 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-05-09 09:04:28.463190984 +0000 2024-05-09T09:10:33.956 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-05-09 09:04:28.463190984 +0000 2024-05-09T09:10:33.956 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-05-09 09:04:28.463190984 +0000 2024-05-09T09:10:33.956 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T09:10:34.023 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-05-09T09:10:34.023 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-05-09T09:10:34.023 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000236584 s, 2.2 MB/s 2024-05-09T09:10:34.025 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T09:10:34.086 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_2 2024-05-09T09:10:34.111 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:34 smithi163 ceph-mon[34354]: Reconfiguring mon.c (monmap changed)... 2024-05-09T09:10:34.111 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:34 smithi163 ceph-mon[34354]: Reconfiguring daemon mon.c on smithi163 2024-05-09T09:10:34.111 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:34 smithi163 ceph-mon[34354]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:34.111 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-05-09 09:09:41.499387425 +0000 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-05-09 09:04:28.802183269 +0000 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-05-09 09:04:28.802183269 +0000 2024-05-09T09:10:34.144 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-05-09 09:04:28.802183269 +0000 2024-05-09T09:10:34.144 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T09:10:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-05-09T09:10:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-05-09T09:10:34.210 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000187049 s, 2.7 MB/s 2024-05-09T09:10:34.211 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T09:10:34.269 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_3 2024-05-09T09:10:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T09:10:34.326 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-05-09 09:09:41.523386880 +0000 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-05-09 09:04:29.137175644 +0000 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-05-09 09:04:29.137175644 +0000 2024-05-09T09:10:34.327 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-05-09 09:04:29.137175644 +0000 2024-05-09T09:10:34.327 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T09:10:34.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:34 smithi103 ceph-mon[34421]: Reconfiguring mon.c (monmap changed)... 2024-05-09T09:10:34.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:34 smithi103 ceph-mon[34421]: Reconfiguring daemon mon.c on smithi163 2024-05-09T09:10:34.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:34 smithi103 ceph-mon[34421]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:34.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:34 smithi069 ceph-mon[30123]: Reconfiguring mon.c (monmap changed)... 2024-05-09T09:10:34.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:34 smithi069 ceph-mon[30123]: Reconfiguring daemon mon.c on smithi163 2024-05-09T09:10:34.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:34 smithi069 ceph-mon[30123]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:34.389 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:34.391 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-05-09T09:10:34.392 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-05-09T09:10:34.392 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000188534 s, 2.7 MB/s 2024-05-09T09:10:34.393 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T09:10:34.450 DEBUG:teuthology.orchestra.run.smithi069:> stat /dev/vg_nvme/lv_4 2024-05-09T09:10:34.510 INFO:teuthology.orchestra.run.smithi069.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T09:10:34.510 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 5h/5d Inode: 722 Links: 1 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-05-09 09:09:41.548386311 +0000 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-05-09 09:04:29.476167928 +0000 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-05-09 09:04:29.476167928 +0000 2024-05-09T09:10:34.511 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-05-09 09:04:29.476167928 +0000 2024-05-09T09:10:34.511 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T09:10:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records in 2024-05-09T09:10:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:1+0 records out 2024-05-09T09:10:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:512 bytes copied, 0.000207525 s, 2.5 MB/s 2024-05-09T09:10:34.576 DEBUG:teuthology.orchestra.run.smithi069:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T09:10:34.633 DEBUG:teuthology.orchestra.run.smithi103:> set -ex 2024-05-09T09:10:34.633 DEBUG:teuthology.orchestra.run.smithi103:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T09:10:34.652 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T09:10:34.652 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_1 2024-05-09T09:10:34.709 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T09:10:34.709 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:34.709 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-09 09:10:20.985458271 +0000 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-09 09:05:09.775566092 +0000 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-09 09:05:09.775566092 +0000 2024-05-09T09:10:34.710 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-09 09:05:09.775566092 +0000 2024-05-09T09:10:34.710 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T09:10:34.775 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-09T09:10:34.775 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-09T09:10:34.775 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000229143 s, 2.2 MB/s 2024-05-09T09:10:34.776 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T09:10:34.834 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_2 2024-05-09T09:10:34.890 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-09 09:10:20.985458271 +0000 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-09 09:05:10.130559146 +0000 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-09 09:05:10.130559146 +0000 2024-05-09T09:10:34.891 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-09 09:05:10.130559146 +0000 2024-05-09T09:10:34.891 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T09:10:34.954 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-09T09:10:34.954 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-09T09:10:34.954 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000178269 s, 2.9 MB/s 2024-05-09T09:10:34.956 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T09:10:35.012 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_3 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.069 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-09 09:10:20.985458271 +0000 2024-05-09T09:10:35.070 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-09 09:05:10.487552160 +0000 2024-05-09T09:10:35.070 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-09 09:05:10.487552160 +0000 2024-05-09T09:10:35.070 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-09 09:05:10.487552160 +0000 2024-05-09T09:10:35.070 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T09:10:35.131 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-09T09:10:35.132 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-09T09:10:35.132 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000171446 s, 3.0 MB/s 2024-05-09T09:10:35.133 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T09:10:35.188 DEBUG:teuthology.orchestra.run.smithi103:> stat /dev/vg_nvme/lv_4 2024-05-09T09:10:35.243 INFO:teuthology.orchestra.run.smithi103.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T09:10:35.243 INFO:teuthology.orchestra.run.smithi103.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.243 INFO:teuthology.orchestra.run.smithi103.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T09:10:35.243 INFO:teuthology.orchestra.run.smithi103.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.243 INFO:teuthology.orchestra.run.smithi103.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.244 INFO:teuthology.orchestra.run.smithi103.stdout:Access: 2024-05-09 09:10:20.986458252 +0000 2024-05-09T09:10:35.244 INFO:teuthology.orchestra.run.smithi103.stdout:Modify: 2024-05-09 09:05:10.825545546 +0000 2024-05-09T09:10:35.244 INFO:teuthology.orchestra.run.smithi103.stdout:Change: 2024-05-09 09:05:10.825545546 +0000 2024-05-09T09:10:35.244 INFO:teuthology.orchestra.run.smithi103.stdout: Birth: 2024-05-09 09:05:10.825545546 +0000 2024-05-09T09:10:35.244 DEBUG:teuthology.orchestra.run.smithi103:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T09:10:35.305 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records in 2024-05-09T09:10:35.306 INFO:teuthology.orchestra.run.smithi103.stderr:1+0 records out 2024-05-09T09:10:35.306 INFO:teuthology.orchestra.run.smithi103.stderr:512 bytes copied, 0.000173142 s, 3.0 MB/s 2024-05-09T09:10:35.307 DEBUG:teuthology.orchestra.run.smithi103:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T09:10:35.362 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2024-05-09T09:10:35.362 DEBUG:teuthology.orchestra.run.smithi163:> dd if=/scratch_devs of=/dev/stdout 2024-05-09T09:10:35.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:35 smithi103 ceph-mon[34421]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi069=x;smithi103=y;smithi163=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:35.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:35 smithi103 ceph-mon[34421]: Saving service mgr spec with placement smithi069=x;smithi103=y;smithi163=z;count:3 2024-05-09T09:10:35.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:35.382 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-05-09T09:10:35.382 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_1 2024-05-09T09:10:35.440 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-05-09T09:10:35.440 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.440 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-09 09:10:11.280178291 +0000 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-09 09:04:50.174530875 +0000 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-09 09:04:50.174530875 +0000 2024-05-09T09:10:35.441 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-09 09:04:50.174530875 +0000 2024-05-09T09:10:35.441 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-05-09T09:10:35.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:35 smithi163 ceph-mon[34354]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi069=x;smithi103=y;smithi163=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:35.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:35 smithi163 ceph-mon[34354]: Saving service mgr spec with placement smithi069=x;smithi103=y;smithi163=z;count:3 2024-05-09T09:10:35.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.504 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.505 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:35.506 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-09T09:10:35.507 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-09T09:10:35.507 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000287305 s, 1.8 MB/s 2024-05-09T09:10:35.508 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-05-09T09:10:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:35 smithi069 ceph-mon[30123]: from='client.24103 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi069=x;smithi103=y;smithi163=z", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:35 smithi069 ceph-mon[30123]: Saving service mgr spec with placement smithi069=x;smithi103=y;smithi163=z;count:3 2024-05-09T09:10:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:35.567 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_2 2024-05-09T09:10:35.623 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-09 09:10:11.281178269 +0000 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-09 09:04:50.525522824 +0000 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-09 09:04:50.525522824 +0000 2024-05-09T09:10:35.624 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-09 09:04:50.525522824 +0000 2024-05-09T09:10:35.624 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-05-09T09:10:35.687 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-09T09:10:35.688 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-09T09:10:35.688 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000234216 s, 2.2 MB/s 2024-05-09T09:10:35.689 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-05-09T09:10:35.745 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_3 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 702 Links: 1 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-09 09:10:11.281178269 +0000 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-09 09:04:50.852515323 +0000 2024-05-09T09:10:35.800 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-09 09:04:50.852515323 +0000 2024-05-09T09:10:35.801 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-09 09:04:50.852515323 +0000 2024-05-09T09:10:35.801 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-05-09T09:10:35.864 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-09T09:10:35.864 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-09T09:10:35.864 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.00116554 s, 439 kB/s 2024-05-09T09:10:35.865 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-05-09T09:10:35.922 DEBUG:teuthology.orchestra.run.smithi163:> stat /dev/vg_nvme/lv_4 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Context: system_u:object_r:device_t:s0 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Access: 2024-05-09 09:10:11.282178247 +0000 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Modify: 2024-05-09 09:04:51.197507410 +0000 2024-05-09T09:10:35.977 INFO:teuthology.orchestra.run.smithi163.stdout:Change: 2024-05-09 09:04:51.197507410 +0000 2024-05-09T09:10:35.978 INFO:teuthology.orchestra.run.smithi163.stdout: Birth: 2024-05-09 09:04:51.197507410 +0000 2024-05-09T09:10:35.978 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-05-09T09:10:36.040 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records in 2024-05-09T09:10:36.041 INFO:teuthology.orchestra.run.smithi163.stderr:1+0 records out 2024-05-09T09:10:36.041 INFO:teuthology.orchestra.run.smithi163.stderr:512 bytes copied, 0.000173732 s, 2.9 MB/s 2024-05-09T09:10:36.042 DEBUG:teuthology.orchestra.run.smithi163:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-05-09T09:10:36.099 INFO:tasks.cephadm:Deploying osd.0 on smithi069 with /dev/vg_nvme/lv_4... 2024-05-09T09:10:36.099 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T09:10:36.198 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:36.198 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:36.198 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:36.198 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:36.199 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:36.199 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:36.199 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:36.199 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:36.330 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:10:36.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:36.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:36.453 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:37.242 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:37 smithi163 ceph-mon[34354]: Deploying daemon mgr.z on smithi163 2024-05-09T09:10:37.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:37 smithi103 ceph-mon[34421]: Deploying daemon mgr.z on smithi163 2024-05-09T09:10:37.456 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:37 smithi069 ceph-mon[30123]: Deploying daemon mgr.z on smithi163 2024-05-09T09:10:37.554 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 systemd[1]: Starting Ceph mgr.z for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 podman[36193]: 2024-05-09 09:10:37.535608425 +0000 UTC m=+0.017996370 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 podman[36193]: 2024-05-09 09:10:37.635591001 +0000 UTC m=+0.117978918 container create ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 podman[36193]: 2024-05-09 09:10:37.837958075 +0000 UTC m=+0.320345993 container init ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 podman[36193]: 2024-05-09 09:10:37.840454105 +0000 UTC m=+0.322842024 container start ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc) 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 ceph-mgr[36212]: -- 172.21.15.163:0/3038615632 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5571fda005a0 con 0x5571fd9d4c00 2024-05-09T09:10:37.920 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 bash[36193]: ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 systemd[1]: Started Ceph mgr.z for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:37.991+0000 7f9960918200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:38.176+0000 7f9960918200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:38.176 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:38.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:38.177 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:38.324 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T09:10:38.325 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T09:10:38.325 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-05-09T09:10:38.325 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-05-09T09:10:38.325 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0126884 s, 826 MB/s 2024-05-09T09:10:38.325 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-05-09T09:10:38.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:38.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:38.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:38.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:38.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:38.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:38.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:38.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:38.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-05-09T09:10:38.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:38.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:38.554 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:38.278+0000 7f9960918200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T09:10:38.959 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_4 2024-05-09T09:10:39.054 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:38.582+0000 7f9960918200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T09:10:39.054 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:38.666+0000 7f9960918200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T09:10:39.196 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:10:39.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:39 smithi069 ceph-mon[30123]: Deploying daemon mgr.y on smithi103 2024-05-09T09:10:39.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:39 smithi103 ceph-mon[34421]: Deploying daemon mgr.y on smithi103 2024-05-09T09:10:39.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:39 smithi163 ceph-mon[34354]: Deploying daemon mgr.y on smithi103 2024-05-09T09:10:40.103 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:39.846+0000 7f9960918200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T09:10:40.104 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:40.015+0000 7f9960918200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T09:10:40.104 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:39 smithi103 systemd[1]: Starting Ceph mgr.y for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:10:40.204 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:40 smithi069 ceph-mon[30123]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:40.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:40 smithi103 ceph-mon[34421]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:40.359 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 podman[35800]: 2024-05-09 09:10:40.056566085 +0000 UTC m=+0.022131014 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:10:40.359 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 podman[35800]: 2024-05-09 09:10:40.1588546 +0000 UTC m=+0.124419520 container create 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=squid-96805dc) 2024-05-09T09:10:40.360 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 podman[35800]: 2024-05-09 09:10:40.361194631 +0000 UTC m=+0.326759551 container init 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, RELEASE=squid-96805dc, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:10:40.427 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:40 smithi163 ceph-mon[34354]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:40.427 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:40.193+0000 7f9960918200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T09:10:40.616 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 podman[35800]: 2024-05-09 09:10:40.363852741 +0000 UTC m=+0.329417661 container start 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T09:10:40.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 ceph-mgr[35819]: -- 172.21.15.103:0/2898331467 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b946b705a0 con 0x55b946b45000 2024-05-09T09:10:40.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 bash[35800]: 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 2024-05-09T09:10:40.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 systemd[1]: Started Ceph mgr.y for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:10:40.617 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:40.515+0000 7f10c1dd6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-05-09T09:10:40.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 -- 172.21.15.69:0/3167117069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc102ed0 msgr2=0x7f64bc1052c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 --2- 172.21.15.69:0/3167117069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc102ed0 0x7f64bc1052c0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009990 tx=0x7f64b8009ca0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 -- 172.21.15.69:0/3167117069 shutdown_connections 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 --2- 172.21.15.69:0/3167117069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc108130 0x7f64bc10a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 --2- 172.21.15.69:0/3167117069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc102ed0 0x7f64bc1052c0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 --2- 172.21.15.69:0/3167117069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc105800 0x7f64bc107bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 -- 172.21.15.69:0/3167117069 >> 172.21.15.69:0/3167117069 conn(0x7f64bc0fc820 msgr2=0x7f64bc0fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 -- 172.21.15.69:0/3167117069 shutdown_connections 2024-05-09T09:10:40.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.698+0000 7f64c3f60700 1 -- 172.21.15.69:0/3167117069 wait complete. 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 Processor -- start 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 -- start start 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 0x7f64bc198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc108130 0x7f64bc19fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10df90 con 0x7f64bc105800 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10de10 con 0x7f64bc108130 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c3f60700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64bc10e110 con 0x7f64bc102ed0 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:40.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57900/0 (socket says 172.21.15.69:57900) 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 -- 172.21.15.69:0/36624154 learned_addr learned my addr 172.21.15.69:0/36624154 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c24fd700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc108130 0x7f64bc19fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c1cfc700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 0x7f64bc198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 -- 172.21.15.69:0/36624154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 msgr2=0x7f64bc198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 0x7f64bc198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 -- 172.21.15.69:0/36624154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc108130 msgr2=0x7f64bc19fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc108130 0x7f64bc19fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 -- 172.21.15.69:0/36624154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64bc1a0260 con 0x7f64bc105800 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c1cfc700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 0x7f64bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.699+0000 7f64c14fb700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00d980 tx=0x7f64ac00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac016070 con 0x7f64bc105800 2024-05-09T09:10:40.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64bc1a0550 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64ac00de70 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64ac00f3c0 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64bc1a0ae0 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.700+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 13) ==== 46343+0+0 (secure 0 0 0) 0x7f64ac019070 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.701+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f64bc069010 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.701+0000 7f64b2ffd700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 0x7f64a803b7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.701+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(4..4 src has 1..4) ==== 1155+0+0 (secure 0 0 0) 0x7f64ac04be60 con 0x7f64bc105800 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.701+0000 7f64c1cfc700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 0x7f64a803b7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:10:40.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.701+0000 7f64c1cfc700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 0x7f64a803b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f64b8006010 tx=0x7f64b8003980 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:10:40.704 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.706+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f64ac00fe90 con 0x7f64bc105800 2024-05-09T09:10:40.804 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:40.496+0000 7f9960918200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T09:10:40.804 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:40.593+0000 7f9960918200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T09:10:40.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:40.932+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f64bc062760 con 0x7f64a8039330 2024-05-09T09:10:41.116 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:40.704+0000 7f10c1dd6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:10:41.116 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:40.812+0000 7f10c1dd6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-05-09T09:10:41.247 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:40.806+0000 7f9960918200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T09:10:41.247 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:41.009+0000 7f9960918200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:10:41.472 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:41.140+0000 7f10c1dd6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-05-09T09:10:41.472 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:41.252+0000 7f10c1dd6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-05-09T09:10:41.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.628 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:41.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:41 smithi069 ceph-mon[30123]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:41.764 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:41.654+0000 7f9960918200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T09:10:41.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.764 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:41.765 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:41 smithi163 ceph-mon[34354]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:41.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='client.14223 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:41 smithi103 ceph-mon[34421]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:42.372 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:42.079+0000 7f9960918200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T09:10:42.372 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:42.174+0000 7f9960918200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T09:10:42.373 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:42.373+0000 7f9960918200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T09:10:42.821 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:42 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:42.485+0000 7f10c1dd6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-05-09T09:10:42.821 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:42 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:42.652+0000 7f10c1dd6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-05-09T09:10:42.821 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:42 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:42.822+0000 7f10c1dd6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:43.106 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:43.107 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.214 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:42.888+0000 7f9960918200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T09:10:43.214 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:42.979+0000 7f9960918200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T09:10:43.214 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.214 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:43.215 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-05-09T09:10:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-05-09T09:10:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:43.366 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:43 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:43.107+0000 7f10c1dd6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-05-09T09:10:43.366 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:43 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:43.199+0000 7f10c1dd6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-05-09T09:10:43.551 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:43.214+0000 7f9960918200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T09:10:43.551 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:43.346+0000 7f9960918200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T09:10:43.551 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:43.439+0000 7f9960918200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T09:10:43.551 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:43.552+0000 7f9960918200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T09:10:43.866 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:43 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:43.411+0000 7f10c1dd6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-05-09T09:10:43.866 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:43 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:43.602+0000 7f10c1dd6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: Reconfiguring daemon mgr.x on smithi069 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]: dispatch 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]': finished 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T09:10:44.199 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:10:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T09:10:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: Reconfiguring daemon mgr.x on smithi069 2024-05-09T09:10:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]: dispatch 2024-05-09T09:10:44.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]': finished 2024-05-09T09:10:44.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T09:10:44.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:10:44.304 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:43.852+0000 7f9960918200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T09:10:44.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: Reconfiguring mgr.x (unknown last config time)... 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: Reconfiguring daemon mgr.x on smithi069 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]: dispatch 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/830369199' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54339b38-12b5-46a7-a293-1cea9feb97e6"}]': finished 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: osdmap e5: 1 total, 0 up, 1 in 2024-05-09T09:10:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:10:44.616 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:44.200+0000 7f10c1dd6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-05-09T09:10:44.804 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:44 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:44.442+0000 7f9960918200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T09:10:44.804 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:10:44 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z[36208]: 2024-05-09T09:10:44.652+0000 7f9960918200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T09:10:44.913 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:44.619+0000 7f10c1dd6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-05-09T09:10:44.913 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:44.714+0000 7f10c1dd6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-05-09T09:10:44.914 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:44.914+0000 7f10c1dd6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-05-09T09:10:45.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:45.092+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 14) ==== 92060+0+0 (secure 0 0 0) 0x7f64ac04f030 con 0x7f64bc105800 2024-05-09T09:10:45.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/832938446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: Standby manager daemon z started 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T09:10:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:45 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/832938446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:10:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: Standby manager daemon z started 2024-05-09T09:10:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T09:10:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T09:10:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:45 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:45.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:45.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/832938446' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:10:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: Standby manager daemon z started 2024-05-09T09:10:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-05-09T09:10:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-05-09T09:10:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:45 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.163:0/1053126516' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:45.770 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:45.440+0000 7f10c1dd6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-05-09T09:10:45.770 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:45.531+0000 7f10c1dd6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-05-09T09:10:46.089 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:45.770+0000 7f10c1dd6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-05-09T09:10:46.089 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:45.902+0000 7f10c1dd6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-05-09T09:10:46.089 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:45.997+0000 7f10c1dd6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-05-09T09:10:46.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:46 smithi103 ceph-mon[34421]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:46.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:46 smithi103 ceph-mon[34421]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T09:10:46.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T09:10:46.367 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:46 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:46.113+0000 7f10c1dd6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-05-09T09:10:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:46 smithi069 ceph-mon[30123]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:46 smithi069 ceph-mon[30123]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T09:10:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T09:10:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:46 smithi163 ceph-mon[34354]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:46 smithi163 ceph-mon[34354]: mgrmap e14: x(active, since 76s), standbys: z 2024-05-09T09:10:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-05-09T09:10:46.866 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:46 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:46.433+0000 7f10c1dd6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-05-09T09:10:47.366 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:47 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:47.027+0000 7f10c1dd6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-05-09T09:10:47.366 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:10:47 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y[35815]: 2024-05-09T09:10:47.240+0000 7f10c1dd6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-05-09T09:10:48.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:10:48.103+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f64ac01acd0 con 0x7f64bc105800 2024-05-09T09:10:48.193 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:48.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: Standby manager daemon y started 2024-05-09T09:10:48.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T09:10:48.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:48.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T09:10:48.194 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:48 smithi069 ceph-mon[30123]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:48.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:48.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: Standby manager daemon y started 2024-05-09T09:10:48.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T09:10:48.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:48.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T09:10:48.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:48 smithi103 ceph-mon[34421]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:48.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:48.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: Standby manager daemon y started 2024-05-09T09:10:48.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-05-09T09:10:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-05-09T09:10:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-05-09T09:10:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:48 smithi163 ceph-mon[34354]: from='mgr.? 172.21.15.103:0/4044301093' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-05-09T09:10:49.353 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:49.354 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:49.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: mgrmap e15: x(active, since 79s), standbys: z, y 2024-05-09T09:10:49.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-05-09T09:10:49.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:10:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:50.205 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:50 smithi069 ceph-mon[30123]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:50.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:50 smithi103 ceph-mon[34421]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:50.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:50 smithi163 ceph-mon[34354]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:52.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:52 smithi103 ceph-mon[34421]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:52.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T09:10:52.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:52.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:52 smithi103 ceph-mon[34421]: Deploying daemon osd.0 on smithi069 2024-05-09T09:10:52.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:52 smithi069 ceph-mon[30123]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T09:10:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:52 smithi069 ceph-mon[30123]: Deploying daemon osd.0 on smithi069 2024-05-09T09:10:52.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:52 smithi163 ceph-mon[34354]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:52.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-05-09T09:10:52.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:10:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:52 smithi163 ceph-mon[34354]: Deploying daemon osd.0 on smithi069 2024-05-09T09:10:54.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:54 smithi103 ceph-mon[34421]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:54.442 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:54 smithi069 ceph-mon[30123]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:54 smithi163 ceph-mon[34354]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:56 smithi069 ceph-mon[30123]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:56.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:56 smithi103 ceph-mon[34421]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:56.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:56 smithi163 ceph-mon[34354]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:57.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:57.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:57.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:57 smithi103 ceph-mon[34421]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:58.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:58.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:58.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:57 smithi069 ceph-mon[30123]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:10:58.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:58.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:10:58.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:10:58.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:57 smithi163 ceph-mon[34354]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:00.187 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:00.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:00.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.188 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:10:59 smithi069 ceph-mon[30123]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:00.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.239+0000 7f64b2ffd700 1 -- 172.21.15.69:0/36624154 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f64bc062760 con 0x7f64a8039330 2024-05-09T09:11:00.239 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 0 on host 'smithi069' 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 msgr2=0x7f64a803b7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 0x7f64a803b7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f64b8006010 tx=0x7f64b8003980 comp rx=0 tx=0).stop 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 msgr2=0x7f64bc198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f64ac00d980 tx=0x7f64ac00dc90 comp rx=0 tx=0).stop 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 shutdown_connections 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64a8039330 0x7f64a803b7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:00.241 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64bc108130 0x7f64bc19fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64bc102ed0 0x7f64bc198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 --2- 172.21.15.69:0/36624154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64bc105800 0x7f64bc198b40 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 >> 172.21.15.69:0/36624154 conn(0x7f64bc0fc820 msgr2=0x7f64bc1038f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 shutdown_connections 2024-05-09T09:11:00.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:00.242+0000 7f64c3f60700 1 -- 172.21.15.69:0/36624154 wait complete. 2024-05-09T09:11:00.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:00.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:00.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:10:59 smithi163 ceph-mon[34354]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:00.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:10:59 smithi103 ceph-mon[34421]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:00.867 DEBUG:teuthology.orchestra.run.smithi069:osd.0> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.0.service 2024-05-09T09:11:00.869 INFO:tasks.cephadm:Deploying osd.1 on smithi103 with /dev/vg_nvme/lv_4... 2024-05-09T09:11:00.870 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T09:11:01.116 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:11:01.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:01.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:01 smithi069 ceph-mon[30123]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T09:11:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:01 smithi069 ceph-mon[30123]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:01.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:01.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:01 smithi163 ceph-mon[34354]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T09:11:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:01 smithi163 ceph-mon[34354]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:01.565 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.565 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:01.566 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:01.566 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:01 smithi103 ceph-mon[34421]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-05-09T09:11:01.566 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:01 smithi103 ceph-mon[34421]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:02.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:02 smithi103 ceph-mon[34421]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T09:11:02.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:02 smithi103 ceph-mon[34421]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T09:11:02.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:02 smithi103 ceph-mon[34421]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:11:02.412 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:02.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:02 smithi069 ceph-mon[30123]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T09:11:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:02 smithi069 ceph-mon[30123]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T09:11:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:02 smithi069 ceph-mon[30123]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:11:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:02.510 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0[44850]: 2024-05-09T09:11:02.238+0000 7f1b59678700 -1 osd.0 0 waiting for initial osdmap 2024-05-09T09:11:02.511 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0[44850]: 2024-05-09T09:11:02.241+0000 7f1b5546b700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:11:02.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:02 smithi163 ceph-mon[34354]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-05-09T09:11:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:02 smithi163 ceph-mon[34354]: osdmap e6: 1 total, 0 up, 1 in 2024-05-09T09:11:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:02 smithi163 ceph-mon[34354]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:11:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.001 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T09:11:03.002 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T09:11:03.002 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-09T09:11:03.002 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-09T09:11:03.002 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123612 s, 848 MB/s 2024-05-09T09:11:03.002 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-09T09:11:03.303 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:03 smithi103 ceph-mon[34421]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:11:03.303 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:03 smithi103 ceph-mon[34421]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T09:11:03.303 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.303 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.304 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:03 smithi103 ceph-mon[34421]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:03.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:03 smithi069 ceph-mon[30123]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:11:03.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:03 smithi069 ceph-mon[30123]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T09:11:03.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.395 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:03 smithi069 ceph-mon[30123]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:03.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:03 smithi163 ceph-mon[34354]: from='osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:11:03.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:03 smithi163 ceph-mon[34354]: osdmap e7: 1 total, 0 up, 1 in 2024-05-09T09:11:03.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:03 smithi163 ceph-mon[34354]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-05-09T09:11:03.630 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_4 2024-05-09T09:11:03.867 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] boot 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi069 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 17088M 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:11:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] boot 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi069 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 17088M 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: osd.0 [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] boot 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: osdmap e8: 1 total, 1 up, 1 in 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi069 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 17088M 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.390+0000 7f4dbb39d700 1 -- 172.21.15.103:0/555370645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db4102d40 msgr2=0x7f4db41095f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.390+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/555370645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db4102d40 0x7f4db41095f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f4dac00b0d0 tx=0x7f4dac00b3e0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- 172.21.15.103:0/555370645 shutdown_connections 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/555370645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db4102d40 0x7f4db41095f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/555370645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db41011a0 0x7f4db41015c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/555370645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41023a0 0x7f4db4102800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- 172.21.15.103:0/555370645 >> 172.21.15.103:0/555370645 conn(0x7f4db40fc740 msgr2=0x7f4db40feb80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- 172.21.15.103:0/555370645 shutdown_connections 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- 172.21.15.103:0/555370645 wait complete. 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 Processor -- start 2024-05-09T09:11:05.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- start start 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41011a0 0x7f4db41972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db4102d40 0x7f4db419e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410cd90 con 0x7f4db4102d40 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410cc10 con 0x7f4db41023a0 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.391+0000 7f4dbb39d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4db410cf10 con 0x7f4db41011a0 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:56510/0 (socket says 172.21.15.103:56510) 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 -- 172.21.15.103:0/853158956 learned_addr learned my addr 172.21.15.103:0/853158956 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dba39b700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41011a0 0x7f4db41972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dbab9c700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db4102d40 0x7f4db419e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:05.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 -- 172.21.15.103:0/853158956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41011a0 msgr2=0x7f4db41972f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41011a0 0x7f4db41972f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 -- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db4102d40 msgr2=0x7f4db419e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db4102d40 0x7f4db419e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 -- 172.21.15.103:0/853158956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4db419ef50 con 0x7f4db41023a0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4db9b9a700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4db000d980 tx=0x7f4db000dc90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0017070 con 0x7f4db41023a0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4db419f240 con 0x7f4db41023a0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4db0005020 con 0x7f4db41023a0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4db0009bf0 con 0x7f4db41023a0 2024-05-09T09:11:05.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.392+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4db419f730 con 0x7f4db41023a0 2024-05-09T09:11:05.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.393+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4db4069010 con 0x7f4db41023a0 2024-05-09T09:11:05.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.394+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f4db00051d0 con 0x7f4db41023a0 2024-05-09T09:11:05.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.395+0000 7f4dab7fe700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 0x7f4da00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:05.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.395+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(9..9 src has 1..9) ==== 1767+0+0 (secure 0 0 0) 0x7f4db00cd8e0 con 0x7f4db41023a0 2024-05-09T09:11:05.395 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.395+0000 7f4dba39b700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 0x7f4da00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:05.396 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.396+0000 7f4dba39b700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 0x7f4da00a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4db4102200 tx=0x7f4da400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:05.397 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.399+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4db0020020 con 0x7f4db41023a0 2024-05-09T09:11:05.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:05 smithi069 ceph-mon[30123]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T09:11:05.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:05 smithi069 ceph-mon[30123]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:05 smithi163 ceph-mon[34354]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T09:11:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:05 smithi163 ceph-mon[34354]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:05.599 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:05.600+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f4db4062760 con 0x7f4da00a2b90 2024-05-09T09:11:05.599 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:05 smithi103 ceph-mon[34421]: osdmap e9: 1 total, 1 up, 1 in 2024-05-09T09:11:05.599 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:05 smithi103 ceph-mon[34421]: pgmap v47: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:06 smithi069 ceph-mon[30123]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:06.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:06 smithi103 ceph-mon[34421]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:06.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:06.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:06.531 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:06 smithi163 ceph-mon[34354]: from='client.24143 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:07.410 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:07 smithi103 ceph-mon[34421]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:07 smithi069 ceph-mon[30123]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:07.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:07 smithi163 ceph-mon[34354]: pgmap v48: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:08.538 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:08 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/1180722398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.538 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:08 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.538 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:08 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]': finished 2024-05-09T09:11:08.538 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:08 smithi103 ceph-mon[34421]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T09:11:08.539 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:08 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:08 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/1180722398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:08 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:08 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]': finished 2024-05-09T09:11:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:08 smithi163 ceph-mon[34354]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T09:11:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:08 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:08.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:08 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/1180722398' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:08 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]: dispatch 2024-05-09T09:11:08.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:08 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9edd74d4-c926-46b9-b564-36ea82da5321"}]': finished 2024-05-09T09:11:08.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:08 smithi069 ceph-mon[30123]: osdmap e10: 2 total, 1 up, 2 in 2024-05-09T09:11:08.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:09.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:09 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/1766355413' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:09.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:09 smithi163 ceph-mon[34354]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:09.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:09 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/1766355413' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:09 smithi103 ceph-mon[34421]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:09.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:09 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/1766355413' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:09.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:09 smithi069 ceph-mon[30123]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:12.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:12 smithi103 ceph-mon[34421]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:12 smithi069 ceph-mon[30123]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:12.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:12 smithi163 ceph-mon[34354]: pgmap v51: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:14.344 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:14 smithi103 ceph-mon[34421]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:14.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:14 smithi069 ceph-mon[30123]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:14.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:14 smithi163 ceph-mon[34354]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:16.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:16 smithi103 ceph-mon[34421]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:16.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T09:11:16.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:16.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:16 smithi069 ceph-mon[30123]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:16.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T09:11:16.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:16.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:16 smithi163 ceph-mon[34354]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:16.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-05-09T09:11:16.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:17.352 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:17 smithi103 ceph-mon[34421]: Deploying daemon osd.1 on smithi103 2024-05-09T09:11:17.411 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:17 smithi069 ceph-mon[30123]: Deploying daemon osd.1 on smithi103 2024-05-09T09:11:17.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:17 smithi163 ceph-mon[34354]: Deploying daemon osd.1 on smithi103 2024-05-09T09:11:18.336 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:18 smithi103 ceph-mon[34421]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:18.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:18 smithi069 ceph-mon[30123]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:18.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:18 smithi163 ceph-mon[34354]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:20.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:20 smithi103 ceph-mon[34421]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:20 smithi069 ceph-mon[30123]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:20.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:20 smithi163 ceph-mon[34354]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:22.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:22.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:21 smithi069 ceph-mon[30123]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:22.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:22.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:21 smithi163 ceph-mon[34354]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:22.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:22.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:22.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:21 smithi103 ceph-mon[34421]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:23.613 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.614+0000 7f4dab7fe700 1 -- 172.21.15.103:0/853158956 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f4db4062760 con 0x7f4da00a2b90 2024-05-09T09:11:23.614 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 1 on host 'smithi103' 2024-05-09T09:11:23.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 msgr2=0x7f4da00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:23.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 0x7f4da00a5050 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f4db4102200 tx=0x7f4da400b410 comp rx=0 tx=0).stop 2024-05-09T09:11:23.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 msgr2=0x7f4db4197830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:23.615 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4db000d980 tx=0x7f4db000dc90 comp rx=0 tx=0).stop 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 shutdown_connections 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4db4102d40 0x7f4db419e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4db41011a0 0x7f4db41972f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4da00a2b90 0x7f4da00a5050 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 --2- 172.21.15.103:0/853158956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4db41023a0 0x7f4db4197830 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 >> 172.21.15.103:0/853158956 conn(0x7f4db40fc740 msgr2=0x7f4db41055d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 shutdown_connections 2024-05-09T09:11:23.616 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:11:23.616+0000 7f4dbb39d700 1 -- 172.21.15.103:0/853158956 wait complete. 2024-05-09T09:11:24.225 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:24 smithi103 ceph-mon[34421]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:24.225 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.225 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.252 DEBUG:teuthology.orchestra.run.smithi103:osd.1> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.1.service 2024-05-09T09:11:24.254 INFO:tasks.cephadm:Deploying osd.2 on smithi163 with /dev/vg_nvme/lv_4... 2024-05-09T09:11:24.255 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-05-09T09:11:24.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:24 smithi069 ceph-mon[30123]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:24.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.514 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:11:24.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:24 smithi163 ceph-mon[34354]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:24.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:24.617 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:11:24.474+0000 7ff10a8837c0 -1 osd.1 0 log_to_monitors true 2024-05-09T09:11:25.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:25 smithi163 ceph-mon[34354]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:25 smithi163 ceph-mon[34354]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:25.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:25 smithi103 ceph-mon[34421]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:25 smithi103 ceph-mon[34421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:25.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:25 smithi069 ceph-mon[30123]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:25 smithi069 ceph-mon[30123]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-05-09T09:11:25.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:25.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0142105 s, 738 MB/s 2024-05-09T09:11:26.263 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:26.264 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:26 smithi163 ceph-mon[34354]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.366 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:11:26.115+0000 7ff0ffee7700 -1 osd.1 0 waiting for initial osdmap 2024-05-09T09:11:26.367 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:11:26.118+0000 7ff0fbcda700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:26 smithi103 ceph-mon[34421]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: from='osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: osdmap e11: 2 total, 1 up, 2 in 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:26 smithi069 ceph-mon[30123]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:11:26.914 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_4 2024-05-09T09:11:27.166 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:11:27.278 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:27 smithi163 ceph-mon[34354]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:11:27.279 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:27 smithi163 ceph-mon[34354]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T09:11:27.279 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.279 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.279 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:27 smithi103 ceph-mon[34421]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:11:27.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:27 smithi103 ceph-mon[34421]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T09:11:27.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.384 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:27 smithi069 ceph-mon[30123]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:11:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:27 smithi069 ceph-mon[30123]: osdmap e12: 2 total, 1 up, 2 in 2024-05-09T09:11:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:28.283 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:11:28.283 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] boot 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:28.284 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:28.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:11:28.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:11:28.683 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] boot 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:28.684 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: osd.1 [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] boot 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: osdmap e13: 2 total, 2 up, 2 in 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:28.685 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:28.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:28.686 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/2983599844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 msgr2=0x7f6b881023f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/2983599844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881023f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000b0a0 tx=0x7f6b8000b3b0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/2983599844 shutdown_connections 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/2983599844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881023f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/2983599844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b88100c50 0x7f6b88101070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/2983599844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b881015b0 0x7f6b88101a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/2983599844 >> 172.21.15.163:0/2983599844 conn(0x7f6b880fc810 msgr2=0x7f6b880fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/2983599844 shutdown_connections 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.791+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/2983599844 wait complete. 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 Processor -- start 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 -- start start 2024-05-09T09:11:28.792 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b88100c50 0x7f6b8819a780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b881015b0 0x7f6b8819acc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6b881101c0 con 0x7f6b88100c50 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6b88110040 con 0x7f6b881015b0 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8f8fe700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6b88110340 con 0x7f6b88101f70 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:57154/0 (socket says 172.21.15.163:57154) 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 -- 172.21.15.163:0/343285801 learned_addr learned my addr 172.21.15.163:0/343285801 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8d69a700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b88100c50 0x7f6b8819a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8ce99700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b881015b0 0x7f6b8819acc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 -- 172.21.15.163:0/343285801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b881015b0 msgr2=0x7f6b8819acc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b881015b0 0x7f6b8819acc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 -- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b88100c50 msgr2=0x7f6b8819a780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:28.793 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b88100c50 0x7f6b8819a780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 -- 172.21.15.163:0/343285801 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6b881a23e0 con 0x7f6b88101f70 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.792+0000 7f6b8de9b700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d010 tx=0x7f6b8000be60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.793+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b80021070 con 0x7f6b88101f70 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.793+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6b881a2670 con 0x7f6b88101f70 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.793+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6b80007c80 con 0x7f6b88101f70 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.793+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6b8001d6b0 con 0x7f6b88101f70 2024-05-09T09:11:28.794 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.793+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6b881a2b20 con 0x7f6b88101f70 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.794+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6b88069010 con 0x7f6b88101f70 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.795+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f6b800042d0 con 0x7f6b88101f70 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.795+0000 7f6b7e7fc700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 0x7f6b740a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.795+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(14..14 src has 1..14) ==== 2309+0+0 (secure 0 0 0) 0x7f6b8000f020 con 0x7f6b88101f70 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.796+0000 7f6b8d69a700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 0x7f6b740a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:28.795 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.796+0000 7f6b8d69a700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 0x7f6b740a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6b84005950 tx=0x7f6b8400a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:28.798 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:28.800+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6b8002a020 con 0x7f6b88101f70 2024-05-09T09:11:29.002 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:29.003+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) -- 0x7f6b88062760 con 0x7f6b740a2b90 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi103 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 17088M 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:29.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi103 2024-05-09T09:11:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 17088M 2024-05-09T09:11:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi103 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 17088M 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: osdmap e14: 2 total, 2 up, 2 in 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:30.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:30 smithi163 ceph-mon[34354]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:30.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:30 smithi163 ceph-mon[34354]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:30 smithi069 ceph-mon[30123]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:30 smithi069 ceph-mon[30123]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:30.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:30 smithi103 ceph-mon[34421]: from='client.24172 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:30.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:30 smithi103 ceph-mon[34421]: pgmap v64: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/2739771623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]': finished 2024-05-09T09:11:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T09:11:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/2739771623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]': finished 2024-05-09T09:11:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T09:11:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:32.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: pgmap v65: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:32.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/2739771623' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]: dispatch 2024-05-09T09:11:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b8268e98-1c34-44a4-b53f-779dd3ea68ab"}]': finished 2024-05-09T09:11:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: osdmap e15: 3 total, 2 up, 3 in 2024-05-09T09:11:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:33 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/4151712876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:33.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:33 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/4151712876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:33.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:33 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/4151712876' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:34.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:34 smithi069 ceph-mon[30123]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:34.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:34 smithi163 ceph-mon[34354]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:34.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:34 smithi103 ceph-mon[34421]: pgmap v67: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:36.475 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:36 smithi163 ceph-mon[34354]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:36 smithi069 ceph-mon[30123]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:36.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:36 smithi103 ceph-mon[34421]: pgmap v68: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:38.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:38 smithi163 ceph-mon[34354]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:38.419 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:38 smithi103 ceph-mon[34421]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:38 smithi069 ceph-mon[30123]: pgmap v69: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:39.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:39 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T09:11:39.477 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:39 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:39 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T09:11:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:39 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:39.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:39 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-05-09T09:11:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:39 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:40.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:40 smithi163 ceph-mon[34354]: Deploying daemon osd.2 on smithi163 2024-05-09T09:11:40.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:40 smithi163 ceph-mon[34354]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:40 smithi069 ceph-mon[30123]: Deploying daemon osd.2 on smithi163 2024-05-09T09:11:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:40 smithi069 ceph-mon[30123]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:40.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:40 smithi103 ceph-mon[34421]: Deploying daemon osd.2 on smithi163 2024-05-09T09:11:40.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:40 smithi103 ceph-mon[34421]: pgmap v70: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:42.469 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:42 smithi163 ceph-mon[34354]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:42.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:42 smithi069 ceph-mon[30123]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:42.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:42 smithi103 ceph-mon[34421]: pgmap v71: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:44.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:44 smithi163 ceph-mon[34354]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:44.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:44 smithi069 ceph-mon[30123]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:44 smithi103 ceph-mon[34421]: pgmap v72: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:45.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:45 smithi163 ceph-mon[34354]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:45.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:45 smithi103 ceph-mon[34421]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:45.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:45.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:45.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:45 smithi069 ceph-mon[30123]: pgmap v73: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:47.768 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.770+0000 7f6b7e7fc700 1 -- 172.21.15.163:0/343285801 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f6b88062760 con 0x7f6b740a2b90 2024-05-09T09:11:47.769 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 2 on host 'smithi163' 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 msgr2=0x7f6b740a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 0x7f6b740a5050 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6b84005950 tx=0x7f6b8400a300 comp rx=0 tx=0).stop 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 msgr2=0x7f6b881a1d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f6b8000d010 tx=0x7f6b8000be60 comp rx=0 tx=0).stop 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 shutdown_connections 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6b88101f70 0x7f6b881a1d40 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:47.771 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6b88100c50 0x7f6b8819a780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:47.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6b740a2b90 0x7f6b740a5050 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:47.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 --2- 172.21.15.163:0/343285801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6b881015b0 0x7f6b8819acc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:47.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 >> 172.21.15.163:0/343285801 conn(0x7f6b880fc810 msgr2=0x7f6b880fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:47.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 shutdown_connections 2024-05-09T09:11:47.772 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:11:47.773+0000 7f6b8f8fe700 1 -- 172.21.15.163:0/343285801 wait complete. 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:47 smithi069 ceph-mon[30123]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:48.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:47 smithi163 ceph-mon[34354]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:48.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:47 smithi103 ceph-mon[34421]: pgmap v74: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:48.436 DEBUG:teuthology.orchestra.run.smithi163:osd.2> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.2.service 2024-05-09T09:11:48.439 INFO:tasks.cephadm:Deploying osd.3 on smithi069 with /dev/vg_nvme/lv_3... 2024-05-09T09:11:48.439 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T09:11:48.687 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:11:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:48 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:48 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:48 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:48 smithi069 ceph-mon[30123]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:48 smithi069 ceph-mon[30123]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:48 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:48 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:48 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:49.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:48 smithi163 ceph-mon[34354]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:48 smithi163 ceph-mon[34354]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:48 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:48 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:48 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:11:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:48 smithi103 ceph-mon[34421]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:48 smithi103 ceph-mon[34421]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-05-09T09:11:49.927 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T09:11:49.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T09:11:49.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:49.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:49.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:49.928 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:49 smithi163 ceph-mon[34354]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:49.928 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2[39509]: 2024-05-09T09:11:49.778+0000 7fd6ce718700 -1 osd.2 0 waiting for initial osdmap 2024-05-09T09:11:49.928 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2[39509]: 2024-05-09T09:11:49.781+0000 7fd6ca50b700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:11:49.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T09:11:49.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T09:11:49.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:49.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:49.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:49.995 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:49 smithi069 ceph-mon[30123]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:50.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-05-09T09:11:50.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: osdmap e16: 3 total, 2 up, 3 in 2024-05-09T09:11:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: from='osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:11:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:49 smithi103 ceph-mon[34421]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T09:11:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T09:11:50.681 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-05-09T09:11:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-05-09T09:11:50.681 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0150285 s, 698 MB/s 2024-05-09T09:11:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-05-09T09:11:51.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:50 smithi069 ceph-mon[30123]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:11:51.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:50 smithi069 ceph-mon[30123]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T09:11:51.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:50 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:50 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:50 smithi163 ceph-mon[34354]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:11:51.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:50 smithi163 ceph-mon[34354]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T09:11:51.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:50 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:50 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:50 smithi103 ceph-mon[34421]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:11:51.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:50 smithi103 ceph-mon[34421]: osdmap e17: 3 total, 2 up, 3 in 2024-05-09T09:11:51.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:50 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:50 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.359 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_3 2024-05-09T09:11:51.613 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:11:51.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:11:51.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:11:51.993 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] boot 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:51.994 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] boot 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:52.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:52.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:52.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: osd.2 [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] boot 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: osdmap e18: 3 total, 3 up, 3 in 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:11:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:53.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:52 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi163 2024-05-09T09:11:53.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:52 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-09T09:11:53.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:52 smithi163 ceph-mon[34354]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T09:11:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:53.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:52 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi163 2024-05-09T09:11:53.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:52 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-09T09:11:53.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:52 smithi103 ceph-mon[34421]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T09:11:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:53.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:52 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi163 2024-05-09T09:11:53.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:52 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 17088M 2024-05-09T09:11:53.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:53.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:52 smithi069 ceph-mon[30123]: osdmap e19: 3 total, 3 up, 3 in 2024-05-09T09:11:53.140 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:11:53.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 -- 172.21.15.69:0/1990272678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 msgr2=0x7f08201096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 --2- 172.21.15.69:0/1990272678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f08201096c0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f081800b0d0 tx=0x7f081800b3e0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 -- 172.21.15.69:0/1990272678 shutdown_connections 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 --2- 172.21.15.69:0/1990272678 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f08201096c0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 --2- 172.21.15.69:0/1990272678 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820101270 0x7f0820101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 --2- 172.21.15.69:0/1990272678 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820102470 0x7f08201028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 -- 172.21.15.69:0/1990272678 >> 172.21.15.69:0/1990272678 conn(0x7f08200fc810 msgr2=0x7f08200fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 -- 172.21.15.69:0/1990272678 shutdown_connections 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.289+0000 7f0826a2a700 1 -- 172.21.15.69:0/1990272678 wait complete. 2024-05-09T09:11:53.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.290+0000 7f0826a2a700 1 Processor -- start 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.290+0000 7f0826a2a700 1 -- start start 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820101270 0x7f0820197400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820102470 0x7f0820197940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f082010ce50 con 0x7f0820102e10 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f082010ccd0 con 0x7f0820101270 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f082010cfd0 con 0x7f0820102470 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:53.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46350/0 (socket says 172.21.15.69:46350) 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 -- 172.21.15.69:0/3302157935 learned_addr learned my addr 172.21.15.69:0/3302157935 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f081f7fe700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820102470 0x7f0820197940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f081ffff700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820101270 0x7f0820197400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 -- 172.21.15.69:0/3302157935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820102470 msgr2=0x7f0820197940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820102470 0x7f0820197940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 -- 172.21.15.69:0/3302157935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820101270 msgr2=0x7f0820197400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820101270 0x7f0820197400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 -- 172.21.15.69:0/3302157935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f082019f060 con 0x7f0820102e10 2024-05-09T09:11:53.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0824fc7700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f081800b770 tx=0x7f0818007a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0818003c00 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f082019f2f0 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0818003da0 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f081801b9d0 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.291+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f082019f740 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.292+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0820069010 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.294+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 15) ==== 137777+0+0 (secure 0 0 0) 0x7f0818005670 con 0x7f0820102e10 2024-05-09T09:11:53.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.294+0000 7f081d7fa700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 0x7f08080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:11:53.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.294+0000 7f081ffff700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 0x7f08080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:11:53.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.295+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(20..20 src has 1..20) ==== 3237+0+0 (secure 0 0 0) 0x7f081800f020 con 0x7f0820102e10 2024-05-09T09:11:53.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.295+0000 7f081ffff700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 0x7f08080a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0810005950 tx=0x7f081000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:11:53.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.298+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f081801a3c0 con 0x7f0820102e10 2024-05-09T09:11:53.409 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49200]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:53.410 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49200]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:53.410 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49200]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:53.410 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49200]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:53.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.513+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f0820062760 con 0x7f08080a2b90 2024-05-09T09:11:53.759 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49225]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:53.759 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49225]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:53.759 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49225]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:53.759 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:11:53 smithi069 sudo[49225]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:53.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:11:53.811+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f0818099250 con 0x7f0820102e10 2024-05-09T09:11:53.932 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43157]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:53.932 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43157]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:53.932 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43157]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:53.932 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43157]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:53.933 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:53 smithi103 ceph-mon[34421]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T09:11:54.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:54.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T09:11:54.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:53 smithi163 ceph-mon[34354]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: osdmap e20: 3 total, 3 up, 3 in 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: pgmap v82: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:11:54.126 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:11:54.127 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:11:54.127 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:53 smithi069 ceph-mon[30123]: osdmap e21: 3 total, 3 up, 3 in 2024-05-09T09:11:54.366 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43160]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:54.366 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43160]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:54.367 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43160]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:54.367 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:11:53 smithi103 sudo[43160]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:54.452 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43566]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:54.452 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43566]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:54.452 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43566]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:54.452 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43566]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:54.804 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43569]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:54.804 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43569]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:54.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43569]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:54.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:11:54 smithi163 sudo[43569]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49366]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:54.975 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49366]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49366]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:54.976 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49366]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:54 smithi163 ceph-mon[34354]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:54 smithi163 ceph-mon[34354]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:54 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43572]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43572]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43572]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:55.236 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43572]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:55.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49374]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:55.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49374]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:55.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49374]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:55.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 sudo[49374]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 ceph-mon[30123]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 ceph-mon[30123]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T09:11:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:54 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:55.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:54 smithi103 ceph-mon[34421]: from='client.14289 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:11:55.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:54 smithi103 ceph-mon[34421]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-05-09T09:11:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:11:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:54 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:55.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43575]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:55.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43575]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:55.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43575]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:55.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 sudo[43575]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:55.964 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43163]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-05-09T09:11:55.964 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43163]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:55.964 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43163]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:55.964 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43163]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:55 smithi069 ceph-mon[30123]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:55 smithi163 ceph-mon[34354]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43166]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-05-09T09:11:56.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43166]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43166]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 sudo[43166]: pam_unix(sudo:session): session closed for user root 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: pgmap v84: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-05-09T09:11:56.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-05-09T09:11:56.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-05-09T09:11:56.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-05-09T09:11:56.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:55 smithi103 ceph-mon[34421]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-05-09T09:11:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]: dispatch 2024-05-09T09:11:57.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]': finished 2024-05-09T09:11:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:56 smithi069 ceph-mon[30123]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T09:11:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:11:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1781709704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:57.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]: dispatch 2024-05-09T09:11:57.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]': finished 2024-05-09T09:11:57.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:56 smithi163 ceph-mon[34354]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T09:11:57.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:11:57.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1781709704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:57.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]: dispatch 2024-05-09T09:11:57.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3659804622' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ebea5b0f-0931-447f-848c-342af0a730a7"}]': finished 2024-05-09T09:11:57.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:56 smithi103 ceph-mon[34421]: osdmap e22: 4 total, 3 up, 4 in 2024-05-09T09:11:57.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:11:57.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1781709704' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:11:58.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:11:57 smithi069 ceph-mon[30123]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:58.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:11:57 smithi163 ceph-mon[34354]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:11:58.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:11:57 smithi103 ceph-mon[34421]: pgmap v86: 1 pgs: 1 active+clean; 449 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:00.240 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:00 smithi069 ceph-mon[30123]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 348 KiB/s wr, 23 op/s 2024-05-09T09:12:00.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:00 smithi103 ceph-mon[34421]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 348 KiB/s wr, 23 op/s 2024-05-09T09:12:00.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:00 smithi163 ceph-mon[34354]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 6.9 KiB/s rd, 348 KiB/s wr, 23 op/s 2024-05-09T09:12:02.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:02 smithi103 ceph-mon[34421]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-05-09T09:12:02.449 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:02 smithi069 ceph-mon[30123]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-05-09T09:12:02.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:02 smithi163 ceph-mon[34354]: pgmap v88: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 5.0 KiB/s rd, 252 KiB/s wr, 16 op/s 2024-05-09T09:12:04.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:04 smithi103 ceph-mon[34421]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-05-09T09:12:04.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T09:12:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:04.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:04 smithi069 ceph-mon[30123]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-05-09T09:12:04.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T09:12:04.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:04 smithi163 ceph-mon[34354]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.9 KiB/s rd, 198 KiB/s wr, 13 op/s 2024-05-09T09:12:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-05-09T09:12:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:05.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:05 smithi103 ceph-mon[34421]: Deploying daemon osd.3 on smithi069 2024-05-09T09:12:05.458 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:05 smithi069 ceph-mon[30123]: Deploying daemon osd.3 on smithi069 2024-05-09T09:12:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:05 smithi163 ceph-mon[34354]: Deploying daemon osd.3 on smithi069 2024-05-09T09:12:06.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:06 smithi103 ceph-mon[34421]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T09:12:06.409 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:06 smithi069 ceph-mon[30123]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T09:12:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:06 smithi163 ceph-mon[34354]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.6 KiB/s rd, 183 KiB/s wr, 12 op/s 2024-05-09T09:12:08.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:08 smithi103 ceph-mon[34421]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-09T09:12:08.379 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:08 smithi163 ceph-mon[34354]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-09T09:12:08.468 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:08 smithi069 ceph-mon[30123]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.3 KiB/s rd, 169 KiB/s wr, 11 op/s 2024-05-09T09:12:10.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:10 smithi103 ceph-mon[34421]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T09:12:10.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:10 smithi069 ceph-mon[30123]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T09:12:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:10.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:10 smithi163 ceph-mon[34354]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 3.0 KiB/s rd, 152 KiB/s wr, 10 op/s 2024-05-09T09:12:10.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:12.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:12.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.338 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.339 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:12.339 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:12.339 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:12.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.978+0000 7f081d7fa700 1 -- 172.21.15.69:0/3302157935 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f0820062760 con 0x7f08080a2b90 2024-05-09T09:12:12.978 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 3 on host 'smithi069' 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 msgr2=0x7f08080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 0x7f08080a5050 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f0810005950 tx=0x7f081000a300 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 msgr2=0x7f082019e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f081800b770 tx=0x7f0818007a20 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 shutdown_connections 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0820102e10 0x7f082019e9c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08080a2b90 0x7f08080a5050 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0820101270 0x7f0820197400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 --2- 172.21.15.69:0/3302157935 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0820102470 0x7f0820197940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 >> 172.21.15.69:0/3302157935 conn(0x7f08200fc810 msgr2=0x7f0820105710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:12:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 shutdown_connections 2024-05-09T09:12:12.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:12:12.982+0000 7f0826a2a700 1 -- 172.21.15.69:0/3302157935 wait complete. 2024-05-09T09:12:13.688 DEBUG:teuthology.orchestra.run.smithi069:osd.3> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.3.service 2024-05-09T09:12:13.690 INFO:tasks.cephadm:Deploying osd.4 on smithi103 with /dev/vg_nvme/lv_3... 2024-05-09T09:12:13.690 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T09:12:13.937 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:12:14.059 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.059 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:14.059 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.059 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:13 smithi103 ceph-mon[34421]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:14.059 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:13 smithi103 ceph-mon[34421]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T09:12:14.214 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.215 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:14.215 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.215 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:13 smithi069 ceph-mon[30123]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:14.215 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:13 smithi069 ceph-mon[30123]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T09:12:14.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:14.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:14.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:13 smithi163 ceph-mon[34354]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:14.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:13 smithi163 ceph-mon[34354]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-05-09T09:12:15.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:14 smithi069 ceph-mon[30123]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T09:12:15.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:14 smithi069 ceph-mon[30123]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T09:12:15.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:14 smithi069 ceph-mon[30123]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:12:15.093 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:14 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:15.093 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:12:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:12:14.982+0000 7f0e4d6b9700 -1 osd.3 0 waiting for initial osdmap 2024-05-09T09:12:15.093 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:12:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:12:14.990+0000 7f0e494ac700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:12:15.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:14 smithi163 ceph-mon[34354]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T09:12:15.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:14 smithi163 ceph-mon[34354]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T09:12:15.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:14 smithi163 ceph-mon[34354]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:12:15.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:14 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:15.331 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:14 smithi103 ceph-mon[34421]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-05-09T09:12:15.331 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:14 smithi103 ceph-mon[34421]: osdmap e23: 4 total, 3 up, 4 in 2024-05-09T09:12:15.331 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:14 smithi103 ceph-mon[34421]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:12:15.331 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:14 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.052 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T09:12:16.052 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T09:12:16.053 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-09T09:12:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-09T09:12:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0149336 s, 702 MB/s 2024-05-09T09:12:16.053 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] boot 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T09:12:16.192 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:12:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T09:12:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:16.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] boot 2024-05-09T09:12:16.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T09:12:16.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: from='osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: osdmap e24: 4 total, 3 up, 4 in 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: pgmap v97: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: osd.3 [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] boot 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: osdmap e25: 4 total, 4 up, 4 in 2024-05-09T09:12:16.359 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-05-09T09:12:16.681 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_3 2024-05-09T09:12:16.926 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:12:17.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:12:17.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:12:17.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T09:12:17.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:12:17.305 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: osdmap e26: 4 total, 4 up, 4 in 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:17.306 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:18.075 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:17 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi069 2024-05-09T09:12:18.076 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:17 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 8544M 2024-05-09T09:12:18.076 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:17 smithi103 ceph-mon[34421]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:18.076 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:17 smithi103 ceph-mon[34421]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T09:12:18.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:17 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi069 2024-05-09T09:12:18.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:17 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 8544M 2024-05-09T09:12:18.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:17 smithi069 ceph-mon[30123]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:18.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:17 smithi069 ceph-mon[30123]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T09:12:18.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:17 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi069 2024-05-09T09:12:18.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:17 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 8544M 2024-05-09T09:12:18.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:17 smithi163 ceph-mon[34354]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 106 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:18.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:17 smithi163 ceph-mon[34354]: osdmap e27: 4 total, 4 up, 4 in 2024-05-09T09:12:18.619 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 -- 172.21.15.103:0/1381327254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe0102e40 msgr2=0x7fabe01096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 --2- 172.21.15.103:0/1381327254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe0102e40 0x7fabe01096f0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fabd800b0d0 tx=0x7fabd800b3e0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 -- 172.21.15.103:0/1381327254 shutdown_connections 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 --2- 172.21.15.103:0/1381327254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe0102e40 0x7fabe01096f0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 --2- 172.21.15.103:0/1381327254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe01012a0 0x7fabe01016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 --2- 172.21.15.103:0/1381327254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01024a0 0x7fabe0102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 -- 172.21.15.103:0/1381327254 >> 172.21.15.103:0/1381327254 conn(0x7fabe00fc880 msgr2=0x7fabe00feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 -- 172.21.15.103:0/1381327254 shutdown_connections 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 -- 172.21.15.103:0/1381327254 wait complete. 2024-05-09T09:12:18.620 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.618+0000 7fabe7386700 1 Processor -- start 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 -- start start 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 0x7fabe006b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe0102e40 0x7fabe019f910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabe010cee0 con 0x7fabe0102e40 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabe010cd60 con 0x7fabe01024a0 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe7386700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fabe010d060 con 0x7fabe01012a0 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:18.621 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:38050/0 (socket says 172.21.15.103:38050) 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 -- 172.21.15.103:0/3472430502 learned_addr learned my addr 172.21.15.103:0/3472430502 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe5923700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe0102e40 0x7fabe019f910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe5122700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 0x7fabe006b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 -- 172.21.15.103:0/3472430502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 msgr2=0x7fabe006b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 0x7fabe006b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 -- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe0102e40 msgr2=0x7fabe019f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe0102e40 0x7fabe019f910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 -- 172.21.15.103:0/3472430502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabe01a01e0 con 0x7fabe01024a0 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe5122700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 0x7fabe006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:12:18.622 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabe4921700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fabd000b530 tx=0x7fabd000b840 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.619+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd0015020 con 0x7fabe01024a0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.620+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd0011d10 con 0x7fabe01024a0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.620+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabe01a04d0 con 0x7fabe01024a0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.620+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd000f2e0 con 0x7fabe01024a0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.620+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabe01a09c0 con 0x7fabe01024a0 2024-05-09T09:12:18.623 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.621+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabe0069010 con 0x7fabe01024a0 2024-05-09T09:12:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.621+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fabd0018020 con 0x7fabe01024a0 2024-05-09T09:12:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.622+0000 7fabd67fc700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 0x7fabcc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.622+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(28..28 src has 1..28) ==== 3669+0+0 (secure 0 0 0) 0x7fabd00cd6e0 con 0x7fabe01024a0 2024-05-09T09:12:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.622+0000 7fabe5122700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 0x7fabcc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:18.624 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.623+0000 7fabe5122700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 0x7fabcc0a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabe0102300 tx=0x7fabdc00b670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:12:18.625 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.627+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fabd001f020 con 0x7fabe01024a0 2024-05-09T09:12:18.832 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:18.833+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7fabe0062760 con 0x7fabcc0a2c20 2024-05-09T09:12:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T09:12:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:19.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:19 smithi103 ceph-mon[34421]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:12:19.652 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T09:12:19.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:19.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:19.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:19.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:19.653 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:19 smithi163 ceph-mon[34354]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: osdmap e28: 4 total, 4 up, 4 in 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: from='client.24209 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:19.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:19 smithi069 ceph-mon[30123]: pgmap v103: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:12:20.615 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:20 smithi103 ceph-mon[34421]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T09:12:20.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:20 smithi069 ceph-mon[30123]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T09:12:20.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:20 smithi163 ceph-mon[34354]: osdmap e29: 4 total, 4 up, 4 in 2024-05-09T09:12:21.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:21 smithi069 ceph-mon[30123]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:12:21.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:21 smithi163 ceph-mon[34354]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:12:21.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:21 smithi103 ceph-mon[34421]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/172732860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]': finished 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:22.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:22 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/422720825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:22.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/172732860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]': finished 2024-05-09T09:12:22.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T09:12:22.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:22.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:22 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/422720825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:22.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/172732860' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]: dispatch 2024-05-09T09:12:22.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7cf7c161-cb83-47cc-be48-1a7bcba78412"}]': finished 2024-05-09T09:12:22.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: osdmap e30: 5 total, 4 up, 5 in 2024-05-09T09:12:22.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:22.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:22 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/422720825' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:23.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:23 smithi069 ceph-mon[30123]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:12:23.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:23 smithi163 ceph-mon[34354]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:12:23.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:23 smithi103 ceph-mon[34421]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:12:26.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:26 smithi103 ceph-mon[34421]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:26 smithi069 ceph-mon[30123]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:26.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:26 smithi163 ceph-mon[34354]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:28.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:28 smithi103 ceph-mon[34421]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:28 smithi069 ceph-mon[30123]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:28.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:28 smithi163 ceph-mon[34354]: pgmap v109: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:29.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T09:12:29.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T09:12:29.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:29.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-05-09T09:12:29.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:30.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:30 smithi103 ceph-mon[34421]: Deploying daemon osd.4 on smithi103 2024-05-09T09:12:30.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:30 smithi103 ceph-mon[34421]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:30 smithi069 ceph-mon[30123]: Deploying daemon osd.4 on smithi103 2024-05-09T09:12:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:30 smithi069 ceph-mon[30123]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:30 smithi163 ceph-mon[34354]: Deploying daemon osd.4 on smithi103 2024-05-09T09:12:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:30 smithi163 ceph-mon[34354]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:32.301 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:32 smithi103 ceph-mon[34421]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:32 smithi069 ceph-mon[30123]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:32.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:32 smithi163 ceph-mon[34354]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:34.390 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:34 smithi103 ceph-mon[34421]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:34.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:34 smithi069 ceph-mon[30123]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:34.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:34 smithi163 ceph-mon[34354]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:35.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:35.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:35 smithi069 ceph-mon[30123]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:35.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:35.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:35 smithi163 ceph-mon[34354]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:35.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:35.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:35.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:35 smithi103 ceph-mon[34421]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:38.308 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:38.308 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.308 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:38.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:38.309 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.354 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.355+0000 7fabd67fc700 1 -- 172.21.15.103:0/3472430502 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fabe0062760 con 0x7fabcc0a2c20 2024-05-09T09:12:38.354 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 4 on host 'smithi103' 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.357+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 msgr2=0x7fabcc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.357+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 0x7fabcc0a50e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fabe0102300 tx=0x7fabdc00b670 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.357+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 msgr2=0x7fabe019d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.357+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fabd000b530 tx=0x7fabd000b840 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.357+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 shutdown_connections 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabe0102e40 0x7fabe019f910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabe01012a0 0x7fabe006b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabcc0a2c20 0x7fabcc0a50e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 --2- 172.21.15.103:0/3472430502 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabe01024a0 0x7fabe019d2d0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 >> 172.21.15.103:0/3472430502 conn(0x7fabe00fc880 msgr2=0x7fabe01057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 shutdown_connections 2024-05-09T09:12:38.357 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:12:38.358+0000 7fabe7386700 1 -- 172.21.15.103:0/3472430502 wait complete. 2024-05-09T09:12:38.507 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:38.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:38.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:38.508 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.072 DEBUG:teuthology.orchestra.run.smithi103:osd.4> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.4.service 2024-05-09T09:12:39.074 INFO:tasks.cephadm:Deploying osd.5 on smithi163 with /dev/vg_nvme/lv_3... 2024-05-09T09:12:39.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-05-09T09:12:39.313 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:39 smithi103 ceph-mon[34421]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.313 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:39 smithi103 ceph-mon[34421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.313 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:39 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.314 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:39 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.314 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:39 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:39.321 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:12:39.444 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:39 smithi163 ceph-mon[34354]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:39 smithi163 ceph-mon[34354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:39 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:39 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.445 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:39 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:39 smithi069 ceph-mon[30123]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:39 smithi069 ceph-mon[30123]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-05-09T09:12:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:39 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:39 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:39 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:12:40.277 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:40.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T09:12:40.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T09:12:40.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:40.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:40.278 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:40 smithi103 ceph-mon[34421]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:40.278 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:12:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:12:40.142+0000 7f69729b8700 -1 osd.4 0 waiting for initial osdmap 2024-05-09T09:12:40.278 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:12:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:12:40.149+0000 7f696e7ab700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:40.448 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:40 smithi163 ceph-mon[34354]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-05-09T09:12:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: osdmap e31: 5 total, 4 up, 5 in 2024-05-09T09:12:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: from='osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:40 smithi069 ceph-mon[30123]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:12:41.333 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:41 smithi163 ceph-mon[34354]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:12:41.333 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:41 smithi163 ceph-mon[34354]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T09:12:41.333 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.334 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.334 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-05-09T09:12:41.334 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-05-09T09:12:41.334 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-09T09:12:41.335 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-09T09:12:41.335 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127696 s, 821 MB/s 2024-05-09T09:12:41.335 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-09T09:12:41.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:41 smithi103 ceph-mon[34421]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:12:41.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:41 smithi103 ceph-mon[34421]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T09:12:41.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:41 smithi069 ceph-mon[30123]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:12:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:41 smithi069 ceph-mon[30123]: osdmap e32: 5 total, 4 up, 5 in 2024-05-09T09:12:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:41.974 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_3 2024-05-09T09:12:42.217 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:12:42.294 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:12:42.294 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] boot 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:42.295 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] boot 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 107 MiB used, 358 GiB / 358 GiB avail 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: osd.4 [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] boot 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: osdmap e33: 5 total, 5 up, 5 in 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:42.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:12:42.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:12:43.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:43 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi103 2024-05-09T09:12:43.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:43 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-09T09:12:43.331 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:43 smithi163 ceph-mon[34354]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T09:12:43.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:43 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi103 2024-05-09T09:12:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:43 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-09T09:12:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:43 smithi069 ceph-mon[30123]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T09:12:43.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:43 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi103 2024-05-09T09:12:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:43 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 8544M 2024-05-09T09:12:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:43 smithi103 ceph-mon[34421]: osdmap e34: 5 total, 5 up, 5 in 2024-05-09T09:12:43.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1305758386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103580 msgr2=0x7f3e941039e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:43.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1305758386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103580 0x7f3e941039e0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3e84009990 tx=0x7f3e84009ca0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1305758386 shutdown_connections 2024-05-09T09:12:43.807 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1305758386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94103f20 0x7f3e9410a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1305758386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94102380 0x7f3e941027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1305758386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103580 0x7f3e941039e0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.805+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1305758386 >> 172.21.15.163:0/1305758386 conn(0x7f3e940fd920 msgr2=0x7f3e940ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1305758386 shutdown_connections 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1305758386 wait complete. 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 Processor -- start 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- start start 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 0x7f3e9419a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94103580 0x7f3e9419ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:43.808 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410dfa0 con 0x7f3e94102380 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410de20 con 0x7f3e94103580 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e9b70e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3e9410e120 con 0x7f3e94103f20 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.806+0000 7f3e99cab700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47288/0 (socket says 172.21.15.163:47288) 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 -- 172.21.15.163:0/1804722655 learned_addr learned my addr 172.21.15.163:0/1804722655 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:12:43.809 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e98ca9700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94103580 0x7f3e9419ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e994aa700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 0x7f3e9419a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 -- 172.21.15.163:0/1804722655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94103580 msgr2=0x7f3e9419ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94103580 0x7f3e9419ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 -- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 msgr2=0x7f3e9419a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 0x7f3e9419a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 -- 172.21.15.163:0/1804722655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3e941a2460 con 0x7f3e94103f20 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e994aa700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 0x7f3e9419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e99cab700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00ef60 tx=0x7f3e8c00ef90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:12:43.810 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c0092e0 con 0x7f3e94103f20 2024-05-09T09:12:43.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3e941a26f0 con 0x7f3e94103f20 2024-05-09T09:12:43.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3e8c004bb0 con 0x7f3e94103f20 2024-05-09T09:12:43.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3e8c004280 con 0x7f3e94103f20 2024-05-09T09:12:43.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.807+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3e941a2be0 con 0x7f3e94103f20 2024-05-09T09:12:43.811 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.808+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3e94069010 con 0x7f3e94103f20 2024-05-09T09:12:43.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.809+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3e8c01b030 con 0x7f3e94103f20 2024-05-09T09:12:43.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.810+0000 7f3e8a7fc700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 0x7f3e800a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:12:43.812 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.810+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(34..34 src has 1..34) ==== 4101+0+0 (secure 0 0 0) 0x7f3e8c0cdad0 con 0x7f3e94103f20 2024-05-09T09:12:43.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.810+0000 7f3e994aa700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 0x7f3e800a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:12:43.813 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.810+0000 7f3e994aa700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 0x7f3e800a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005950 tx=0x7f3e900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:12:43.814 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:43.814+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3e8c022020 con 0x7f3e94103f20 2024-05-09T09:12:44.017 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:12:44.017+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) -- 0x7f3e94075c70 con 0x7f3e800a2c20 2024-05-09T09:12:44.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:44 smithi163 ceph-mon[34354]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:44.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:44.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:44 smithi069 ceph-mon[30123]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:44.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:44.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:44.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:44.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:44 smithi103 ceph-mon[34421]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:44.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:12:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:12:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:45 smithi069 ceph-mon[30123]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:45.532 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:45 smithi163 ceph-mon[34354]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:45.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:45 smithi103 ceph-mon[34421]: from='client.24241 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:12:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:46 smithi069 ceph-mon[30123]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:46 smithi163 ceph-mon[34354]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:46.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:46 smithi103 ceph-mon[34421]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:47.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/3920284222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:47 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:47 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]': finished 2024-05-09T09:12:47.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:47 smithi163 ceph-mon[34354]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T09:12:47.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:12:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/3920284222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:47 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:47 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]': finished 2024-05-09T09:12:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:47 smithi069 ceph-mon[30123]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T09:12:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:12:47.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/3920284222' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:47 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]: dispatch 2024-05-09T09:12:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:47 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3e0e1bf3-cc14-4202-a96a-a557f5041ac0"}]': finished 2024-05-09T09:12:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:47 smithi103 ceph-mon[34421]: osdmap e35: 6 total, 5 up, 6 in 2024-05-09T09:12:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:12:48.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:48 smithi069 ceph-mon[30123]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:48.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/3879091321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:48.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:48 smithi163 ceph-mon[34354]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:48.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/3879091321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:48.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:48 smithi103 ceph-mon[34421]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:48.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/3879091321' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:12:50.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:50 smithi069 ceph-mon[30123]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:50.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:50 smithi163 ceph-mon[34354]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:50.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:50 smithi103 ceph-mon[34421]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 133 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:52.355 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:52 smithi163 ceph-mon[34354]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:52.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:52 smithi069 ceph-mon[30123]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:52.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:52 smithi103 ceph-mon[34421]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:54 smithi069 ceph-mon[30123]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:54 smithi163 ceph-mon[34354]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:54.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:54 smithi103 ceph-mon[34421]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:55.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T09:12:55.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:55.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T09:12:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:55.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-05-09T09:12:55.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:12:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:56 smithi069 ceph-mon[30123]: Deploying daemon osd.5 on smithi163 2024-05-09T09:12:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:56 smithi069 ceph-mon[30123]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:56.547 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:56 smithi163 ceph-mon[34354]: Deploying daemon osd.5 on smithi163 2024-05-09T09:12:56.548 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:56 smithi163 ceph-mon[34354]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:56 smithi103 ceph-mon[34421]: Deploying daemon osd.5 on smithi163 2024-05-09T09:12:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:56 smithi103 ceph-mon[34421]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:58.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:12:58 smithi069 ceph-mon[30123]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:58.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:12:58 smithi163 ceph-mon[34354]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:12:58.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:12:58 smithi103 ceph-mon[34421]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:00 smithi163 ceph-mon[34354]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.494 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:00 smithi069 ceph-mon[30123]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:00.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:00 smithi103 ceph-mon[34421]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:00.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:00.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:02.415 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:02 smithi163 ceph-mon[34354]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:02.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:02 smithi069 ceph-mon[30123]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:02 smithi103 ceph-mon[34421]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:03.634 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.636+0000 7f3e8a7fc700 1 -- 172.21.15.163:0/1804722655 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f3e94075c70 con 0x7f3e800a2c20 2024-05-09T09:13:03.635 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 5 on host 'smithi163' 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 msgr2=0x7f3e800a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 0x7f3e800a50e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f3e90005950 tx=0x7f3e900058e0 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 msgr2=0x7f3e941a1dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3e8c00ef60 tx=0x7f3e8c00ef90 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 shutdown_connections 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3e94103f20 0x7f3e941a1dc0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3e94102380 0x7f3e9419a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3e800a2c20 0x7f3e800a50e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 --2- 172.21.15.163:0/1804722655 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3e94103580 0x7f3e9419ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 >> 172.21.15.163:0/1804722655 conn(0x7f3e940fd920 msgr2=0x7f3e94074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 shutdown_connections 2024-05-09T09:13:03.637 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:13:03.638+0000 7f3e9b70e700 1 -- 172.21.15.163:0/1804722655 wait complete. 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:03.948 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:03.949 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:03.949 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:03.949 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:04.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:04.291 DEBUG:teuthology.orchestra.run.smithi163:osd.5> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.5.service 2024-05-09T09:13:04.293 INFO:tasks.cephadm:Deploying osd.6 on smithi069 with /dev/vg_nvme/lv_2... 2024-05-09T09:13:04.293 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T09:13:04.543 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:13:04.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:04 smithi163 ceph-mon[34354]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:04.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:04 smithi163 ceph-mon[34354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:04.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:04 smithi069 ceph-mon[30123]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:04.920 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:04 smithi069 ceph-mon[30123]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:05.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:04 smithi103 ceph-mon[34421]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:05.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:04 smithi103 ceph-mon[34421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-05-09T09:13:05.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T09:13:05.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T09:13:05.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:05.896 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:05.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:05.897 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:05 smithi069 ceph-mon[30123]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:05.971 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:05 smithi163 ceph-mon[34354]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:06.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-05-09T09:13:06.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: osdmap e36: 6 total, 5 up, 6 in 2024-05-09T09:13:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: from='osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:13:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:05 smithi103 ceph-mon[34421]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 134 MiB used, 447 GiB / 447 GiB avail 2024-05-09T09:13:06.305 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:13:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:13:06.209+0000 7fe00042c700 -1 osd.5 0 waiting for initial osdmap 2024-05-09T09:13:06.305 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:13:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:13:06.217+0000 7fdff8217700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:13:06.492 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T09:13:06.492 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T09:13:06.492 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-05-09T09:13:06.492 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-05-09T09:13:06.493 INFO:teuthology.orchestra.run.smithi069.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0126065 s, 832 MB/s 2024-05-09T09:13:06.493 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 2024-05-09T09:13:06.493 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-05-09T09:13:06.909 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:13:06.909 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: from='osd.5 ' entity='osd.5' 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] boot 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T09:13:06.910 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: from='osd.5 ' entity='osd.5' 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] boot 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T09:13:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:13:07.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:13:07.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: osdmap e37: 6 total, 5 up, 6 in 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: from='osd.5 ' entity='osd.5' 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: osd.5 [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] boot 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: osdmap e38: 6 total, 6 up, 6 in 2024-05-09T09:13:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-05-09T09:13:07.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_2 2024-05-09T09:13:07.488 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:13:07.957 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:07.957 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:07.958 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:08.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T09:13:08.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:08.019 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: osdmap e39: 6 total, 6 up, 6 in 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:08 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi163 2024-05-09T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:08 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-09T09:13:08.926 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:08 smithi069 ceph-mon[30123]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T09:13:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:08 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi163 2024-05-09T09:13:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:08 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-09T09:13:09.034 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:08 smithi103 ceph-mon[34421]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T09:13:09.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:08 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi163 2024-05-09T09:13:09.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:08 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 8544M 2024-05-09T09:13:09.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:08 smithi163 ceph-mon[34354]: osdmap e40: 6 total, 6 up, 6 in 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/193705692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8102480 msgr2=0x7ffba81028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/193705692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8102480 0x7ffba81028e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7ffb980099c0 tx=0x7ffb98009cd0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/193705692 shutdown_connections 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/193705692 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 0x7ffba81096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/193705692 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8101280 0x7ffba81016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/193705692 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8102480 0x7ffba81028e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/193705692 >> 172.21.15.69:0/193705692 conn(0x7ffba80fc820 msgr2=0x7ffba80fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/193705692 shutdown_connections 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.146+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/193705692 wait complete. 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 Processor -- start 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 -- start start 2024-05-09T09:13:09.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8102480 0x7ffba8198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 0x7ffba819fb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffba810ce40 con 0x7ffba8101280 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffba810ccc0 con 0x7ffba8102e20 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbaf8c4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffba810cfc0 con 0x7ffba8102480 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43324/0 (socket says 172.21.15.69:43324) 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 -- 172.21.15.69:0/2274328100 learned_addr learned my addr 172.21.15.69:0/2274328100 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbade61700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 0x7ffba819fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbace5f700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8102480 0x7ffba8198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:09.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 -- 172.21.15.69:0/2274328100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8102480 msgr2=0x7ffba8198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8102480 0x7ffba8198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 -- 172.21.15.69:0/2274328100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 msgr2=0x7ffba819fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 0x7ffba819fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.147+0000 7ffbad660700 1 -- 172.21.15.69:0/2274328100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffba81a0200 con 0x7ffba8101280 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffbad660700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffba400d9b0 tx=0x7ffba400dcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4017030 con 0x7ffba8101280 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffba81a04f0 con 0x7ffba8101280 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffba4004500 con 0x7ffba8101280 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffba4003970 con 0x7ffba8101280 2024-05-09T09:13:09.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.148+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffba81a0a30 con 0x7ffba8101280 2024-05-09T09:13:09.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.149+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffba8069010 con 0x7ffba8101280 2024-05-09T09:13:09.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.150+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ffba401a070 con 0x7ffba8101280 2024-05-09T09:13:09.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.151+0000 7ffb9e7fc700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 0x7ffb940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:09.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.151+0000 7ffbace5f700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 0x7ffb940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:09.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.151+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(40..40 src has 1..40) ==== 4562+0+0 (secure 0 0 0) 0x7ffba40cddc0 con 0x7ffba8101280 2024-05-09T09:13:09.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.151+0000 7ffbace5f700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 0x7ffb940a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000c00 tx=0x7ffb98005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:13:09.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.154+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffba400e350 con 0x7ffba8101280 2024-05-09T09:13:09.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:09.360+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7ffba80fd580 con 0x7ffb940a2c20 2024-05-09T09:13:10.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:09 smithi069 ceph-mon[30123]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:10.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:10.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:10.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:10.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:09 smithi069 ceph-mon[30123]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T09:13:10.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:09 smithi163 ceph-mon[34354]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:10.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:10.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:10.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:10.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:09 smithi163 ceph-mon[34354]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T09:13:10.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:09 smithi103 ceph-mon[34421]: pgmap v140: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:09 smithi103 ceph-mon[34421]: osdmap e41: 6 total, 6 up, 6 in 2024-05-09T09:13:11.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:10 smithi069 ceph-mon[30123]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:11.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:10 smithi069 ceph-mon[30123]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T09:13:11.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:10 smithi163 ceph-mon[34354]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:11.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:10 smithi163 ceph-mon[34354]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T09:13:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:10 smithi103 ceph-mon[34421]: from='client.14358 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:10 smithi103 ceph-mon[34421]: osdmap e42: 6 total, 6 up, 6 in 2024-05-09T09:13:11.748 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:11 smithi069 ceph-mon[30123]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:12.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:11 smithi163 ceph-mon[34354]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:12.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:11 smithi103 ceph-mon[34421]: pgmap v143: 1 pgs: 1 remapped+peering; 577 KiB data, 167 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:13.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]: dispatch 2024-05-09T09:13:13.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]': finished 2024-05-09T09:13:13.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:12 smithi069 ceph-mon[30123]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T09:13:13.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:13.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2059111602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:13.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]: dispatch 2024-05-09T09:13:13.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]': finished 2024-05-09T09:13:13.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:12 smithi163 ceph-mon[34354]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T09:13:13.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:13.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2059111602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:13.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]: dispatch 2024-05-09T09:13:13.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2192213755' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ef0f7d90-b664-4519-a34a-27549cfe3406"}]': finished 2024-05-09T09:13:13.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:12 smithi103 ceph-mon[34421]: osdmap e43: 7 total, 6 up, 7 in 2024-05-09T09:13:13.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:13.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2059111602' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:14.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:13 smithi069 ceph-mon[30123]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:14.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:13 smithi163 ceph-mon[34354]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:14.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:13 smithi103 ceph-mon[34421]: pgmap v145: 1 pgs: 1 remapped+peering; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:16.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:16 smithi103 ceph-mon[34421]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T09:13:16.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:16 smithi069 ceph-mon[30123]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T09:13:16.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:16 smithi163 ceph-mon[34354]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-05-09T09:13:18.166 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:18 smithi069 ceph-mon[30123]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-09T09:13:18.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:18 smithi103 ceph-mon[34421]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-09T09:13:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:18 smithi163 ceph-mon[34354]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 75 KiB/s, 0 objects/s recovering 2024-05-09T09:13:20.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:20 smithi069 ceph-mon[30123]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-09T09:13:20.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T09:13:20.256 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:20.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:20 smithi163 ceph-mon[34354]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-09T09:13:20.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T09:13:20.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:20.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:20 smithi103 ceph-mon[34421]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 66 KiB/s, 0 objects/s recovering 2024-05-09T09:13:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-05-09T09:13:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:21.421 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:21 smithi069 ceph-mon[30123]: Deploying daemon osd.6 on smithi069 2024-05-09T09:13:21.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:21 smithi163 ceph-mon[34354]: Deploying daemon osd.6 on smithi069 2024-05-09T09:13:21.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:21 smithi103 ceph-mon[34421]: Deploying daemon osd.6 on smithi069 2024-05-09T09:13:22.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:22 smithi069 ceph-mon[30123]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:13:22.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:22 smithi163 ceph-mon[34354]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:13:22.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:22 smithi103 ceph-mon[34421]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:13:24.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:24 smithi069 ceph-mon[30123]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-09T09:13:24.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:24 smithi163 ceph-mon[34354]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-09T09:13:24.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:24 smithi103 ceph-mon[34421]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 52 KiB/s, 0 objects/s recovering 2024-05-09T09:13:26.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:26 smithi069 ceph-mon[30123]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T09:13:26.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.251 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:26.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:26 smithi163 ceph-mon[34354]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T09:13:26.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:26.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:26 smithi103 ceph-mon[34421]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail; 48 KiB/s, 0 objects/s recovering 2024-05-09T09:13:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:28.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:28 smithi069 ceph-mon[30123]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:28.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:28 smithi163 ceph-mon[34354]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:28.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:28 smithi103 ceph-mon[34421]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:28.868 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 6 on host 'smithi069' 2024-05-09T09:13:28.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.869+0000 7ffb9e7fc700 1 -- 172.21.15.69:0/2274328100 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7ffba80fd580 con 0x7ffb940a2c20 2024-05-09T09:13:28.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 msgr2=0x7ffb940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:28.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 0x7ffb940a50e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ffb98000c00 tx=0x7ffb98005fb0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 msgr2=0x7ffba81985a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ffba400d9b0 tx=0x7ffba400dcc0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 shutdown_connections 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffb940a2c20 0x7ffb940a50e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffba8102e20 0x7ffba819fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffba8101280 0x7ffba81985a0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 --2- 172.21.15.69:0/2274328100 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffba8102480 0x7ffba8198ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:28.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 >> 172.21.15.69:0/2274328100 conn(0x7ffba80fc820 msgr2=0x7ffba80fce70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:13:28.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 shutdown_connections 2024-05-09T09:13:28.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:13:28.871+0000 7ffbaf8c4700 1 -- 172.21.15.69:0/2274328100 wait complete. 2024-05-09T09:13:29.629 DEBUG:teuthology.orchestra.run.smithi069:osd.6> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.6.service 2024-05-09T09:13:29.632 INFO:tasks.cephadm:Deploying osd.7 on smithi103 with /dev/vg_nvme/lv_2... 2024-05-09T09:13:29.632 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T09:13:29.878 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:13:30.000 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.000 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.001 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:30.001 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.001 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.001 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:29 smithi103 ceph-mon[34421]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T09:13:30.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.116 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:30.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.117 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:29 smithi069 ceph-mon[30123]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T09:13:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:30.307 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:29 smithi163 ceph-mon[34354]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-05-09T09:13:31.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:31 smithi103 ceph-mon[34421]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T09:13:31.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:31 smithi103 ceph-mon[34421]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T09:13:31.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:31 smithi103 ceph-mon[34421]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:13:31.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:31.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:31 smithi069 ceph-mon[30123]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T09:13:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:31 smithi069 ceph-mon[30123]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T09:13:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:31 smithi069 ceph-mon[30123]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:13:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:31 smithi163 ceph-mon[34354]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-05-09T09:13:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:31 smithi163 ceph-mon[34354]: osdmap e44: 7 total, 6 up, 7 in 2024-05-09T09:13:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:31 smithi163 ceph-mon[34354]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:13:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:13:31 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:13:31.594+0000 7fed7f748700 -1 osd.6 0 waiting for initial osdmap 2024-05-09T09:13:32.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:13:31 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:13:31.603+0000 7fed77533700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:32 smithi103 ceph-mon[34421]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0141049 s, 743 MB/s 2024-05-09T09:13:32.281 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-09T09:13:32.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:32.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:13:32.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T09:13:32.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:32 smithi069 ceph-mon[30123]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' 2024-05-09T09:13:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:32.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:13:32.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: osdmap e45: 7 total, 6 up, 7 in 2024-05-09T09:13:32.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:32.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:32 smithi163 ceph-mon[34354]: from='osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338]' entity='osd.6' 2024-05-09T09:13:32.993 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_2 2024-05-09T09:13:33.244 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:13:33.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] boot 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:33.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] boot 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: osd.6 [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] boot 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: osdmap e46: 7 total, 7 up, 7 in 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:33.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:34.289 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:34 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi069 2024-05-09T09:13:34.290 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:34 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 5696M 2024-05-09T09:13:34.290 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:34 smithi103 ceph-mon[34421]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:34.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:34 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi069 2024-05-09T09:13:34.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:34 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 5696M 2024-05-09T09:13:34.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:34 smithi069 ceph-mon[30123]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:34.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:34 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi069 2024-05-09T09:13:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:34 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 5696M 2024-05-09T09:13:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:34 smithi163 ceph-mon[34354]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 165 MiB used, 536 GiB / 536 GiB avail 2024-05-09T09:13:34.846 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/1030945368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 msgr2=0x7fa2ac1027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/1030945368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac1027a0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa2a80099c0 tx=0x7fa2a8009cd0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/1030945368 shutdown_connections 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/1030945368 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 0x7fa2ac10a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/1030945368 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac1027a0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/1030945368 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 0x7fa2ac1039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/1030945368 >> 172.21.15.103:0/1030945368 conn(0x7fa2ac0fd920 msgr2=0x7fa2ac0ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.847+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/1030945368 shutdown_connections 2024-05-09T09:13:34.847 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/1030945368 wait complete. 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 Processor -- start 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 -- start start 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 0x7fa2ac198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 0x7fa2ac19fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac10dfe0 con 0x7fa2ac103f20 2024-05-09T09:13:34.849 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac10de60 con 0x7fa2ac102380 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b3d2c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2ac10e160 con 0x7fa2ac103580 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b1ac8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b1ac8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:41540/0 (socket says 172.21.15.103:41540) 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.848+0000 7fa2b1ac8700 1 -- 172.21.15.103:0/3257813309 learned_addr learned my addr 172.21.15.103:0/3257813309 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b22c9700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 0x7fa2ac19fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b12c7700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 0x7fa2ac198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 -- 172.21.15.103:0/3257813309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 msgr2=0x7fa2ac198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 0x7fa2ac198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 -- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 msgr2=0x7fa2ac19fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 0x7fa2ac19fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 -- 172.21.15.103:0/3257813309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa2ac1a0280 con 0x7fa2ac102380 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b1ac8700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c010 tx=0x7fa2a8009510 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:13:34.850 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8027070 con 0x7fa2ac102380 2024-05-09T09:13:34.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa2a8005940 con 0x7fa2ac102380 2024-05-09T09:13:34.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa2ac1a0510 con 0x7fa2ac102380 2024-05-09T09:13:34.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa2a8022d90 con 0x7fa2ac102380 2024-05-09T09:13:34.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.849+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2ac1a0a70 con 0x7fa2ac102380 2024-05-09T09:13:34.851 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.850+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa2ac069010 con 0x7fa2ac102380 2024-05-09T09:13:34.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.851+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa2a802a070 con 0x7fa2ac102380 2024-05-09T09:13:34.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.852+0000 7fa2a2ffd700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 0x7fa2980a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:13:34.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.852+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(47..47 src has 1..47) ==== 4965+0+0 (secure 0 0 0) 0x7fa2a80cf1a0 con 0x7fa2ac102380 2024-05-09T09:13:34.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.852+0000 7fa2b12c7700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 0x7fa2980a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:13:34.852 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.852+0000 7fa2b12c7700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 0x7fa2980a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa29c005950 tx=0x7fa29c0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:13:34.855 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:34.856+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa2a805e740 con 0x7fa2ac102380 2024-05-09T09:13:35.057 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:35.058+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7fa2ac062760 con 0x7fa2980a2c20 2024-05-09T09:13:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:35 smithi069 ceph-mon[30123]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T09:13:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:35.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:35.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:35 smithi163 ceph-mon[34354]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T09:13:35.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:35.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:35 smithi103 ceph-mon[34421]: osdmap e47: 7 total, 7 up, 7 in 2024-05-09T09:13:35.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:13:35.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:13:35.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:36 smithi069 ceph-mon[30123]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:36 smithi069 ceph-mon[30123]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:36.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:36 smithi163 ceph-mon[34354]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:36.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:36 smithi163 ceph-mon[34354]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:36.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:36 smithi103 ceph-mon[34421]: from='client.24266 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:13:36.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:36 smithi103 ceph-mon[34421]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/3877239045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]': finished 2024-05-09T09:13:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T09:13:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:38 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:38.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/3877239045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]': finished 2024-05-09T09:13:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T09:13:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:38 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/3877239045' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]: dispatch 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7236537c-3738-4ad0-bc46-1a96a7bd8b70"}]': finished 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: osdmap e48: 8 total, 7 up, 8 in 2024-05-09T09:13:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:38 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:39.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/3620722310' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:39.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/3620722310' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:39.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/3620722310' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:13:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:40 smithi069 ceph-mon[30123]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:40.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:40 smithi163 ceph-mon[34354]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:40.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:40 smithi103 ceph-mon[34421]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:42 smithi103 ceph-mon[34421]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:42.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:42 smithi069 ceph-mon[30123]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:42.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:42 smithi163 ceph-mon[34354]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:44.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:44 smithi069 ceph-mon[30123]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:44.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:44 smithi163 ceph-mon[34354]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:44 smithi103 ceph-mon[34421]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:45.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T09:13:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:45 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:45.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T09:13:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:45 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:45.581 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-05-09T09:13:45.581 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:45 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:46 smithi069 ceph-mon[30123]: Deploying daemon osd.7 on smithi103 2024-05-09T09:13:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:46 smithi069 ceph-mon[30123]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:46 smithi163 ceph-mon[34354]: Deploying daemon osd.7 on smithi103 2024-05-09T09:13:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:46 smithi163 ceph-mon[34354]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:46.581 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:46 smithi103 ceph-mon[34421]: Deploying daemon osd.7 on smithi103 2024-05-09T09:13:46.582 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:46 smithi103 ceph-mon[34421]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:47 smithi069 ceph-mon[30123]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:47.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:47 smithi163 ceph-mon[34354]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:47.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:47 smithi103 ceph-mon[34421]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:50.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:50 smithi103 ceph-mon[34421]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:50.490 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:50 smithi069 ceph-mon[30123]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:50.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:50 smithi163 ceph-mon[34354]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:51.624 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:51.624 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:51.624 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:51.625 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:51 smithi103 ceph-mon[34421]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:51.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:51.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:51.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:51.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:51 smithi069 ceph-mon[30123]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:52.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:51 smithi163 ceph-mon[34354]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:54.474 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:54 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:54 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.523 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.523+0000 7fa2a2ffd700 1 -- 172.21.15.103:0/3257813309 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fa2ac062760 con 0x7fa2980a2c20 2024-05-09T09:13:54.523 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 7 on host 'smithi103' 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 msgr2=0x7fa2980a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 0x7fa2980a50e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fa29c005950 tx=0x7fa29c0058e0 comp rx=0 tx=0).stop 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 msgr2=0x7fa2ac198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7fa2a800c010 tx=0x7fa2a8009510 comp rx=0 tx=0).stop 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 shutdown_connections 2024-05-09T09:13:54.525 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa2ac103f20 0x7fa2ac19fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2980a2c20 0x7fa2980a50e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2ac102380 0x7fa2ac198620 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 --2- 172.21.15.103:0/3257813309 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2ac103580 0x7fa2ac198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 >> 172.21.15.103:0/3257813309 conn(0x7fa2ac0fd920 msgr2=0x7fa2ac106970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 shutdown_connections 2024-05-09T09:13:54.526 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:13:54.526+0000 7fa2b3d2c700 1 -- 172.21.15.103:0/3257813309 wait complete. 2024-05-09T09:13:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:54 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.229 DEBUG:teuthology.orchestra.run.smithi103:osd.7> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.7.service 2024-05-09T09:13:55.231 INFO:tasks.cephadm:Deploying osd.8 on smithi163 with /dev/vg_nvme/lv_2... 2024-05-09T09:13:55.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-05-09T09:13:55.476 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:13:55.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:55 smithi069 ceph-mon[30123]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:55.802 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:55.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.803 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:55 smithi163 ceph-mon[34354]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:55.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:13:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-05-09T09:13:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:55 smithi103 ceph-mon[34421]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:56.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:56 smithi103 ceph-mon[34421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T09:13:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:56 smithi103 ceph-mon[34421]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T09:13:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:56 smithi103 ceph-mon[34421]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:56 smithi103 ceph-mon[34421]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:56.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:56 smithi163 ceph-mon[34354]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T09:13:56.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:56 smithi163 ceph-mon[34354]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T09:13:56.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:56 smithi163 ceph-mon[34354]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:56.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:56.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:56 smithi163 ceph-mon[34354]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:57.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:56 smithi069 ceph-mon[30123]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-05-09T09:13:57.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:56 smithi069 ceph-mon[30123]: osdmap e49: 8 total, 7 up, 8 in 2024-05-09T09:13:57.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:56 smithi069 ceph-mon[30123]: from='osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:57.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:57.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:56 smithi069 ceph-mon[30123]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:13:57.040 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:13:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:13:56.937+0000 7f61f5b82700 -1 osd.7 0 waiting for initial osdmap 2024-05-09T09:13:57.040 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:13:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:13:56.948+0000 7f61ed96d700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124102 s, 845 MB/s 2024-05-09T09:13:57.504 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-09T09:13:57.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:13:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T09:13:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: from='osd.7 ' entity='osd.7' 2024-05-09T09:13:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:57 smithi163 ceph-mon[34354]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:57.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:13:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T09:13:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: from='osd.7 ' entity='osd.7' 2024-05-09T09:13:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:57 smithi103 ceph-mon[34421]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:58.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:13:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: osdmap e50: 8 total, 7 up, 8 in 2024-05-09T09:13:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: from='osd.7 ' entity='osd.7' 2024-05-09T09:13:58.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:57 smithi069 ceph-mon[30123]: pgmap v174: 1 pgs: 1 unknown; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-05-09T09:13:58.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_2 2024-05-09T09:13:58.443 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:13:58.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] boot 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:58.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:58.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.785 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:13:58.785 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:13:58.785 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.785 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] boot 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:58.786 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:59.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: osd.7 [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] boot 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: osdmap e51: 8 total, 8 up, 8 in 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:13:59.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:13:59.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:13:59.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:59 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi103 2024-05-09T09:13:59.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:59 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-09T09:13:59.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:59 smithi163 ceph-mon[34354]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T09:13:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:13:59 smithi163 ceph-mon[34354]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:13:59.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:59 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi103 2024-05-09T09:13:59.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:59 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-09T09:13:59.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:59 smithi103 ceph-mon[34421]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T09:13:59.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:13:59 smithi103 ceph-mon[34421]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:00.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:59 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi103 2024-05-09T09:14:00.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:59 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 5696M 2024-05-09T09:14:00.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:59 smithi069 ceph-mon[30123]: osdmap e52: 8 total, 8 up, 8 in 2024-05-09T09:14:00.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:13:59 smithi069 ceph-mon[30123]: pgmap v177: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:00.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 -- 172.21.15.163:0/3798715286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8102380 msgr2=0x7f1fd81027a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:00.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 --2- 172.21.15.163:0/3798715286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8102380 0x7f1fd81027a0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc009990 tx=0x7f1fcc009ca0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.159 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 -- 172.21.15.163:0/3798715286 shutdown_connections 2024-05-09T09:14:00.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 --2- 172.21.15.163:0/3798715286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8103f20 0x7f1fd810a7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 --2- 172.21.15.163:0/3798715286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8102380 0x7f1fd81027a0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.160 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 --2- 172.21.15.163:0/3798715286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd81039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 -- 172.21.15.163:0/3798715286 >> 172.21.15.163:0/3798715286 conn(0x7f1fd80fd920 msgr2=0x7f1fd80ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 -- 172.21.15.163:0/3798715286 shutdown_connections 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.154+0000 7f1fdd896700 1 -- 172.21.15.163:0/3798715286 wait complete. 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 Processor -- start 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 -- start start 2024-05-09T09:14:00.161 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8102380 0x7f1fd80760c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd8074710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810df70 con 0x7f1fd8103580 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810ddf0 con 0x7f1fd8102380 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fdd896700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1fd810e0f0 con 0x7f1fd8103f20 2024-05-09T09:14:00.162 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.155+0000 7f1fd77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:52394/0 (socket says 172.21.15.163:52394) 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 -- 172.21.15.163:0/1203418097 learned_addr learned my addr 172.21.15.163:0/1203418097 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd6ffd700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8102380 0x7f1fd80760c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd67fc700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd8074710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 -- 172.21.15.163:0/1203418097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8102380 msgr2=0x7f1fd80760c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8102380 0x7f1fd80760c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 -- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 msgr2=0x7f1fd8074710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd8074710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 -- 172.21.15.163:0/1203418097 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1fd8075500 con 0x7f1fd8103f20 2024-05-09T09:14:00.163 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd67fc700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd8074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fd77fe700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ef30 tx=0x7f1fd000c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd00092e0 con 0x7f1fd8103f20 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1fd81a26e0 con 0x7f1fd8103f20 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1fd0004bb0 con 0x7f1fd8103f20 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1fd0004280 con 0x7f1fd8103f20 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.156+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1fd81a2c20 con 0x7f1fd8103f20 2024-05-09T09:14:00.164 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.157+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1fd8069010 con 0x7f1fd8103f20 2024-05-09T09:14:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.158+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1fd001b030 con 0x7f1fd8103f20 2024-05-09T09:14:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.159+0000 7f1fdc894700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 0x7f1fc00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.159+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(53..53 src has 1..53) ==== 5426+0+0 (secure 0 0 0) 0x7f1fd00ce9f0 con 0x7f1fd8103f20 2024-05-09T09:14:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.159+0000 7f1fd6ffd700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 0x7f1fc00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:00.165 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.160+0000 7f1fd6ffd700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 0x7f1fc00a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00b820 tx=0x7f1fcc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:00.166 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.163+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1fd0022020 con 0x7f1fd8103f20 2024-05-09T09:14:00.368 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:00.366+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) -- 0x7f1fd8062760 con 0x7f1fc00a2c20 2024-05-09T09:14:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:00 smithi163 ceph-mon[34354]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T09:14:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:00.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:00 smithi103 ceph-mon[34421]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T09:14:00.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:00.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:00.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:00.931 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:00 smithi069 ceph-mon[30123]: osdmap e53: 8 total, 8 up, 8 in 2024-05-09T09:14:00.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:00.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:00.932 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:01.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:01 smithi069 ceph-mon[30123]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:01.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:01 smithi069 ceph-mon[30123]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T09:14:01.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:01 smithi069 ceph-mon[30123]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:01.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:01 smithi163 ceph-mon[34354]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:01.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:01 smithi163 ceph-mon[34354]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T09:14:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:01 smithi163 ceph-mon[34354]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:01 smithi103 ceph-mon[34421]: from='client.24313 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:01 smithi103 ceph-mon[34421]: osdmap e54: 8 total, 8 up, 8 in 2024-05-09T09:14:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:01 smithi103 ceph-mon[34421]: pgmap v180: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:02.547 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:02 smithi163 ceph-mon[34354]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T09:14:02.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:02 smithi103 ceph-mon[34421]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T09:14:03.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:02 smithi069 ceph-mon[30123]: osdmap e55: 8 total, 8 up, 8 in 2024-05-09T09:14:03.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/4096080843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]': finished 2024-05-09T09:14:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T09:14:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:03 smithi163 ceph-mon[34354]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:03.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/4096080843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]': finished 2024-05-09T09:14:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T09:14:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:03 smithi103 ceph-mon[34421]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:04.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/4096080843' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:04.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]: dispatch 2024-05-09T09:14:04.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e7cfa791-797c-4dab-84a1-d2ff938e96c4"}]': finished 2024-05-09T09:14:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: osdmap e56: 9 total, 8 up, 9 in 2024-05-09T09:14:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:04.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:03 smithi069 ceph-mon[30123]: pgmap v183: 1 pgs: 1 unknown; 577 KiB data, 227 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:04.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:04 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/8310255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:04.866 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:04 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/8310255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:05.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:04 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/8310255' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:05.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:05 smithi103 ceph-mon[34421]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:05 smithi069 ceph-mon[30123]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:06.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:05 smithi163 ceph-mon[34354]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:08.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:08 smithi163 ceph-mon[34354]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:08 smithi069 ceph-mon[30123]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:08.570 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:08 smithi103 ceph-mon[34421]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:10.451 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:10 smithi163 ceph-mon[34354]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:10.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:10 smithi069 ceph-mon[30123]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:10 smithi103 ceph-mon[34421]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:11.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:11 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T09:14:11.462 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:11 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:11.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:11 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T09:14:11.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:11 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:11.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:11 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-05-09T09:14:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:11 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:12.433 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:12 smithi163 ceph-mon[34354]: Deploying daemon osd.8 on smithi163 2024-05-09T09:14:12.434 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:12 smithi163 ceph-mon[34354]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:12 smithi069 ceph-mon[30123]: Deploying daemon osd.8 on smithi163 2024-05-09T09:14:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:12 smithi069 ceph-mon[30123]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:12.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:12 smithi103 ceph-mon[34421]: Deploying daemon osd.8 on smithi163 2024-05-09T09:14:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:12 smithi103 ceph-mon[34421]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:14.503 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:14 smithi163 ceph-mon[34354]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:14.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:14 smithi069 ceph-mon[30123]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:14.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:14 smithi103 ceph-mon[34421]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:16.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:16 smithi163 ceph-mon[34354]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:16.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:16 smithi069 ceph-mon[30123]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:16.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:16 smithi103 ceph-mon[34421]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:17.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:17 smithi163 ceph-mon[34354]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:17 smithi103 ceph-mon[34421]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:17.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:17.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:17 smithi069 ceph-mon[30123]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:19.880 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.881+0000 7f1fdc894700 1 -- 172.21.15.163:0/1203418097 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7f1fd8062760 con 0x7f1fc00a2c20 2024-05-09T09:14:19.881 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 8 on host 'smithi163' 2024-05-09T09:14:19.883 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 msgr2=0x7f1fc00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 0x7f1fc00a50e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f1fcc00b820 tx=0x7f1fcc005fd0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 msgr2=0x7f1fd8074c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f1fd000ef30 tx=0x7f1fd000c3a0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 shutdown_connections 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1fd8103f20 0x7f1fd8074c50 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1fc00a2c20 0x7f1fc00a50e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1fd8102380 0x7f1fd80760c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 --2- 172.21.15.163:0/1203418097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1fd8103580 0x7f1fd8074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 >> 172.21.15.163:0/1203418097 conn(0x7f1fd80fd920 msgr2=0x7f1fd8106900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 shutdown_connections 2024-05-09T09:14:19.884 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:14:19.884+0000 7f1fdd896700 1 -- 172.21.15.163:0/1203418097 wait complete. 2024-05-09T09:14:20.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:20 smithi069 ceph-mon[30123]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:20.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.462 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:20 smithi163 ceph-mon[34354]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.592 DEBUG:teuthology.orchestra.run.smithi163:osd.8> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.8.service 2024-05-09T09:14:20.594 INFO:tasks.cephadm:Deploying osd.9 on smithi069 with /dev/vg_nvme/lv_1... 2024-05-09T09:14:20.594 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T09:14:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:20 smithi103 ceph-mon[34421]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:20.838 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:14:21.416 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:21 smithi163 ceph-mon[34354]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:21 smithi163 ceph-mon[34354]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:21.417 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:21 smithi069 ceph-mon[30123]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:21 smithi069 ceph-mon[30123]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:21 smithi103 ceph-mon[34421]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:21 smithi103 ceph-mon[34421]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-05-09T09:14:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:22.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:22 smithi069 ceph-mon[30123]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:14:22.446 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.447 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:22 smithi163 ceph-mon[34354]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:14:22.447 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:14:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:14:22.148+0000 7f76ecd1f700 -1 osd.8 0 waiting for initial osdmap 2024-05-09T09:14:22.447 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:14:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:14:22.157+0000 7f76e8b12700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:14:22.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: from='osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: osdmap e57: 9 total, 8 up, 9 in 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:14:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:22 smithi103 ceph-mon[34421]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:14:22.869 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T09:14:22.870 INFO:teuthology.orchestra.run.smithi069.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T09:14:22.870 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10+0 records in 2024-05-09T09:14:22.870 INFO:teuthology.orchestra.run.smithi069.stderr:10+0 records out 2024-05-09T09:14:22.870 INFO:teuthology.orchestra.run.smithi069.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0123987 s, 846 MB/s 2024-05-09T09:14:22.870 INFO:teuthology.orchestra.run.smithi069.stderr:--> Zapping successful for: 2024-05-09T09:14:23.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:23 smithi163 ceph-mon[34354]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T09:14:23.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.424 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:23 smithi069 ceph-mon[30123]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T09:14:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:23 smithi103 ceph-mon[34421]: osdmap e58: 9 total, 8 up, 9 in 2024-05-09T09:14:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:23.627 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi069:vg_nvme/lv_1 2024-05-09T09:14:23.866 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] boot 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T09:14:24.231 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:24.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] boot 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T09:14:24.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:24.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 228 MiB used, 715 GiB / 715 GiB avail 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: osd.8 [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] boot 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: osdmap e59: 9 total, 9 up, 9 in 2024-05-09T09:14:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-05-09T09:14:25.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:25.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/895437528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 msgr2=0x7fcdb81096c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/895437528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb81096c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcdb000b0d0 tx=0x7fcdb000b3e0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/895437528 shutdown_connections 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/895437528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb81096c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/895437528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8101270 0x7fcdb8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/895437528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8102470 0x7fcdb81028d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/895437528 >> 172.21.15.69:0/895437528 conn(0x7fcdb80fc810 msgr2=0x7fcdb80fec50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/895437528 shutdown_connections 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/895437528 wait complete. 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 Processor -- start 2024-05-09T09:14:25.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 -- start start 2024-05-09T09:14:25.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.559+0000 7fcdbfbb3700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8101270 0x7fcdb8105200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8102470 0x7fcdb8103850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810ce30 con 0x7fcdb8102e10 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810ccb0 con 0x7fcdb8101270 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdb810cfb0 con 0x7fcdb8102470 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48166/0 (socket says 172.21.15.69:48166) 2024-05-09T09:14:25.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 -- 172.21.15.69:0/2205669505 learned_addr learned my addr 172.21.15.69:0/2205669505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbd94f700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8101270 0x7fcdb8105200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbd14e700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8102470 0x7fcdb8103850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 -- 172.21.15.69:0/2205669505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8102470 msgr2=0x7fcdb8103850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8102470 0x7fcdb8103850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 -- 172.21.15.69:0/2205669505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8101270 msgr2=0x7fcdb8105200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8101270 0x7fcdb8105200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 -- 172.21.15.69:0/2205669505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdb8104640 con 0x7fcdb8102e10 2024-05-09T09:14:25.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbe150700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0006070 tx=0x7fcdb0004e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0021070 con 0x7fcdb8102e10 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcdb0007c80 con 0x7fcdb8102e10 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdb81a2670 con 0x7fcdb8102e10 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcdb0005480 con 0x7fcdb8102e10 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.560+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdb81a2ad0 con 0x7fcdb8102e10 2024-05-09T09:14:25.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.561+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcdb8069010 con 0x7fcdb8102e10 2024-05-09T09:14:25.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.563+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcdb0024070 con 0x7fcdb8102e10 2024-05-09T09:14:25.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.564+0000 7fcdaeffd700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 0x7fcda40a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:25.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.564+0000 7fcdbd94f700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 0x7fcda40a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:25.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.564+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(61..61 src has 1..61) ==== 5858+0+0 (secure 0 0 0) 0x7fcdb00cf880 con 0x7fcdb8102e10 2024-05-09T09:14:25.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.564+0000 7fcdbd94f700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 0x7fcda40a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005fd0 tx=0x7fcdb4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:25.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.567+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcdb009d200 con 0x7fcdb8102e10 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: osdmap e60: 9 total, 9 up, 9 in 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:25.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:25.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:25.771+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7fcdb80fe8e0 con 0x7fcda40a2ca0 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi163 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:26 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi163 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:26 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi163 2024-05-09T09:14:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 5696M 2024-05-09T09:14:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: pgmap v198: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:26.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: osdmap e61: 9 total, 9 up, 9 in 2024-05-09T09:14:26.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:26.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:26.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:26 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:27 smithi069 ceph-mon[30123]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:27 smithi069 ceph-mon[30123]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T09:14:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:27 smithi163 ceph-mon[34354]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:27 smithi163 ceph-mon[34354]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T09:14:27.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:27 smithi103 ceph-mon[34421]: from='client.14415 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi069:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:27 smithi103 ceph-mon[34421]: osdmap e62: 9 total, 9 up, 9 in 2024-05-09T09:14:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:28 smithi069 ceph-mon[30123]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:28 smithi069 ceph-mon[30123]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T09:14:28.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:28 smithi163 ceph-mon[34354]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:28.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:28 smithi163 ceph-mon[34354]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T09:14:28.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:28 smithi103 ceph-mon[34421]: pgmap v201: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:28 smithi103 ceph-mon[34421]: osdmap e63: 9 total, 9 up, 9 in 2024-05-09T09:14:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:29 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]: dispatch 2024-05-09T09:14:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:29 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]': finished 2024-05-09T09:14:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:29 smithi069 ceph-mon[30123]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T09:14:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:29.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:29 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]: dispatch 2024-05-09T09:14:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:29 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]': finished 2024-05-09T09:14:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:29 smithi163 ceph-mon[34354]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T09:14:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:29 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]: dispatch 2024-05-09T09:14:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:29 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2760783930' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "55ae1e0d-2d12-4aef-9449-e498e1cc719f"}]': finished 2024-05-09T09:14:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:29 smithi103 ceph-mon[34421]: osdmap e64: 10 total, 9 up, 10 in 2024-05-09T09:14:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:30 smithi069 ceph-mon[30123]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:14:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:30 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3353450702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:30.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:30 smithi163 ceph-mon[34354]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:14:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:30 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3353450702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:30.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:30 smithi103 ceph-mon[34421]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:14:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:30 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3353450702' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:32 smithi069 ceph-mon[30123]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-09T09:14:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:32 smithi163 ceph-mon[34354]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-09T09:14:32.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:32 smithi103 ceph-mon[34421]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 97 KiB/s, 0 objects/s recovering 2024-05-09T09:14:34.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:34 smithi069 ceph-mon[30123]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-09T09:14:34.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:34 smithi163 ceph-mon[34354]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-09T09:14:34.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:34 smithi103 ceph-mon[34421]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 83 KiB/s, 0 objects/s recovering 2024-05-09T09:14:36.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:36 smithi069 ceph-mon[30123]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T09:14:36.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:36 smithi163 ceph-mon[34354]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T09:14:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:36 smithi103 ceph-mon[34421]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-05-09T09:14:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:37 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T09:14:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:37 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:37.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:37 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T09:14:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:37 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:37.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:37 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-05-09T09:14:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:37 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:38.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:38 smithi069 ceph-mon[30123]: Deploying daemon osd.9 on smithi069 2024-05-09T09:14:38.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:38 smithi069 ceph-mon[30123]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:14:38.456 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:38 smithi163 ceph-mon[34354]: Deploying daemon osd.9 on smithi069 2024-05-09T09:14:38.456 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:38 smithi163 ceph-mon[34354]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:14:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:38 smithi103 ceph-mon[34421]: Deploying daemon osd.9 on smithi069 2024-05-09T09:14:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:38 smithi103 ceph-mon[34421]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-05-09T09:14:40.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:40 smithi069 ceph-mon[30123]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:40 smithi163 ceph-mon[34354]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:40.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:40 smithi103 ceph-mon[34421]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:42.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:42 smithi069 ceph-mon[30123]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:42.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:42 smithi163 ceph-mon[34354]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:42 smithi103 ceph-mon[34421]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:43.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:43 smithi103 ceph-mon[34421]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:43 smithi069 ceph-mon[30123]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:43.804 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:43 smithi163 ceph-mon[34354]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:46.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:46.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:46.402 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:46.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:46 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.477+0000 7fcdaeffd700 1 -- 172.21.15.69:0/2205669505 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+37 (secure 0 0 0) 0x7fcdb80fe8e0 con 0x7fcda40a2ca0 2024-05-09T09:14:46.477 INFO:teuthology.orchestra.run.smithi069.stdout:Created osd(s) 9 on host 'smithi069' 2024-05-09T09:14:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 msgr2=0x7fcda40a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 0x7fcda40a5160 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fcdb4005fd0 tx=0x7fcdb4005e50 comp rx=0 tx=0).stop 2024-05-09T09:14:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 msgr2=0x7fcdb8103d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fcdb0006070 tx=0x7fcdb0004e70 comp rx=0 tx=0).stop 2024-05-09T09:14:46.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 shutdown_connections 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdb8102e10 0x7fcdb8103d90 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcda40a2ca0 0x7fcda40a5160 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdb8101270 0x7fcdb8105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 --2- 172.21.15.69:0/2205669505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdb8102470 0x7fcdb8103850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.479+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 >> 172.21.15.69:0/2205669505 conn(0x7fcdb80fc810 msgr2=0x7fcdb80fe1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.480+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 shutdown_connections 2024-05-09T09:14:46.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:14:46.480+0000 7fcdbfbb3700 1 -- 172.21.15.69:0/2205669505 wait complete. 2024-05-09T09:14:46.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:46 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:46 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.132 DEBUG:teuthology.orchestra.run.smithi069:osd.9> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.9.service 2024-05-09T09:14:47.134 INFO:tasks.cephadm:Deploying osd.10 on smithi103 with /dev/vg_nvme/lv_1... 2024-05-09T09:14:47.134 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T09:14:47.382 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:14:47.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:47 smithi069 ceph-mon[30123]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T09:14:47.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.427 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:47 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:47.504 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:47 smithi103 ceph-mon[34421]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T09:14:47.504 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.504 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.504 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:47 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:47 smithi163 ceph-mon[34354]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-05-09T09:14:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:47 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:14:48.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:48 smithi069 ceph-mon[30123]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:48.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:48 smithi069 ceph-mon[30123]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T09:14:48.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:48 smithi069 ceph-mon[30123]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T09:14:48.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:48 smithi069 ceph-mon[30123]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:14:48.418 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:48 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:48.419 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:14:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:14:48.146+0000 7fded5c10700 -1 osd.9 0 waiting for initial osdmap 2024-05-09T09:14:48.419 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:14:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:14:48.157+0000 7fded1a03700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:14:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:48 smithi163 ceph-mon[34354]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:48 smithi163 ceph-mon[34354]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T09:14:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:48 smithi163 ceph-mon[34354]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T09:14:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:48 smithi163 ceph-mon[34354]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:14:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:48 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:48.599 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:48 smithi103 ceph-mon[34421]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:48.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:48 smithi103 ceph-mon[34421]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-05-09T09:14:48.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:48 smithi103 ceph-mon[34421]: osdmap e65: 10 total, 9 up, 10 in 2024-05-09T09:14:48.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:48 smithi103 ceph-mon[34421]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]: dispatch 2024-05-09T09:14:48.600 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:48 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.417 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:49 smithi103 ceph-mon[34421]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:14:49.417 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:49 smithi103 ceph-mon[34421]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T09:14:49.417 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.417 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:49 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.473 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:49 smithi069 ceph-mon[30123]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:14:49.473 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:49 smithi069 ceph-mon[30123]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T09:14:49.473 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.473 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:49 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:49 smithi163 ceph-mon[34354]: from='osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi069", "root=default"]}]': finished 2024-05-09T09:14:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:49 smithi163 ceph-mon[34354]: osdmap e66: 10 total, 9 up, 10 in 2024-05-09T09:14:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:49 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:49.619 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T09:14:49.619 INFO:teuthology.orchestra.run.smithi103.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T09:14:49.619 INFO:teuthology.orchestra.run.smithi103.stderr: stderr: 10+0 records in 2024-05-09T09:14:49.619 INFO:teuthology.orchestra.run.smithi103.stderr:10+0 records out 2024-05-09T09:14:49.620 INFO:teuthology.orchestra.run.smithi103.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0137001 s, 765 MB/s 2024-05-09T09:14:49.620 INFO:teuthology.orchestra.run.smithi103.stderr:--> Zapping successful for: 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] boot 2024-05-09T09:14:50.354 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T09:14:50.355 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:50 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.383 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi103:vg_nvme/lv_1 2024-05-09T09:14:50.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] boot 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T09:14:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:50 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: osd.9 [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] boot 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: osdmap e67: 10 total, 10 up, 10 in 2024-05-09T09:14:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:50 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-05-09T09:14:50.620 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi069 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 4272M 2024-05-09T09:14:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:51.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:51.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:51 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.495 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi069 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 4272M 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:51.496 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:51 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: osdmap e68: 10 total, 10 up, 10 in 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi069 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 4272M 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:51.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:14:51.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:51 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:14:52.358 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:52 smithi103 ceph-mon[34421]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 -- 172.21.15.103:0/2697217709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 msgr2=0x7f1eec1027c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 --2- 172.21.15.103:0/2697217709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec1027c0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7f1edc0099c0 tx=0x7f1edc009cd0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 -- 172.21.15.103:0/2697217709 shutdown_connections 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 --2- 172.21.15.103:0/2697217709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec103f40 0x7f1eec10a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 --2- 172.21.15.103:0/2697217709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec1027c0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 --2- 172.21.15.103:0/2697217709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec1035a0 0x7f1eec103a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.364 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.364+0000 7f1ef2205700 1 -- 172.21.15.103:0/2697217709 >> 172.21.15.103:0/2697217709 conn(0x7f1eec0fd940 msgr2=0x7f1eec0ffd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:14:52.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 -- 172.21.15.103:0/2697217709 shutdown_connections 2024-05-09T09:14:52.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 -- 172.21.15.103:0/2697217709 wait complete. 2024-05-09T09:14:52.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 Processor -- start 2024-05-09T09:14:52.365 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 -- start start 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 0x7f1eec19ad70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.365+0000 7f1ef2205700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec103f40 0x7f1eec1a1df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ef2205700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec10dfe0 con 0x7f1eec103f40 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ef2205700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec10de60 con 0x7f1eec1023a0 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ef2205700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1eec10e160 con 0x7f1eec1035a0 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.103:43460/0 (socket says 172.21.15.103:43460) 2024-05-09T09:14:52.367 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 -- 172.21.15.103:0/1085472328 learned_addr learned my addr 172.21.15.103:0/1085472328 (peer_addr_for_me v2:172.21.15.103:0/0) 2024-05-09T09:14:52.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eebfff700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec103f40 0x7f1eec1a1df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:52.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeaffd700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 0x7f1eec19ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:52.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 -- 172.21.15.103:0/1085472328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 msgr2=0x7f1eec19ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:52.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 0x7f1eec19ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.368 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 -- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec103f40 msgr2=0x7f1eec1a1df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec103f40 0x7f1eec1a1df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 -- 172.21.15.103:0/1085472328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1eec1a2490 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeaffd700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 0x7f1eec19ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1eeb7fe700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00c010 tx=0x7f1edc009580 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc026070 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1eec1a2720 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1edc004150 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1edc00fe30 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.366+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1eec1a2c00 con 0x7f1eec1023a0 2024-05-09T09:14:52.369 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.367+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1eec069010 con 0x7f1eec1023a0 2024-05-09T09:14:52.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.369+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f1edc029070 con 0x7f1eec1023a0 2024-05-09T09:14:52.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.369+0000 7f1ee8ff9700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:14:52.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.370+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(68..68 src has 1..68) ==== 6261+0+0 (secure 0 0 0) 0x7f1edc0cf790 con 0x7f1eec1023a0 2024-05-09T09:14:52.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.370+0000 7f1eeaffd700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:14:52.370 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.370+0000 7f1eeaffd700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0005d90 tx=0x7f1ee0005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:14:52.372 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.373+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1edc030080 con 0x7f1eec1023a0 2024-05-09T09:14:52.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:52 smithi069 ceph-mon[30123]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:52.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:52 smithi163 ceph-mon[34354]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:52.575 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:14:52.576+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f1eec075c70 con 0x7f1ed40a2c20 2024-05-09T09:14:53.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:53.461 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:53.462 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:53.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:14:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:14:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:14:54.441 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:54 smithi103 ceph-mon[34421]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:54.442 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:54 smithi103 ceph-mon[34421]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:54 smithi069 ceph-mon[30123]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:54 smithi069 ceph-mon[30123]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:54.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:54 smithi163 ceph-mon[34354]: from='client.24338 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi103:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:14:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:54 smithi163 ceph-mon[34354]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/1383277628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]': finished 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T09:14:56.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:14:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/1383277628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]': finished 2024-05-09T09:14:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T09:14:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:14:56.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/1383277628' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]: dispatch 2024-05-09T09:14:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "dbc63c27-da12-4420-b03a-0c4d5721bd9c"}]': finished 2024-05-09T09:14:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: osdmap e69: 11 total, 10 up, 11 in 2024-05-09T09:14:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:14:57.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:57 smithi069 ceph-mon[30123]: from='client.? 172.21.15.103:0/291791931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:57.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:57 smithi163 ceph-mon[34354]: from='client.? 172.21.15.103:0/291791931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:57.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:57 smithi103 ceph-mon[34421]: from='client.? 172.21.15.103:0/291791931' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:14:58.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:14:58 smithi069 ceph-mon[30123]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:58.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:14:58 smithi163 ceph-mon[34354]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:14:58.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:14:58 smithi103 ceph-mon[34421]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:00.434 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:00 smithi103 ceph-mon[34421]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:00 smithi069 ceph-mon[30123]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:00.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:00 smithi163 ceph-mon[34354]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:02.468 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:02 smithi103 ceph-mon[34421]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:02 smithi069 ceph-mon[30123]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:02.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:02 smithi163 ceph-mon[34354]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:04.275 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:04 smithi103 ceph-mon[34421]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:04.275 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T09:15:04.276 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:04 smithi069 ceph-mon[30123]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T09:15:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:04 smithi163 ceph-mon[34354]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-05-09T09:15:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:05.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:05 smithi103 ceph-mon[34421]: Deploying daemon osd.10 on smithi103 2024-05-09T09:15:05.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:05 smithi069 ceph-mon[30123]: Deploying daemon osd.10 on smithi103 2024-05-09T09:15:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:05 smithi163 ceph-mon[34354]: Deploying daemon osd.10 on smithi103 2024-05-09T09:15:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:06 smithi069 ceph-mon[30123]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:06.538 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:06 smithi103 ceph-mon[34421]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:06 smithi163 ceph-mon[34354]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:07.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:07 smithi069 ceph-mon[30123]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:07.552 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:07 smithi103 ceph-mon[34421]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:07.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:07 smithi163 ceph-mon[34354]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:10.289 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:10 smithi103 ceph-mon[34421]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:10.289 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.289 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.289 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:10 smithi069 ceph-mon[30123]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:10 smithi163 ceph-mon[34354]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:12.178 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:12 smithi103 ceph-mon[34421]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:12.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:12 smithi069 ceph-mon[30123]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:12.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:12 smithi163 ceph-mon[34354]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:13.392 INFO:teuthology.orchestra.run.smithi103.stdout:Created osd(s) 10 on host 'smithi103' 2024-05-09T09:15:13.392 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.392+0000 7f1ee8ff9700 1 -- 172.21.15.103:0/1085472328 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f1eec075c70 con 0x7f1ed40a2c20 2024-05-09T09:15:13.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 msgr2=0x7f1ed40a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:13.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f1ee0005d90 tx=0x7f1ee0005d20 comp rx=0 tx=0).stop 2024-05-09T09:15:13.393 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 msgr2=0x7f1eec19a830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f1edc00c010 tx=0x7f1edc009580 comp rx=0 tx=0).stop 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 shutdown_connections 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1eec103f40 0x7f1eec1a1df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1ed40a2c20 0x7f1ed40a50e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1eec1023a0 0x7f1eec19a830 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 --2- 172.21.15.103:0/1085472328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1eec1035a0 0x7f1eec19ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 >> 172.21.15.103:0/1085472328 conn(0x7f1eec0fd940 msgr2=0x7f1eec074550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 shutdown_connections 2024-05-09T09:15:13.394 INFO:teuthology.orchestra.run.smithi103.stderr:2024-05-09T09:15:13.394+0000 7f1ef2205700 1 -- 172.21.15.103:0/1085472328 wait complete. 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:14.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:14.096 DEBUG:teuthology.orchestra.run.smithi103:osd.10> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.10.service 2024-05-09T09:15:14.098 INFO:tasks.cephadm:Deploying osd.11 on smithi163 with /dev/vg_nvme/lv_1... 2024-05-09T09:15:14.098 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-05-09T09:15:14.344 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:15:14.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:14 smithi163 ceph-mon[34354]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T09:15:14.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:14 smithi163 ceph-mon[34354]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T09:15:14.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:14 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:14.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:14 smithi163 ceph-mon[34354]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:14.731 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:14 smithi163 ceph-mon[34354]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:14 smithi103 ceph-mon[34421]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T09:15:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:14 smithi103 ceph-mon[34421]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T09:15:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:14 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:14 smithi103 ceph-mon[34421]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:14 smithi103 ceph-mon[34421]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:14.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:15:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:15:14.616+0000 7f8806c09700 -1 osd.10 0 waiting for initial osdmap 2024-05-09T09:15:14.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:15:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:15:14.632+0000 7f88029fc700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:15:15.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:14 smithi069 ceph-mon[30123]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-05-09T09:15:15.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:14 smithi069 ceph-mon[30123]: osdmap e70: 11 total, 10 up, 11 in 2024-05-09T09:15:15.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:14 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:15.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:14 smithi069 ceph-mon[30123]: from='osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:15.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:14 smithi069 ceph-mon[30123]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]: dispatch 2024-05-09T09:15:15.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:15 smithi163 ceph-mon[34354]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:15:15.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:15 smithi163 ceph-mon[34354]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T09:15:15.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:15.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:15.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:15 smithi163 ceph-mon[34354]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:15 smithi103 ceph-mon[34421]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:15:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:15 smithi103 ceph-mon[34421]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T09:15:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:15 smithi103 ceph-mon[34421]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:16.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:15 smithi069 ceph-mon[30123]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi103", "root=default"]}]': finished 2024-05-09T09:15:16.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:15 smithi069 ceph-mon[30123]: osdmap e71: 11 total, 10 up, 11 in 2024-05-09T09:15:16.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:15 smithi069 ceph-mon[30123]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 283 MiB used, 894 GiB / 894 GiB avail 2024-05-09T09:15:16.563 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-05-09T09:15:16.563 INFO:teuthology.orchestra.run.smithi163.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-05-09T09:15:16.563 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10+0 records in 2024-05-09T09:15:16.563 INFO:teuthology.orchestra.run.smithi163.stderr:10+0 records out 2024-05-09T09:15:16.563 INFO:teuthology.orchestra.run.smithi163.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124365 s, 843 MB/s 2024-05-09T09:15:16.564 INFO:teuthology.orchestra.run.smithi163.stderr:--> Zapping successful for: 2024-05-09T09:15:16.845 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] boot 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.846 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:16 smithi103 ceph-mon[34421]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T09:15:16.868 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] boot 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:16.869 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:16 smithi163 ceph-mon[34354]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T09:15:17.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: osd.10 [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] boot 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: osdmap e72: 11 total, 11 up, 11 in 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-05-09T09:15:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:16 smithi069 ceph-mon[30123]: osdmap e73: 11 total, 11 up, 11 in 2024-05-09T09:15:17.254 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch daemon add osd smithi163:vg_nvme/lv_1 2024-05-09T09:15:17.502 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi103 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:18 smithi069 ceph-mon[30123]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T09:15:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi103 2024-05-09T09:15:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:18 smithi163 ceph-mon[34354]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T09:15:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi103 2024-05-09T09:15:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:18.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:18 smithi103 ceph-mon[34421]: osdmap e74: 11 total, 11 up, 11 in 2024-05-09T09:15:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 -- 172.21.15.163:0/63501950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 msgr2=0x7f3b301016a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 --2- 172.21.15.163:0/63501950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b301016a0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f3b24009990 tx=0x7f3b24009ca0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 -- 172.21.15.163:0/63501950 shutdown_connections 2024-05-09T09:15:19.200 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 --2- 172.21.15.163:0/63501950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102e20 0x7f3b301096d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 --2- 172.21.15.163:0/63501950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b301016a0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 --2- 172.21.15.163:0/63501950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102480 0x7f3b301028e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 -- 172.21.15.163:0/63501950 >> 172.21.15.163:0/63501950 conn(0x7f3b300fc820 msgr2=0x7f3b300fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:19.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 -- 172.21.15.163:0/63501950 shutdown_connections 2024-05-09T09:15:19.201 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.201+0000 7f3b34d78700 1 -- 172.21.15.163:0/63501950 wait complete. 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 Processor -- start 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 -- start start 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 0x7f3b30103860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 0x7f3b30103da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:19.203 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3010cf30 con 0x7f3b30102480 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3010cdb0 con 0x7f3b30102e20 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b34d78700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3b3010d0b0 con 0x7f3b30101280 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b2e59c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:41420/0 (socket says 172.21.15.163:41420) 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b2e59c700 1 -- 172.21.15.163:0/4034160243 learned_addr learned my addr 172.21.15.163:0/4034160243 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.202+0000 7f3b2ed9d700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 0x7f3b30103da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2dd9b700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 0x7f3b30103860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 -- 172.21.15.163:0/4034160243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 msgr2=0x7f3b30103da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 0x7f3b30103da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 -- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 msgr2=0x7f3b30103860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 0x7f3b30103860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 -- 172.21.15.163:0/4034160243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3b30104650 con 0x7f3b30101280 2024-05-09T09:15:19.204 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2dd9b700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 0x7f3b30103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2ed9d700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 0x7f3b30103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b2e59c700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b240039c0 tx=0x7f3b2400bfa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24026070 con 0x7f3b30101280 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3b24004150 con 0x7f3b30101280 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3b301a27f0 con 0x7f3b30101280 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3b24004af0 con 0x7f3b30101280 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.203+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3b301a2cc0 con 0x7f3b30101280 2024-05-09T09:15:19.205 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.204+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3b30069010 con 0x7f3b30101280 2024-05-09T09:15:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.205+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3b24029070 con 0x7f3b30101280 2024-05-09T09:15:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.206+0000 7f3b1f7fe700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 0x7f3b180a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.206+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(75..75 src has 1..75) ==== 6694+0+0 (secure 0 0 0) 0x7f3b240cf510 con 0x7f3b30101280 2024-05-09T09:15:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.206+0000 7f3b2dd9b700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 0x7f3b180a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:19.206 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.206+0000 7f3b2dd9b700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 0x7f3b180a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005fd0 tx=0x7f3b20005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:19.209 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.210+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3b2405d740 con 0x7f3b30101280 2024-05-09T09:15:19.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:19 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-09T09:15:19.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:19 smithi163 ceph-mon[34354]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T09:15:19.416 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:19.416+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) -- 0x7f3b300fea50 con 0x7f3b180a2c20 2024-05-09T09:15:19.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:19 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-09T09:15:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:19 smithi069 ceph-mon[30123]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T09:15:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:19 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 4272M 2024-05-09T09:15:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:19 smithi103 ceph-mon[34421]: osdmap e75: 11 total, 11 up, 11 in 2024-05-09T09:15:20.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:20 smithi163 ceph-mon[34354]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:15:20.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:15:20.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:15:20.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:20 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:20.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:20 smithi163 ceph-mon[34354]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T09:15:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:20 smithi069 ceph-mon[30123]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:15:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:15:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:15:20.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:20 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:20.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:20 smithi069 ceph-mon[30123]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T09:15:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:20 smithi103 ceph-mon[34421]: pgmap v240: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 310 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-05-09T09:15:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-05-09T09:15:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-05-09T09:15:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:20 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:20 smithi103 ceph-mon[34421]: osdmap e76: 11 total, 11 up, 11 in 2024-05-09T09:15:21.357 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:21 smithi163 ceph-mon[34354]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:21.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:21 smithi069 ceph-mon[30123]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:21 smithi103 ceph-mon[34421]: from='client.24364 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi163:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:22.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:22 smithi069 ceph-mon[30123]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:15:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:22 smithi163 ceph-mon[34354]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:15:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:22 smithi103 ceph-mon[34421]: pgmap v242: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 123 KiB/s, 0 objects/s recovering 2024-05-09T09:15:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/459435719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]': finished 2024-05-09T09:15:23.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T09:15:23.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:23.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: from='client.? 172.21.15.163:0/584945428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:15:23.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:23 smithi069 ceph-mon[30123]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/459435719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]': finished 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: from='client.? 172.21.15.163:0/584945428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:15:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:23 smithi163 ceph-mon[34354]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/459435719' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]: dispatch 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9e951639-2f9f-4524-9e05-c9b222ac9a05"}]': finished 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: osdmap e77: 12 total, 11 up, 12 in 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: from='client.? 172.21.15.163:0/584945428' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-05-09T09:15:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:23 smithi103 ceph-mon[34421]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail; 101 KiB/s, 0 objects/s recovering 2024-05-09T09:15:26.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:26 smithi163 ceph-mon[34354]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:26 smithi069 ceph-mon[30123]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:26 smithi103 ceph-mon[34421]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:28 smithi069 ceph-mon[30123]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:28.553 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:28 smithi163 ceph-mon[34354]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:28.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:28 smithi103 ceph-mon[34421]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:30.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:30 smithi163 ceph-mon[34354]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T09:15:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:30 smithi069 ceph-mon[30123]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T09:15:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:30 smithi103 ceph-mon[34421]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-05-09T09:15:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:31.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:31 smithi163 ceph-mon[34354]: Deploying daemon osd.11 on smithi163 2024-05-09T09:15:31.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:31 smithi069 ceph-mon[30123]: Deploying daemon osd.11 on smithi163 2024-05-09T09:15:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:31 smithi103 ceph-mon[34421]: Deploying daemon osd.11 on smithi163 2024-05-09T09:15:32.298 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:32 smithi163 ceph-mon[34354]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:32 smithi069 ceph-mon[30123]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:32 smithi103 ceph-mon[34421]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:34.211 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:34 smithi163 ceph-mon[34354]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:34.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:34 smithi069 ceph-mon[30123]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:34.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:34 smithi103 ceph-mon[34421]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:36.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:36 smithi163 ceph-mon[34354]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:36.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.425 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:36 smithi069 ceph-mon[30123]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:36 smithi103 ceph-mon[34421]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:38.419 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:38 smithi103 ceph-mon[34421]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:38.440 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:38 smithi163 ceph-mon[34354]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:38.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:38 smithi069 ceph-mon[30123]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:40.085 INFO:teuthology.orchestra.run.smithi163.stdout:Created osd(s) 11 on host 'smithi163' 2024-05-09T09:15:40.085 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.085+0000 7f3b1f7fe700 1 -- 172.21.15.163:0/4034160243 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+38 (secure 0 0 0) 0x7f3b300fea50 con 0x7f3b180a2c20 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 msgr2=0x7f3b180a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 0x7f3b180a50e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f3b20005fd0 tx=0x7f3b20005ee0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 msgr2=0x7f3b30105210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b240039c0 tx=0x7f3b2400bfa0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 shutdown_connections 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3b180a2c20 0x7f3b180a50e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3b30102e20 0x7f3b30103da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3b30101280 0x7f3b30105210 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 --2- 172.21.15.163:0/4034160243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3b30102480 0x7f3b30103860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 >> 172.21.15.163:0/4034160243 conn(0x7f3b300fc820 msgr2=0x7f3b300fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:40.088 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 shutdown_connections 2024-05-09T09:15:40.089 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:40.088+0000 7f3b34d78700 1 -- 172.21.15.163:0/4034160243 wait complete. 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:40 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:40.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:40 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-05-09T09:15:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:40.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:40 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:40.808 DEBUG:teuthology.orchestra.run.smithi163:osd.11> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.11.service 2024-05-09T09:15:40.811 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-05-09T09:15:40.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd stat -f json 2024-05-09T09:15:41.071 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:15:41.407 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T09:15:41.407 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T09:15:41.407 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:41.408 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.408 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.408 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:41 smithi163 ceph-mon[34354]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:41.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T09:15:41.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T09:15:41.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:41.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.439 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:41 smithi069 ceph-mon[30123]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-05-09T09:15:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: osdmap e78: 12 total, 11 up, 12 in 2024-05-09T09:15:41.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:41.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: from='osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]: dispatch 2024-05-09T09:15:41.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:41 smithi103 ceph-mon[34421]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:42.016 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:15:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:15:41.721+0000 7f7c9de23700 -1 osd.11 0 waiting for initial osdmap 2024-05-09T09:15:42.016 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:15:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:15:41.736+0000 7f7c95c0e700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-05-09T09:15:42.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:42 smithi069 ceph-mon[30123]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:15:42.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:42 smithi069 ceph-mon[30123]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T09:15:42.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:42 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.464 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:42 smithi069 ceph-mon[30123]: from='osd.11 ' entity='osd.11' 2024-05-09T09:15:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:42 smithi163 ceph-mon[34354]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:15:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:42 smithi163 ceph-mon[34354]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T09:15:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:42 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:42 smithi163 ceph-mon[34354]: from='osd.11 ' entity='osd.11' 2024-05-09T09:15:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:42 smithi103 ceph-mon[34421]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi163", "root=default"]}]': finished 2024-05-09T09:15:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:42 smithi103 ceph-mon[34421]: osdmap e79: 12 total, 11 up, 12 in 2024-05-09T09:15:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:42 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:42.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:42 smithi103 ceph-mon[34421]: from='osd.11 ' entity='osd.11' 2024-05-09T09:15:42.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.865+0000 7f5ce4330700 1 -- 172.21.15.69:0/2446242715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 msgr2=0x7f5cdc10a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:42.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.865+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2446242715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc10a9a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f5cd400b0a0 tx=0x7f5cd400b3b0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- 172.21.15.69:0/2446242715 shutdown_connections 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2446242715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc10a9a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2446242715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc103dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2446242715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc100890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- 172.21.15.69:0/2446242715 >> 172.21.15.69:0/2446242715 conn(0x7f5cdc0fbff0 msgr2=0x7f5cdc0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- 172.21.15.69:0/2446242715 shutdown_connections 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- 172.21.15.69:0/2446242715 wait complete. 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 Processor -- start 2024-05-09T09:15:42.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- start start 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc198510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc19fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10dfc0 con 0x7f5cdc100dd0 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10de40 con 0x7f5cdc1039b0 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.866+0000 7f5ce4330700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5cdc10e140 con 0x7f5cdc100430 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:42.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38742/0 (socket says 172.21.15.69:38742) 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 -- 172.21.15.69:0/2391418434 learned_addr learned my addr 172.21.15.69:0/2391418434 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce20cc700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc198510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce28cd700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc19fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 -- 172.21.15.69:0/2391418434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 msgr2=0x7f5cdc198510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc198510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 -- 172.21.15.69:0/2391418434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 msgr2=0x7f5cdc19fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc19fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 -- 172.21.15.69:0/2391418434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5cdc1a0170 con 0x7f5cdc100dd0 2024-05-09T09:15:42.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce28cd700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce20cc700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce18cb700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00e9f0 tx=0x7f5ccc00ed00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc00cbd0 con 0x7f5cdc100dd0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ccc00cd70 con 0x7f5cdc100dd0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5cdc1a0460 con 0x7f5cdc100dd0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ccc010430 con 0x7f5cdc100dd0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.867+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5cdc1a0950 con 0x7f5cdc100dd0 2024-05-09T09:15:42.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.868+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5cdc069010 con 0x7f5cdc100dd0 2024-05-09T09:15:42.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.869+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5ccc004750 con 0x7f5cdc100dd0 2024-05-09T09:15:42.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.870+0000 7f5cd37fe700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 0x7f5cc80a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:42.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.870+0000 7f5ce20cc700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 0x7f5cc80a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:42.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.870+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(80..80 src has 1..80) ==== 7127+0+0 (secure 0 0 0) 0x7f5ccc0cee30 con 0x7f5cdc100dd0 2024-05-09T09:15:42.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.870+0000 7f5ce20cc700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 0x7f5cc80a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06a790 tx=0x7f5cd8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:42.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:42.874+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5ccc09c230 con 0x7f5cdc100dd0 2024-05-09T09:15:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.069+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) -- 0x7f5cdc02db80 con 0x7f5cdc100dd0 2024-05-09T09:15:43.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.070+0000 7f5cd37fe700 1 -- 172.21.15.69:0/2391418434 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v80) ==== 74+0+133 (secure 0 0 0) 0x7f5ccc09bb30 con 0x7f5cdc100dd0 2024-05-09T09:15:43.069 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 msgr2=0x7f5cc80a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 0x7f5cc80a50e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f5cdc06a790 tx=0x7f5cd8009260 comp rx=0 tx=0).stop 2024-05-09T09:15:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 msgr2=0x7f5cdc198a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:43.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f5ccc00e9f0 tx=0x7f5ccc00ed00 comp rx=0 tx=0).stop 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 shutdown_connections 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5cc80a2c20 0x7f5cc80a50e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5cdc1039b0 0x7f5cdc19fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5cdc100430 0x7f5cdc198510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 --2- 172.21.15.69:0/2391418434 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5cdc100dd0 0x7f5cdc198a50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 >> 172.21.15.69:0/2391418434 conn(0x7f5cdc0fbff0 msgr2=0x7f5cdc106980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 shutdown_connections 2024-05-09T09:15:43.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:43.072+0000 7f5ce4330700 1 -- 172.21.15.69:0/2391418434 wait complete. 2024-05-09T09:15:43.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: purged_snaps scrub starts 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: purged_snaps scrub ok 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] boot 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2391418434' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T09:15:43.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:43 smithi069 ceph-mon[30123]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: purged_snaps scrub starts 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: purged_snaps scrub ok 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] boot 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2391418434' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T09:15:43.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:43 smithi163 ceph-mon[34354]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: purged_snaps scrub starts 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: purged_snaps scrub ok 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: osd.11 [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] boot 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: osdmap e80: 12 total, 12 up, 12 in 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2391418434' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-05-09T09:15:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:43 smithi103 ceph-mon[34421]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 311 MiB used, 983 GiB / 983 GiB avail 2024-05-09T09:15:43.738 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":80,"num_osds":12,"num_up_osds":12,"osd_up_since":1715246142,"num_in_osds":12,"osd_in_since":1715246122,"num_remapped_pgs":0} 2024-05-09T09:15:43.739 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T09:15:43.996 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: Detected new or changed devices on smithi163 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-09T09:15:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:45.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:45.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:44 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: Detected new or changed devices on smithi163 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-09T09:15:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:45.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:45.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:44 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: osdmap e81: 12 total, 12 up, 12 in 2024-05-09T09:15:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: Detected new or changed devices on smithi163 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 4272M 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:45.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:44 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:45.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 -- 172.21.15.69:0/1861591457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7806a030 msgr2=0x7f5f7806a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 --2- 172.21.15.69:0/1861591457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7806a030 0x7f5f7806a490 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f5f68009990 tx=0x7f5f68009ca0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 -- 172.21.15.69:0/1861591457 shutdown_connections 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 --2- 172.21.15.69:0/1861591457 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 0x7f5f7810f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 --2- 172.21.15.69:0/1861591457 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f781083a0 0x7f5f78108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 --2- 172.21.15.69:0/1861591457 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f7806a030 0x7f5f7806a490 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 -- 172.21.15.69:0/1861591457 >> 172.21.15.69:0/1861591457 conn(0x7f5f780781d0 msgr2=0x7f5f780785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 -- 172.21.15.69:0/1861591457 shutdown_connections 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.746+0000 7f5f80100700 1 -- 172.21.15.69:0/1861591457 wait complete. 2024-05-09T09:15:45.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 Processor -- start 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 -- start start 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 0x7f5f7819e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 0x7f5f7819eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5f78112650 con 0x7f5f781083a0 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5f781124d0 con 0x7f5f7806a9d0 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f80100700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5f781127d0 con 0x7f5f7806a030 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38770/0 (socket says 172.21.15.69:38770) 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 -- 172.21.15.69:0/275681952 learned_addr learned my addr 172.21.15.69:0/275681952 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:45.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7d69b700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 0x7f5f7819eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7de9c700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 0x7f5f7819e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 -- 172.21.15.69:0/275681952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 msgr2=0x7f5f7819e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 0x7f5f7819e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 -- 172.21.15.69:0/275681952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 msgr2=0x7f5f7819eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 0x7f5f7819eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7e69d700 1 -- 172.21.15.69:0/275681952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5f781990d0 con 0x7f5f781083a0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.747+0000 7f5f7de9c700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 0x7f5f7819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f7e69d700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f70007ee0 tx=0x7f5f7000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f7000dbd0 con 0x7f5f781083a0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5f781993c0 con 0x7f5f781083a0 2024-05-09T09:15:45.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5f7000dd70 con 0x7f5f781083a0 2024-05-09T09:15:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5f70005a60 con 0x7f5f781083a0 2024-05-09T09:15:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.748+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5f781a5630 con 0x7f5f781083a0 2024-05-09T09:15:45.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.749+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5f78069010 con 0x7f5f781083a0 2024-05-09T09:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.750+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5f7001a020 con 0x7f5f781083a0 2024-05-09T09:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.750+0000 7f5f6effd700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 0x7f5f640a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.750+0000 7f5f7de9c700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 0x7f5f640a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.751+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f5f700ce2c0 con 0x7f5f781083a0 2024-05-09T09:15:45.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.751+0000 7f5f7de9c700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 0x7f5f640a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f780ff800 tx=0x7f5f74005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:45.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.755+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5f70010070 con 0x7f5f781083a0 2024-05-09T09:15:45.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:45 smithi069 ceph-mon[30123]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:45.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.951+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5f7802db80 con 0x7f5f781083a0 2024-05-09T09:15:45.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.952+0000 7f5f6effd700 1 -- 172.21.15.69:0/275681952 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18444 (secure 0 0 0) 0x7f5f7009b6c0 con 0x7f5f781083a0 2024-05-09T09:15:45.951 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:45.952 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":81,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:15:43.727619+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:15:45.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.954+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 msgr2=0x7f5f640a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:45.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.954+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 0x7f5f640a50e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5f780ff800 tx=0x7f5f74005d80 comp rx=0 tx=0).stop 2024-05-09T09:15:45.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.954+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 msgr2=0x7f5f78198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:45.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.954+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f5f70007ee0 tx=0x7f5f7000d3b0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 shutdown_connections 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5f781083a0 0x7f5f78198970 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5f7806a030 0x7f5f7819e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5f640a2c20 0x7f5f640a50e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 --2- 172.21.15.69:0/275681952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5f7806a9d0 0x7f5f7819eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 >> 172.21.15.69:0/275681952 conn(0x7f5f780781d0 msgr2=0x7f5f780ffe70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 shutdown_connections 2024-05-09T09:15:45.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:45.955+0000 7f5f80100700 1 -- 172.21.15.69:0/275681952 wait complete. 2024-05-09T09:15:46.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:45 smithi163 ceph-mon[34354]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:45 smithi103 ceph-mon[34421]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:46.669 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T09:15:46.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-05-09T09:15:46.910 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:15:47.001 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/275681952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:47.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/275681952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:47.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/275681952' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:48.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:47 smithi069 ceph-mon[30123]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:48.054 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:47 smithi163 ceph-mon[34354]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:48.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:47 smithi103 ceph-mon[34421]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 -- 172.21.15.69:0/3676020378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9744100ce0 msgr2=0x7f9744101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 --2- 172.21.15.69:0/3676020378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9744100ce0 0x7f9744101160 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f9734009990 tx=0x7f9734009ca0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 -- 172.21.15.69:0/3676020378 shutdown_connections 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 --2- 172.21.15.69:0/3676020378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f97441016a0 0x7f974410f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 --2- 172.21.15.69:0/3676020378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744107400 0x7f97441077e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 --2- 172.21.15.69:0/3676020378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9744100ce0 0x7f9744101160 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 -- 172.21.15.69:0/3676020378 >> 172.21.15.69:0/3676020378 conn(0x7f9744076540 msgr2=0x7f9744076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:48.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 -- 172.21.15.69:0/3676020378 shutdown_connections 2024-05-09T09:15:48.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 -- 172.21.15.69:0/3676020378 wait complete. 2024-05-09T09:15:48.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.595+0000 7f974916f700 1 Processor -- start 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 -- start start 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744100ce0 0x7f974419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9744107400 0x7f9744198860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f97441126e0 con 0x7f97441016a0 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9744112560 con 0x7f9744100ce0 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974916f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9744112860 con 0x7f9744107400 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39548/0 (socket says 172.21.15.69:39548) 2024-05-09T09:15:48.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 -- 172.21.15.69:0/197088630 learned_addr learned my addr 172.21.15.69:0/197088630 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f9742d9d700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744100ce0 0x7f974419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974359e700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9744107400 0x7f9744198860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 -- 172.21.15.69:0/197088630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9744107400 msgr2=0x7f9744198860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9744107400 0x7f9744198860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 -- 172.21.15.69:0/197088630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744100ce0 msgr2=0x7f974419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744100ce0 0x7f974419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.596+0000 7f974259c700 1 -- 172.21.15.69:0/197088630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9744198fc0 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f974259c700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f973400bff0 tx=0x7f9734005590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9734027070 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9744199250 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f97340040d0 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9734022e00 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.597+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f97441a5570 con 0x7f97441016a0 2024-05-09T09:15:48.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.598+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9744069010 con 0x7f97441016a0 2024-05-09T09:15:48.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.599+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f973402a070 con 0x7f97441016a0 2024-05-09T09:15:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.600+0000 7f972bfff700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 0x7f972c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.600+0000 7f9742d9d700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 0x7f972c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.600+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f97340d00a0 con 0x7f97441016a0 2024-05-09T09:15:48.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.600+0000 7f9742d9d700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 0x7f972c0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f97441999a0 tx=0x7f973800a3b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:48.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.603+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9734031020 con 0x7f97441016a0 2024-05-09T09:15:48.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.797+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f97441a57a0 con 0x7f97441016a0 2024-05-09T09:15:48.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.797+0000 7f972bfff700 1 -- 172.21.15.69:0/197088630 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f9734013070 con 0x7f97441016a0 2024-05-09T09:15:48.796 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:15:48.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 msgr2=0x7f972c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 0x7f972c0a50e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f97441999a0 tx=0x7f973800a3b0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 msgr2=0x7f974419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f973400bff0 tx=0x7f9734005590 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 shutdown_connections 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9744107400 0x7f9744198860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f972c0a2c20 0x7f972c0a50e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9744100ce0 0x7f974419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 --2- 172.21.15.69:0/197088630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f97441016a0 0x7f974419ebe0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 >> 172.21.15.69:0/197088630 conn(0x7f9744076540 msgr2=0x7f97440ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:48.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 shutdown_connections 2024-05-09T09:15:48.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:48.800+0000 7f974916f700 1 -- 172.21.15.69:0/197088630 wait complete. 2024-05-09T09:15:49.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/197088630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:49.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/197088630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:49.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/197088630' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.a on smithi069 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.d on smithi069 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.g on smithi069 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.j on smithi069 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.b on smithi103 2024-05-09T09:15:49.617 INFO:tasks.cephadm:Adding mds.e on smithi103 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.h on smithi103 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.k on smithi103 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.c on smithi163 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.f on smithi163 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.i on smithi163 2024-05-09T09:15:49.618 INFO:tasks.cephadm:Adding mds.l on smithi163 2024-05-09T09:15:49.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph orch apply mds all '12;smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l' 2024-05-09T09:15:49.871 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:15:49.993 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:49 smithi163 ceph-mon[34354]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:49 smithi103 ceph-mon[34421]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:50.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:49 smithi069 ceph-mon[30123]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 -- 172.21.15.163:0/2331834347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 msgr2=0x7fc990107b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 --2- 172.21.15.163:0/2331834347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990107b50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fc984009990 tx=0x7fc984009ca0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 -- 172.21.15.163:0/2331834347 shutdown_connections 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 --2- 172.21.15.163:0/2331834347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990108090 0x7fc99010a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 --2- 172.21.15.163:0/2331834347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990102e30 0x7fc990105220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.555 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 --2- 172.21.15.163:0/2331834347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990107b50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.553+0000 7fc995b18700 1 -- 172.21.15.163:0/2331834347 >> 172.21.15.163:0/2331834347 conn(0x7fc9900fc880 msgr2=0x7fc9900feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.554+0000 7fc995b18700 1 -- 172.21.15.163:0/2331834347 shutdown_connections 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.554+0000 7fc995b18700 1 -- 172.21.15.163:0/2331834347 wait complete. 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.554+0000 7fc995b18700 1 Processor -- start 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.554+0000 7fc995b18700 1 -- start start 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990102e30 0x7fc990198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990108090 0x7fc99019fac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc99010dfb0 con 0x7fc990102e30 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc99010de30 con 0x7fc990108090 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc99010e130 con 0x7fc990105760 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.163:47952/0 (socket says 172.21.15.163:47952) 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 -- 172.21.15.163:0/3824974744 learned_addr learned my addr 172.21.15.163:0/3824974744 (peer_addr_for_me v2:172.21.15.163:0/0) 2024-05-09T09:15:51.556 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98ffff700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990108090 0x7fc99019fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98f7fe700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990102e30 0x7fc990198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 -- 172.21.15.163:0/3824974744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990108090 msgr2=0x7fc99019fac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990108090 0x7fc99019fac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 -- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990102e30 msgr2=0x7fc990198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990102e30 0x7fc990198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 -- 172.21.15.163:0/3824974744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9901a0160 con 0x7fc990105760 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98effd700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc984004790 tx=0x7fc984009580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984027070 con 0x7fc990105760 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9901a03f0 con 0x7fc990105760 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc984004150 con 0x7fc990105760 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc984022e00 con 0x7fc990105760 2024-05-09T09:15:51.557 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.555+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9901a08a0 con 0x7fc990105760 2024-05-09T09:15:51.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.556+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc990069010 con 0x7fc990105760 2024-05-09T09:15:51.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.558+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc98402a070 con 0x7fc990105760 2024-05-09T09:15:51.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.558+0000 7fc98cff9700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 0x7fc9780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:51.558 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.558+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc9840cfd80 con 0x7fc990105760 2024-05-09T09:15:51.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.558+0000 7fc98f7fe700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 0x7fc9780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:51.559 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.559+0000 7fc98f7fe700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 0x7fc9780a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc980005950 tx=0x7fc98000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:51.561 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.562+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc984031020 con 0x7fc990105760 2024-05-09T09:15:51.765 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.765+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l", "target": ["mon-mgr", ""]}) -- 0x7fc990062960 con 0x7fc9780a2c20 2024-05-09T09:15:51.773 INFO:teuthology.orchestra.run.smithi163.stdout:Scheduled mds.all update... 2024-05-09T09:15:51.774 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.773+0000 7fc98cff9700 1 -- 172.21.15.163:0/3824974744 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+28 (secure 0 0 0) 0x7fc990062960 con 0x7fc9780a2c20 2024-05-09T09:15:51.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 msgr2=0x7fc9780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:51.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 0x7fc9780a50e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fc980005950 tx=0x7fc98000b410 comp rx=0 tx=0).stop 2024-05-09T09:15:51.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 msgr2=0x7fc990198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:51.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fc984004790 tx=0x7fc984009580 comp rx=0 tx=0).stop 2024-05-09T09:15:51.775 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 shutdown_connections 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9780a2c20 0x7fc9780a50e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc990108090 0x7fc99019fac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc990102e30 0x7fc990198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 --2- 172.21.15.163:0/3824974744 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc990105760 0x7fc990198a90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.775+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 >> 172.21.15.163:0/3824974744 conn(0x7fc9900fc880 msgr2=0x7fc990103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.776+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 shutdown_connections 2024-05-09T09:15:51.776 INFO:teuthology.orchestra.run.smithi163.stderr:2024-05-09T09:15:51.776+0000 7fc995b18700 1 -- 172.21.15.163:0/3824974744 wait complete. 2024-05-09T09:15:52.396 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:52 smithi163 ceph-mon[34354]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:52.397 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:52.397 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:52 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:52.418 DEBUG:teuthology.orchestra.run.smithi069:mds.a> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.a.service 2024-05-09T09:15:52.421 DEBUG:teuthology.orchestra.run.smithi069:mds.d> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.d.service 2024-05-09T09:15:52.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:52 smithi069 ceph-mon[30123]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:52.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:52.444 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:52 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:52.445 DEBUG:teuthology.orchestra.run.smithi069:mds.g> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.g.service 2024-05-09T09:15:52.495 DEBUG:teuthology.orchestra.run.smithi069:mds.j> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.j.service 2024-05-09T09:15:52.539 DEBUG:teuthology.orchestra.run.smithi103:mds.b> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.b.service 2024-05-09T09:15:52.542 DEBUG:teuthology.orchestra.run.smithi103:mds.e> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.e.service 2024-05-09T09:15:52.561 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:52 smithi103 ceph-mon[34421]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:52.561 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:52.562 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:52 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:15:52.563 DEBUG:teuthology.orchestra.run.smithi103:mds.h> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.h.service 2024-05-09T09:15:52.606 DEBUG:teuthology.orchestra.run.smithi103:mds.k> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.k.service 2024-05-09T09:15:52.649 DEBUG:teuthology.orchestra.run.smithi163:mds.c> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.c.service 2024-05-09T09:15:52.651 DEBUG:teuthology.orchestra.run.smithi163:mds.f> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.f.service 2024-05-09T09:15:52.693 DEBUG:teuthology.orchestra.run.smithi163:mds.i> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.i.service 2024-05-09T09:15:52.736 DEBUG:teuthology.orchestra.run.smithi163:mds.l> sudo journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.l.service 2024-05-09T09:15:52.779 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-05-09T09:15:52.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:15:52.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 -- 172.21.15.69:0/2130269441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f40757f0 msgr2=0x7f19f4073c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/2130269441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f40757f0 0x7f19f4073c40 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009900 tx=0x7f19e402a480 comp rx=0 tx=0).stop 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 -- 172.21.15.69:0/2130269441 shutdown_connections 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/2130269441 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 0x7f19f410bbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/2130269441 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f4074210 0x7f19f4074670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/2130269441 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f40757f0 0x7f19f4073c40 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 -- 172.21.15.69:0/2130269441 >> 172.21.15.69:0/2130269441 conn(0x7f19f40fce10 msgr2=0x7f19f40ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.888+0000 7f19fa4ee640 1 -- 172.21.15.69:0/2130269441 shutdown_connections 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 -- 172.21.15.69:0/2130269441 wait complete. 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 Processor -- start 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 -- start start 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:52.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f40757f0 0x7f19f419a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 0x7f19f41a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410f6e0 con 0x7f19f4074210 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410f560 con 0x7f19f4109730 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19fa4ee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f19f410f860 con 0x7f19f40757f0 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19f3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19f3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39572/0 (socket says 172.21.15.69:39572) 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.889+0000 7f19f3fff640 1 -- 172.21.15.69:0/1611823529 learned_addr learned my addr 172.21.15.69:0/1611823529 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f8a8b640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 0x7f19f41a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:52.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f37fe640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f40757f0 0x7f19f419a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 -- 172.21.15.69:0/1611823529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f40757f0 msgr2=0x7f19f419a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f40757f0 0x7f19f419a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 -- 172.21.15.69:0/1611823529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 msgr2=0x7f19f41a1310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 0x7f19f41a1310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 -- 172.21.15.69:0/1611823529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f19f41a19b0 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f3fff640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009230 tx=0x7f19e4004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e403f030 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f19e4003ce0 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f19f41a1c40 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19e4035070 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.890+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f19f41a2130 con 0x7f19f4074210 2024-05-09T09:15:52.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.891+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f19f41084f0 con 0x7f19f4074210 2024-05-09T09:15:52.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.892+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f19e4042030 con 0x7f19f4074210 2024-05-09T09:15:52.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.893+0000 7f19f17fa640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 0x7f19dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:52.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.893+0000 7f19f37fe640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 0x7f19dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:52.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.893+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f19e40ef5d0 con 0x7f19f4074210 2024-05-09T09:15:52.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.893+0000 7f19f37fe640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 0x7f19dc0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005d00 tx=0x7f19e800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:52.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:52.896+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f19e40bc4a0 con 0x7f19f4074210 2024-05-09T09:15:53.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.076+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f19f402d410 con 0x7f19f4074210 2024-05-09T09:15:53.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.077+0000 7f19f17fa640 1 -- 172.21.15.69:0/1611823529 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18444 (secure 0 0 0) 0x7f19e40c07a0 con 0x7f19f4074210 2024-05-09T09:15:53.077 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:53.077 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":81,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:15:43.727619+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 msgr2=0x7f19dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 0x7f19dc0a5050 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f19e8005d00 tx=0x7f19e800b410 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 msgr2=0x7f19f4199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f19e4009230 tx=0x7f19e4004360 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 shutdown_connections 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19dc0a2b90 0x7f19dc0a5050 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f19f4109730 0x7f19f41a1310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f19f40757f0 0x7f19f419a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 --2- 172.21.15.69:0/1611823529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f19f4074210 0x7f19f4199da0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 >> 172.21.15.69:0/1611823529 conn(0x7f19f40fce10 msgr2=0x7f19f41056c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 shutdown_connections 2024-05-09T09:15:53.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.079+0000 7f19fa4ee640 1 -- 172.21.15.69:0/1611823529 wait complete. 2024-05-09T09:15:53.090 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T09:15:53.090 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:15:53.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.195+0000 7f2067abd640 1 -- 172.21.15.69:0/2306116308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060101960 msgr2=0x7f2060101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.195+0000 7f2067abd640 1 --2- 172.21.15.69:0/2306116308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060101960 0x7f2060101de0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f2050009900 tx=0x7f205002a410 comp rx=0 tx=0).stop 2024-05-09T09:15:53.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- 172.21.15.69:0/2306116308 shutdown_connections 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- 172.21.15.69:0/2306116308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060102320 0x7f2060108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- 172.21.15.69:0/2306116308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060101960 0x7f2060101de0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- 172.21.15.69:0/2306116308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060100770 0x7f2060100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- 172.21.15.69:0/2306116308 >> 172.21.15.69:0/2306116308 conn(0x7f20600fbce0 msgr2=0x7f20600fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- 172.21.15.69:0/2306116308 shutdown_connections 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- 172.21.15.69:0/2306116308 wait complete. 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 Processor -- start 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- start start 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060100770 0x7f2060197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 0x7f2060198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f206010c3d0 con 0x7f2060102320 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f206010c250 con 0x7f2060100770 2024-05-09T09:15:53.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f2067abd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f206010c550 con 0x7f2060101960 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.196+0000 7f206605a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2065058640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 0x7f2060198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39596/0 (socket says 172.21.15.69:39596) 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2065058640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 0x7f2060198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51378/0 (socket says 172.21.15.69:51378) 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 -- 172.21.15.69:0/745361462 learned_addr learned my addr 172.21.15.69:0/745361462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2065859640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060100770 0x7f2060197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 -- 172.21.15.69:0/745361462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 msgr2=0x7f2060198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 0x7f2060198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 -- 172.21.15.69:0/745361462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060100770 msgr2=0x7f2060197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060100770 0x7f2060197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 -- 172.21.15.69:0/745361462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206019f790 con 0x7f2060102320 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f206605a640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f205800c970 tx=0x7f205800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058007bb0 con 0x7f2060102320 2024-05-09T09:15:53.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2058007d50 con 0x7f2060102320 2024-05-09T09:15:53.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f206019fa80 con 0x7f2060102320 2024-05-09T09:15:53.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058018800 con 0x7f2060102320 2024-05-09T09:15:53.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.197+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f206019ffa0 con 0x7f2060102320 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.198+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f20601a0230 con 0x7f2060102320 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.199+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f205800fa50 con 0x7f2060102320 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.200+0000 7f2056ffd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 0x7f204c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.200+0000 7f2065859640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 0x7f204c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.200+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f20580cec80 con 0x7f2060102320 2024-05-09T09:15:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.200+0000 7f2065859640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 0x7f204c0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f20601017c0 tx=0x7f205c009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:53.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.203+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2058014030 con 0x7f2060102320 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: Saving service mds.all spec with placement smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l;count:12 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1611823529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:53.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.380+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f20601a0450 con 0x7f2060102320 2024-05-09T09:15:53.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.380+0000 7f2056ffd640 1 -- 172.21.15.69:0/745361462 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7f205809c100 con 0x7f2060102320 2024-05-09T09:15:53.380 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 msgr2=0x7f204c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 0x7f204c0a50d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f20601017c0 tx=0x7f205c009670 comp rx=0 tx=0).stop 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 msgr2=0x7f206019f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f205800c970 tx=0x7f205800ce30 comp rx=0 tx=0).stop 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 shutdown_connections 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f204c0a2c10 0x7f204c0a50d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2060102320 0x7f206019f120 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2060101960 0x7f2060198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 --2- 172.21.15.69:0/745361462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2060100770 0x7f2060197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 >> 172.21.15.69:0/745361462 conn(0x7f20600fbce0 msgr2=0x7f20600689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:53.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 shutdown_connections 2024-05-09T09:15:53.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.382+0000 7f2067abd640 1 -- 172.21.15.69:0/745361462 wait complete. 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: Saving service mds.all spec with placement smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l;count:12 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.392 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1611823529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:53.393 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:15:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 -- 172.21.15.69:0/1832022956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968101960 msgr2=0x7fa968101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 --2- 172.21.15.69:0/1832022956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968101960 0x7fa968101de0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fa950009900 tx=0x7fa95002a410 comp rx=0 tx=0).stop 2024-05-09T09:15:53.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 -- 172.21.15.69:0/1832022956 shutdown_connections 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 --2- 172.21.15.69:0/1832022956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968102320 0x7fa968108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 --2- 172.21.15.69:0/1832022956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968101960 0x7fa968101de0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 --2- 172.21.15.69:0/1832022956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968100770 0x7fa968100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 -- 172.21.15.69:0/1832022956 >> 172.21.15.69:0/1832022956 conn(0x7fa9680fbce0 msgr2=0x7fa9680fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 -- 172.21.15.69:0/1832022956 shutdown_connections 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.494+0000 7fa96ef42640 1 -- 172.21.15.69:0/1832022956 wait complete. 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 Processor -- start 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 -- start start 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968101960 0x7fa9681980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 0x7fa96819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa96810d510 con 0x7fa968100770 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa96810d390 con 0x7fa968102320 2024-05-09T09:15:53.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ef42640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa96810d690 con 0x7fa968101960 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ccde640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ccde640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39612/0 (socket says 172.21.15.69:39612) 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ccde640 1 -- 172.21.15.69:0/3586858653 learned_addr learned my addr 172.21.15.69:0/3586858653 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96d4df640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 0x7fa96819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa967fff640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968101960 0x7fa9681980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.495+0000 7fa96ccde640 1 -- 172.21.15.69:0/3586858653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968101960 msgr2=0x7fa9681980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ccde640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968101960 0x7fa9681980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ccde640 1 -- 172.21.15.69:0/3586858653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 msgr2=0x7fa96819f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ccde640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 0x7fa96819f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ccde640 1 -- 172.21.15.69:0/3586858653 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa96819f7b0 con 0x7fa968100770 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96d4df640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 0x7fa96819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ccde640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa95800ebc0 tx=0x7fa95800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958003b70 con 0x7fa968100770 2024-05-09T09:15:53.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa958003d10 con 0x7fa968100770 2024-05-09T09:15:53.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa96819faa0 con 0x7fa968100770 2024-05-09T09:15:53.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa958019980 con 0x7fa968100770 2024-05-09T09:15:53.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.496+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa96819ffc0 con 0x7fa968100770 2024-05-09T09:15:53.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.497+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa968068650 con 0x7fa968100770 2024-05-09T09:15:53.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.498+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa958019b20 con 0x7fa968100770 2024-05-09T09:15:53.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.499+0000 7fa965ffb640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 0x7fa9540a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:53.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.499+0000 7fa967fff640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 0x7fa9540a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:53.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.499+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa9580cf510 con 0x7fa968100770 2024-05-09T09:15:53.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.499+0000 7fa967fff640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 0x7fa9540a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa950000c00 tx=0x7fa9500315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:53.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.503+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa95809c910 con 0x7fa968100770 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='client.24406 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: Saving service mds.all spec with placement smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l;count:12 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1611823529' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:15:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.719+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa968107a90 con 0x7fa968100770 2024-05-09T09:15:53.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.719+0000 7fa965ffb640 1 -- 172.21.15.69:0/3586858653 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) ==== 93+0+569 (secure 0 0 0) 0x7fa968107a90 con 0x7fa968100770 2024-05-09T09:15:53.719 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:53.719 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-05-09T09:15:53.719 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 1 2024-05-09T09:15:53.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 msgr2=0x7fa9540a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 0x7fa9540a5050 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fa950000c00 tx=0x7fa9500315d0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 msgr2=0x7fa968197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:53.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fa95800ebc0 tx=0x7fa95800c6a0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 shutdown_connections 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9540a2b90 0x7fa9540a5050 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa968102320 0x7fa96819f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa968101960 0x7fa9681980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 --2- 172.21.15.69:0/3586858653 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa968100770 0x7fa968197b90 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.721+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 >> 172.21.15.69:0/3586858653 conn(0x7fa9680fbce0 msgr2=0x7fa968104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.722+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 shutdown_connections 2024-05-09T09:15:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:53.722+0000 7fa96ef42640 1 -- 172.21.15.69:0/3586858653 wait complete. 2024-05-09T09:15:54.354 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:54 smithi163 ceph-mon[34354]: Deploying daemon mds.l on smithi163 2024-05-09T09:15:54.355 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:54 smithi163 ceph-mon[34354]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:54.355 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/745361462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:54.355 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3586858653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:54 smithi069 ceph-mon[30123]: Deploying daemon mds.l on smithi163 2024-05-09T09:15:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:54 smithi069 ceph-mon[30123]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/745361462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3586858653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:54 smithi103 ceph-mon[34421]: Deploying daemon mds.l on smithi163 2024-05-09T09:15:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:54 smithi103 ceph-mon[34421]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/745361462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:15:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3586858653' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:54.752 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:54 smithi163 systemd[1]: Starting Ceph mds.l for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:15:55.025 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:54 smithi163 podman[64891]: 2024-05-09 09:15:54.656937349 +0000 UTC m=+0.018180919 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:15:55.025 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:54 smithi163 podman[64891]: 2024-05-09 09:15:54.790706586 +0000 UTC m=+0.151950142 container create 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T09:15:55.025 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 podman[64891]: 2024-05-09 09:15:55.026522373 +0000 UTC m=+0.387765931 container init 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T09:15:55.305 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 podman[64891]: 2024-05-09 09:15:55.029160948 +0000 UTC m=+0.390404514 container start 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:15:55.305 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 ceph-mds[64910]: -- 172.21.15.163:0/1479341625 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5629c0f07a40 con 0x5629c0eda800 2024-05-09T09:15:55.305 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l[64906]: starting mds.l at 2024-05-09T09:15:55.305 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 bash[64891]: 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa 2024-05-09T09:15:55.305 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:15:55 smithi163 systemd[1]: Started Ceph mds.l for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:15:55.733 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:15:55.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 -- 172.21.15.69:0/183397798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0100780 msgr2=0x7ff4d0100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 --2- 172.21.15.69:0/183397798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0100780 0x7ff4d0100ba0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8009900 tx=0x7ff4c802a410 comp rx=0 tx=0).stop 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 -- 172.21.15.69:0/183397798 shutdown_connections 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 --2- 172.21.15.69:0/183397798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0102330 0x7ff4d0108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 --2- 172.21.15.69:0/183397798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 0x7ff4d0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 --2- 172.21.15.69:0/183397798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0100780 0x7ff4d0100ba0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 -- 172.21.15.69:0/183397798 >> 172.21.15.69:0/183397798 conn(0x7ff4d00fbd10 msgr2=0x7ff4d00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 -- 172.21.15.69:0/183397798 shutdown_connections 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.856+0000 7ff4d8589640 1 -- 172.21.15.69:0/183397798 wait complete. 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 Processor -- start 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 -- start start 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0100780 0x7ff4d0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 0x7ff4d0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:55.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d510 con 0x7ff4d0102330 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d390 con 0x7ff4d0100780 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d8589640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff4d010d690 con 0x7ff4d0101970 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39622/0 (socket says 172.21.15.69:39622) 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 -- 172.21.15.69:0/3431872808 learned_addr learned my addr 172.21.15.69:0/3431872808 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d5b24640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 0x7ff4d0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6325640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0100780 0x7ff4d0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 -- 172.21.15.69:0/3431872808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 msgr2=0x7ff4d0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 0x7ff4d0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 -- 172.21.15.69:0/3431872808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0100780 msgr2=0x7ff4d0197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0100780 0x7ff4d0197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.857+0000 7ff4d6b26640 1 -- 172.21.15.69:0/3431872808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff4d019f7e0 con 0x7ff4d0102330 2024-05-09T09:15:55.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4d6b26640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000eb10 tx=0x7ff4c000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0004b20 con 0x7ff4d0102330 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff4d019fad0 con 0x7ff4d0102330 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff4c0004cc0 con 0x7ff4d0102330 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff4d019ffd0 con 0x7ff4d0102330 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.858+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff4c0010700 con 0x7ff4d0102330 2024-05-09T09:15:55.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.859+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff4d0107b20 con 0x7ff4d0102330 2024-05-09T09:15:55.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.860+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff4c0005000 con 0x7ff4d0102330 2024-05-09T09:15:55.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.861+0000 7ff4cf7fe640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 0x7ff4b80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:55.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.861+0000 7ff4d6325640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 0x7ff4b80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:55.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.861+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7ff4c00cf260 con 0x7ff4d0102330 2024-05-09T09:15:55.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.861+0000 7ff4d6325640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 0x7ff4b80a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8005ab0 tx=0x7ff4c8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:55.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:55.865+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff4c009c660 con 0x7ff4d0102330 2024-05-09T09:15:56.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.081+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7ff4d002d370 con 0x7ff4d0102330 2024-05-09T09:15:56.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.081+0000 7ff4cf7fe640 1 -- 172.21.15.69:0/3431872808 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) ==== 93+0+1396 (secure 0 0 0) 0x7ff4c009bf60 con 0x7ff4d0102330 2024-05-09T09:15:56.081 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:56.081 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2}],"filesystems":[]} 2024-05-09T09:15:56.081 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 2 2024-05-09T09:15:56.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 msgr2=0x7ff4b80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:56.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 0x7ff4b80a5050 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7ff4c8005ab0 tx=0x7ff4c8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 msgr2=0x7ff4d019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7ff4c000eb10 tx=0x7ff4c000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 shutdown_connections 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff4b80a2b90 0x7ff4b80a5050 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff4d0102330 0x7ff4d019f170 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff4d0101970 0x7ff4d0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 --2- 172.21.15.69:0/3431872808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff4d0100780 0x7ff4d0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 >> 172.21.15.69:0/3431872808 conn(0x7ff4d00fbd10 msgr2=0x7ff4d0104cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 shutdown_connections 2024-05-09T09:15:56.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:56.083+0000 7ff4d8589640 1 -- 172.21.15.69:0/3431872808 wait complete. 2024-05-09T09:15:56.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6832/3615156387,v1:172.21.15.163:6833/3615156387] up:boot 2024-05-09T09:15:56.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: fsmap 1 up:standby 2024-05-09T09:15:56.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:15:56.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:56.100 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:56.101 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:56 smithi103 ceph-mon[34421]: Deploying daemon mds.k on smithi103 2024-05-09T09:15:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6832/3615156387,v1:172.21.15.163:6833/3615156387] up:boot 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: fsmap 1 up:standby 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:56 smithi069 ceph-mon[30123]: Deploying daemon mds.k on smithi103 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6832/3615156387,v1:172.21.15.163:6833/3615156387] up:boot 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: fsmap 1 up:standby 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:56 smithi163 ceph-mon[34354]: Deploying daemon mds.k on smithi103 2024-05-09T09:15:57.304 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:57 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3431872808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:57.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:57 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3431872808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:57.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:57 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3431872808' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:57.617 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:57 smithi103 systemd[1]: Starting Ceph mds.k for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:15:57.986 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:57 smithi103 podman[64080]: 2024-05-09 09:15:57.610345928 +0000 UTC m=+0.018682265 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:15:57.986 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:57 smithi103 podman[64080]: 2024-05-09 09:15:57.734981711 +0000 UTC m=+0.143318047 container create e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T09:15:58.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 -- 172.21.15.69:0/63556892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc1075b0 msgr2=0x7fcfdc1099a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/63556892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc1099a0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fcfd4009ca0 tx=0x7fcfd402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 -- 172.21.15.69:0/63556892 shutdown_connections 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/63556892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc1099a0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/63556892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc104c80 0x7fcfdc107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/63556892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 -- 172.21.15.69:0/63556892 >> 172.21.15.69:0/63556892 conn(0x7fcfdc0fbce0 msgr2=0x7fcfdc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 -- 172.21.15.69:0/63556892 shutdown_connections 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.213+0000 7fcfe24f5640 1 -- 172.21.15.69:0/63556892 wait complete. 2024-05-09T09:15:58.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 Processor -- start 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 -- start start 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc10d560 con 0x7fcfdc104c80 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc10d3e0 con 0x7fcfdc102350 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe24f5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcfdc10d6e0 con 0x7fcfdc1075b0 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41750/0 (socket says 172.21.15.69:41750) 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 -- 172.21.15.69:0/695232793 learned_addr learned my addr 172.21.15.69:0/695232793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdbfff640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfe0a92640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:58.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 -- 172.21.15.69:0/695232793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc1075b0 msgr2=0x7fcfdc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 -- 172.21.15.69:0/695232793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 msgr2=0x7fcfdc197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.214+0000 7fcfdb7fe640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfdb7fe640 1 -- 172.21.15.69:0/695232793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfdc19f770 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfdbfff640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfdb7fe640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400d740 tx=0x7fcfc400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4017070 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfdc19fa60 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcfc400de50 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfc4004e00 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.215+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfdc19ffb0 con 0x7fcfdc104c80 2024-05-09T09:15:58.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.216+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcfdc0fd7d0 con 0x7fcfdc104c80 2024-05-09T09:15:58.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.217+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fcfc4004fa0 con 0x7fcfdc104c80 2024-05-09T09:15:58.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.218+0000 7fcfd97fa640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 0x7fcfc80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:15:58.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.218+0000 7fcfdbfff640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 0x7fcfc80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:15:58.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.218+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fcfc40ce650 con 0x7fcfdc104c80 2024-05-09T09:15:58.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.219+0000 7fcfdbfff640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 0x7fcfc80a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009c30 tx=0x7fcfcc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:15:58.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.223+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcfc409ba50 con 0x7fcfdc104c80 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:58 smithi103 ceph-mon[34421]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:57 smithi103 podman[64080]: 2024-05-09 09:15:57.987881195 +0000 UTC m=+0.396217530 container init e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:57 smithi103 podman[64080]: 2024-05-09 09:15:57.990541147 +0000 UTC m=+0.398877480 container start e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=) 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:58 smithi103 ceph-mds[64098]: -- 172.21.15.103:0/1411911548 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561294a13a40 con 0x5612949e6c00 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k[64094]: starting mds.k at 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:58 smithi103 bash[64080]: e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a 2024-05-09T09:15:58.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:15:58 smithi103 systemd[1]: Started Ceph mds.k for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:15:58.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.445+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcfdc02d3c0 con 0x7fcfdc104c80 2024-05-09T09:15:58.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.446+0000 7fcfd97fa640 1 -- 172.21.15.69:0/695232793 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fcfc409b350 con 0x7fcfdc104c80 2024-05-09T09:15:58.446 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:15:58.446 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T09:15:58.446 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 3 2024-05-09T09:15:58.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 msgr2=0x7fcfc80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 0x7fcfc80a5050 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7fcfcc009c30 tx=0x7fcfcc005f90 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 msgr2=0x7fcfdc198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fcfc400d740 tx=0x7fcfc400dc00 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 shutdown_connections 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcfc80a2b90 0x7fcfc80a5050 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfdc1075b0 0x7fcfdc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfdc104c80 0x7fcfdc198090 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 --2- 172.21.15.69:0/695232793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfdc102350 0x7fcfdc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 >> 172.21.15.69:0/695232793 conn(0x7fcfdc0fbce0 msgr2=0x7fcfdc102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 shutdown_connections 2024-05-09T09:15:58.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:15:58.448+0000 7fcfe24f5640 1 -- 172.21.15.69:0/695232793 wait complete. 2024-05-09T09:15:58.457 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:58 smithi069 ceph-mon[30123]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:58 smithi163 ceph-mon[34354]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6832/1248493104,v1:172.21.15.103:6833/1248493104] up:boot 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: fsmap 2 up:standby 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: Deploying daemon mds.j on smithi069 2024-05-09T09:15:59.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:15:59 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/695232793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6832/1248493104,v1:172.21.15.103:6833/1248493104] up:boot 2024-05-09T09:15:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: fsmap 2 up:standby 2024-05-09T09:15:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:15:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: Deploying daemon mds.j on smithi069 2024-05-09T09:15:59.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:15:59 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/695232793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6832/1248493104,v1:172.21.15.103:6833/1248493104] up:boot 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: fsmap 2 up:standby 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:15:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: Deploying daemon mds.j on smithi069 2024-05-09T09:15:59.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:15:59 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/695232793' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:00.332 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:00 smithi103 ceph-mon[34421]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:00.339 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:00 smithi069 ceph-mon[30123]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:00.460 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:00.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:00 smithi163 ceph-mon[34354]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:00.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- 172.21.15.69:0/2474261826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18102f20 msgr2=0x7fed18103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/2474261826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18102f20 0x7fed18103340 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7fed08009ca0 tx=0x7fed0802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- 172.21.15.69:0/2474261826 shutdown_connections 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/2474261826 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18100350 0x7fed18109f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/2474261826 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 0x7fed180ffe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/2474261826 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18102f20 0x7fed18103340 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- 172.21.15.69:0/2474261826 >> 172.21.15.69:0/2474261826 conn(0x7fed180fb510 msgr2=0x7fed180fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- 172.21.15.69:0/2474261826 shutdown_connections 2024-05-09T09:16:00.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- 172.21.15.69:0/2474261826 wait complete. 2024-05-09T09:16:00.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 Processor -- start 2024-05-09T09:16:00.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.570+0000 7fed1cbf9640 1 -- start start 2024-05-09T09:16:00.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 0x7fed18197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:00.574 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18102f20 0x7fed1819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed1810d5a0 con 0x7fed18100350 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed1810d420 con 0x7fed180ff990 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fed1810d720 con 0x7fed18102f20 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41762/0 (socket says 172.21.15.69:41762) 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 -- 172.21.15.69:0/3545767664 learned_addr learned my addr 172.21.15.69:0/3545767664 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1659c640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 0x7fed18197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed16d9d640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18102f20 0x7fed1819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 -- 172.21.15.69:0/3545767664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18102f20 msgr2=0x7fed1819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18102f20 0x7fed1819f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 -- 172.21.15.69:0/3545767664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 msgr2=0x7fed18197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 0x7fed18197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 -- 172.21.15.69:0/3545767664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed1819f840 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed15d9b640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d950 tx=0x7fed0000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed00017070 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed1819fb30 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed00009c30 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed0000f6b0 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.571+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed181a0080 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.572+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed18108e60 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.573+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fed000109d0 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.574+0000 7fed0f7fe640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 0x7fed040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.574+0000 7fed1659c640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 0x7fed040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.574+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fed000ce7f0 con 0x7fed18100350 2024-05-09T09:16:00.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.574+0000 7fed1659c640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 0x7fed040a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fed08031ae0 tx=0x7fed08000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:00.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.578+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed00009de0 con 0x7fed18100350 2024-05-09T09:16:00.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.799+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fed1802d410 con 0x7fed18100350 2024-05-09T09:16:00.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.800+0000 7fed0f7fe640 1 -- 172.21.15.69:0/3545767664 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) ==== 93+0+2224 (secure 0 0 0) 0x7fed0009b5a0 con 0x7fed18100350 2024-05-09T09:16:00.800 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:00.800 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T09:16:00.800 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 3 2024-05-09T09:16:00.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.801+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 msgr2=0x7fed040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:00.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.801+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 0x7fed040a5050 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7fed08031ae0 tx=0x7fed08000f40 comp rx=0 tx=0).stop 2024-05-09T09:16:00.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.801+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 msgr2=0x7fed18198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.801+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7fed0000d950 tx=0x7fed0000de10 comp rx=0 tx=0).stop 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 shutdown_connections 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed040a2b90 0x7fed040a5050 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed18102f20 0x7fed1819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed18100350 0x7fed18198170 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 --2- 172.21.15.69:0/3545767664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed180ff990 0x7fed18197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 >> 172.21.15.69:0/3545767664 conn(0x7fed180fb510 msgr2=0x7fed18106030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 shutdown_connections 2024-05-09T09:16:00.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:00.802+0000 7fed1cbf9640 1 -- 172.21.15.69:0/3545767664 wait complete. 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3545767664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:01.088 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3545767664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3545767664' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:02.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:02 smithi103 ceph-mon[34421]: Deploying daemon mds.d on smithi069 2024-05-09T09:16:02.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:02 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.69:6834/3886932870,v1:172.21.15.69:6835/3886932870] up:boot 2024-05-09T09:16:02.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:02 smithi103 ceph-mon[34421]: fsmap 3 up:standby 2024-05-09T09:16:02.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T09:16:02.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:02 smithi103 ceph-mon[34421]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:02.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:02 smithi069 ceph-mon[30123]: Deploying daemon mds.d on smithi069 2024-05-09T09:16:02.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:02 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.69:6834/3886932870,v1:172.21.15.69:6835/3886932870] up:boot 2024-05-09T09:16:02.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:02 smithi069 ceph-mon[30123]: fsmap 3 up:standby 2024-05-09T09:16:02.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T09:16:02.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:02 smithi069 ceph-mon[30123]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:02.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:02 smithi163 ceph-mon[34354]: Deploying daemon mds.d on smithi069 2024-05-09T09:16:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:02 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.69:6834/3886932870,v1:172.21.15.69:6835/3886932870] up:boot 2024-05-09T09:16:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:02 smithi163 ceph-mon[34354]: fsmap 3 up:standby 2024-05-09T09:16:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-05-09T09:16:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:02 smithi163 ceph-mon[34354]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:02.813 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:02.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.935+0000 7f23ec334640 1 -- 172.21.15.69:0/881255797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 msgr2=0x7f23e4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.935+0000 7f23ec334640 1 --2- 172.21.15.69:0/881255797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4100b90 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009900 tx=0x7f23e002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- 172.21.15.69:0/881255797 shutdown_connections 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- 172.21.15.69:0/881255797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4102320 0x7f23e4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- 172.21.15.69:0/881255797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4101960 0x7f23e4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- 172.21.15.69:0/881255797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4100b90 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- 172.21.15.69:0/881255797 >> 172.21.15.69:0/881255797 conn(0x7f23e40fbce0 msgr2=0x7f23e40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- 172.21.15.69:0/881255797 shutdown_connections 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- 172.21.15.69:0/881255797 wait complete. 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 Processor -- start 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- start start 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:02.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 0x7f23e4102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4102320 0x7f23e4103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c3e0 con 0x7f23e4100770 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c260 con 0x7f23e4101960 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ec334640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f23e410c560 con 0x7f23e4102320 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.936+0000 7f23ea0d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41802/0 (socket says 172.21.15.69:41802) 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 -- 172.21.15.69:0/298905125 learned_addr learned my addr 172.21.15.69:0/298905125 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea8d1640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4102320 0x7f23e4103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23e98cf640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 0x7f23e4102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 -- 172.21.15.69:0/298905125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4102320 msgr2=0x7f23e4103280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4102320 0x7f23e4103280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 -- 172.21.15.69:0/298905125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 msgr2=0x7f23e4102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 0x7f23e4102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:02.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 -- 172.21.15.69:0/298905125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23e4103b10 con 0x7f23e4100770 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23e98cf640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 0x7f23e4102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ea0d0640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009230 tx=0x7f23e0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e003f030 con 0x7f23e4100770 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23e0003ad0 con 0x7f23e4100770 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23e41a1cc0 con 0x7f23e4100770 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23e0035070 con 0x7f23e4100770 2024-05-09T09:16:02.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.937+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23e41a21c0 con 0x7f23e4100770 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.938+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f23e41a2390 con 0x7f23e4100770 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.939+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f23e0042030 con 0x7f23e4100770 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.940+0000 7f23db7fe640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 0x7f23d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.940+0000 7f23e98cf640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 0x7f23d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.940+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f23e00ef6c0 con 0x7f23e4100770 2024-05-09T09:16:02.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.941+0000 7f23e98cf640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 0x7f23d00a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23e4104470 tx=0x7f23d4005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:02.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:02.948+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f23e0049030 con 0x7f23e4100770 2024-05-09T09:16:03.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.182+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f23e41a2560 con 0x7f23e4100770 2024-05-09T09:16:03.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.182+0000 7f23db7fe640 1 -- 172.21.15.69:0/298905125 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) ==== 93+0+3049 (secure 0 0 0) 0x7f23e00bcaa0 con 0x7f23e4100770 2024-05-09T09:16:03.182 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:03.182 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T09:16:03.182 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 4 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 msgr2=0x7f23d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 0x7f23d00a5050 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f23e4104470 tx=0x7f23d4005d30 comp rx=0 tx=0).stop 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 msgr2=0x7f23e4104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009230 tx=0x7f23e0031b60 comp rx=0 tx=0).stop 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 shutdown_connections 2024-05-09T09:16:03.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23d00a2b90 0x7f23d00a5050 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23e4102320 0x7f23e4103280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23e4101960 0x7f23e4102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 --2- 172.21.15.69:0/298905125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f23e4100770 0x7f23e4104700 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 >> 172.21.15.69:0/298905125 conn(0x7f23e40fbce0 msgr2=0x7f23e40fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 shutdown_connections 2024-05-09T09:16:03.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:03.184+0000 7f23ec334640 1 -- 172.21.15.69:0/298905125 wait complete. 2024-05-09T09:16:03.501 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 systemd[1]: Starting Ceph mds.d for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:03.760 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 podman[70232]: 2024-05-09 09:16:03.405369934 +0000 UTC m=+0.017520202 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:03.760 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 podman[70232]: 2024-05-09 09:16:03.521205416 +0000 UTC m=+0.133355683 container create eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:16:04.100 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 podman[70232]: 2024-05-09 09:16:03.782412851 +0000 UTC m=+0.394563109 container init eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc) 2024-05-09T09:16:04.100 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 podman[70232]: 2024-05-09 09:16:03.785545504 +0000 UTC m=+0.397695771 container start eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T09:16:04.100 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 ceph-mds[70251]: -- 172.21.15.69:0/2199361539 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55d6c05f1a40 con 0x55d6c05c4800 2024-05-09T09:16:04.101 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d[70247]: starting mds.d at 2024-05-09T09:16:04.101 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 bash[70232]: eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 2024-05-09T09:16:04.101 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:16:03 smithi069 systemd[1]: Started Ceph mds.d for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/298905125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:04.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:04.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/298905125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/298905125' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:05.196 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 -- 172.21.15.69:0/3298509002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0100770 msgr2=0x7f46d0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 --2- 172.21.15.69:0/3298509002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0100770 0x7f46d0100b90 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f46c0009ca0 tx=0x7f46c002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 -- 172.21.15.69:0/3298509002 shutdown_connections 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 --2- 172.21.15.69:0/3298509002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0102320 0x7f46d0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 --2- 172.21.15.69:0/3298509002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0101960 0x7f46d0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 --2- 172.21.15.69:0/3298509002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0100770 0x7f46d0100b90 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 -- 172.21.15.69:0/3298509002 >> 172.21.15.69:0/3298509002 conn(0x7f46d00fbce0 msgr2=0x7f46d00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 -- 172.21.15.69:0/3298509002 shutdown_connections 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 -- 172.21.15.69:0/3298509002 wait complete. 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.314+0000 7f46d5615640 1 Processor -- start 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 -- start start 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0100770 0x7f46d0069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0102320 0x7f46d019ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010c3f0 con 0x7f46d0101960 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010c270 con 0x7f46d0102320 2024-05-09T09:16:05.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46d5615640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f46d010c570 con 0x7f46d0100770 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41838/0 (socket says 172.21.15.69:41838) 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 -- 172.21.15.69:0/1949418993 learned_addr learned my addr 172.21.15.69:0/1949418993 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ceffd640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0100770 0x7f46d0069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46cf7fe640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0102320 0x7f46d019ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 -- 172.21.15.69:0/1949418993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0100770 msgr2=0x7f46d0069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0100770 0x7f46d0069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 -- 172.21.15.69:0/1949418993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0102320 msgr2=0x7f46d019ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0102320 0x7f46d019ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 -- 172.21.15.69:0/1949418993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f46d019f780 con 0x7f46d0101960 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46ce7fc640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c920 tx=0x7f46b800cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:05.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.315+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8007c70 con 0x7f46d0101960 2024-05-09T09:16:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.316+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f46d019fa10 con 0x7f46d0101960 2024-05-09T09:16:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.316+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f46b8007e10 con 0x7f46d0101960 2024-05-09T09:16:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.316+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f46b8018890 con 0x7f46d0101960 2024-05-09T09:16:05.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.316+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f46d019ff60 con 0x7f46d0101960 2024-05-09T09:16:05.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.317+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f46d0107a30 con 0x7f46d0101960 2024-05-09T09:16:05.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.318+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f46b8018a70 con 0x7f46d0101960 2024-05-09T09:16:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.318+0000 7f46c7fff640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 0x7f46bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.319+0000 7f46ceffd640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 0x7f46bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.319+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f46b80ceda0 con 0x7f46d0101960 2024-05-09T09:16:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.319+0000 7f46ceffd640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 0x7f46bc0a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006010 tx=0x7f46c0009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:05.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.323+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f46b8014030 con 0x7f46d0101960 2024-05-09T09:16:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:05 smithi103 ceph-mon[34421]: Deploying daemon mds.e on smithi103 2024-05-09T09:16:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:05 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.69:6836/4268769273,v1:172.21.15.69:6837/4268769273] up:boot 2024-05-09T09:16:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:05 smithi103 ceph-mon[34421]: fsmap 4 up:standby 2024-05-09T09:16:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:16:05.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:05 smithi069 ceph-mon[30123]: Deploying daemon mds.e on smithi103 2024-05-09T09:16:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:05 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.69:6836/4268769273,v1:172.21.15.69:6837/4268769273] up:boot 2024-05-09T09:16:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:05 smithi069 ceph-mon[30123]: fsmap 4 up:standby 2024-05-09T09:16:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:16:05.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.539+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f46d0062430 con 0x7f46d0101960 2024-05-09T09:16:05.540 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.540+0000 7f46c7fff640 1 -- 172.21.15.69:0/1949418993 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) ==== 93+0+3874 (secure 0 0 0) 0x7f46b800de20 con 0x7f46d0101960 2024-05-09T09:16:05.540 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:05.540 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[]} 2024-05-09T09:16:05.541 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 5 2024-05-09T09:16:05.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 msgr2=0x7f46bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 0x7f46bc0a5050 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f46c0006010 tx=0x7f46c0009360 comp rx=0 tx=0).stop 2024-05-09T09:16:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 msgr2=0x7f46d019c850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f46b800c920 tx=0x7f46b800cde0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 shutdown_connections 2024-05-09T09:16:05.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f46bc0a2b90 0x7f46bc0a5050 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f46d0102320 0x7f46d019ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f46d0101960 0x7f46d019c850 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 --2- 172.21.15.69:0/1949418993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f46d0100770 0x7f46d0069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 >> 172.21.15.69:0/1949418993 conn(0x7f46d00fbce0 msgr2=0x7f46d0104c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 shutdown_connections 2024-05-09T09:16:05.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:05.542+0000 7f46d5615640 1 -- 172.21.15.69:0/1949418993 wait complete. 2024-05-09T09:16:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:05 smithi163 ceph-mon[34354]: Deploying daemon mds.e on smithi103 2024-05-09T09:16:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:05 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.69:6836/4268769273,v1:172.21.15.69:6837/4268769273] up:boot 2024-05-09T09:16:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:05 smithi163 ceph-mon[34354]: fsmap 4 up:standby 2024-05-09T09:16:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-05-09T09:16:06.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:06 smithi103 ceph-mon[34421]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:06.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:06 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1949418993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:06.367 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 systemd[1]: Starting Ceph mds.e for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:06 smithi069 ceph-mon[30123]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:06 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1949418993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:06 smithi163 ceph-mon[34354]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:06.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:06 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1949418993' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:06.671 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 podman[64399]: 2024-05-09 09:16:06.306266142 +0000 UTC m=+0.018416898 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:06.671 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 podman[64399]: 2024-05-09 09:16:06.418369491 +0000 UTC m=+0.130520242 container create 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:16:06.671 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 podman[64399]: 2024-05-09 09:16:06.646498764 +0000 UTC m=+0.358649515 container init 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:16:06.671 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 podman[64399]: 2024-05-09 09:16:06.649135112 +0000 UTC m=+0.361285863 container start 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e, RELEASE=squid-96805dc, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507) 2024-05-09T09:16:06.672 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 ceph-mds[64418]: -- 172.21.15.103:0/3307435781 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55aadb8e3a40 con 0x55aadb8b6800 2024-05-09T09:16:07.117 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: starting mds.e at 2024-05-09T09:16:07.117 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 bash[64399]: 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c 2024-05-09T09:16:07.117 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:16:06 smithi103 systemd[1]: Started Ceph mds.e for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:07.554 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 -- 172.21.15.69:0/327270835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 msgr2=0x7f6600109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 --2- 172.21.15.69:0/327270835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f6600109cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f65f8009900 tx=0x7f65f802a410 comp rx=0 tx=0).stop 2024-05-09T09:16:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 -- 172.21.15.69:0/327270835 shutdown_connections 2024-05-09T09:16:07.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 --2- 172.21.15.69:0/327270835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f6600109cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 --2- 172.21.15.69:0/327270835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600102a70 0x7f6600102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 --2- 172.21.15.69:0/327270835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600101880 0x7f6600101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 -- 172.21.15.69:0/327270835 >> 172.21.15.69:0/327270835 conn(0x7f66000fce10 msgr2=0x7f66000ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 -- 172.21.15.69:0/327270835 shutdown_connections 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.667+0000 7f66087b6640 1 -- 172.21.15.69:0/327270835 wait complete. 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 Processor -- start 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 -- start start 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 0x7f6600197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600102a70 0x7f6600198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:07.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f660010d4e0 con 0x7f6600103430 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f660010d360 con 0x7f6600101880 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f66087b6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f660010d660 con 0x7f6600102a70 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57390/0 (socket says 172.21.15.69:57390) 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 -- 172.21.15.69:0/2510750559 learned_addr learned my addr 172.21.15.69:0/2510750559 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606552640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 0x7f6600197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6605d51640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600102a70 0x7f6600198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 -- 172.21.15.69:0/2510750559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600102a70 msgr2=0x7f6600198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600102a70 0x7f6600198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 -- 172.21.15.69:0/2510750559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 msgr2=0x7f6600197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.668+0000 7f6606d53640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 0x7f6600197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f6606d53640 1 -- 172.21.15.69:0/2510750559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f660019f710 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f6606552640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 0x7f6600197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f6606d53640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f65f80314e0 tx=0x7f65f8004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f803f030 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f660019f9a0 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65f8003ce0 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65f8035070 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.669+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f660019fee0 con 0x7f6600103430 2024-05-09T09:16:07.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.670+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6600108b40 con 0x7f6600103430 2024-05-09T09:16:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.671+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f65f8042030 con 0x7f6600103430 2024-05-09T09:16:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.672+0000 7f65ff7fe640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 0x7f65e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:07.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.672+0000 7f6606552640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 0x7f65e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.672+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f65f80ef6f0 con 0x7f6600103430 2024-05-09T09:16:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.672+0000 7f6606552640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 0x7f65e80a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f66001028d0 tx=0x7f65f0009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:07.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.675+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65f80bcc10 con 0x7f6600103430 2024-05-09T09:16:07.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.890+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f660002d380 con 0x7f6600103430 2024-05-09T09:16:07.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.890+0000 7f65ff7fe640 1 -- 172.21.15.69:0/2510750559 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) ==== 93+0+4702 (secure 0 0 0) 0x7f65f80bc510 con 0x7f6600103430 2024-05-09T09:16:07.890 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:07.891 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6}],"filesystems":[]} 2024-05-09T09:16:07.891 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 6 2024-05-09T09:16:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 msgr2=0x7f65e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 0x7f65e80a5050 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f66001028d0 tx=0x7f65f0009540 comp rx=0 tx=0).stop 2024-05-09T09:16:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 msgr2=0x7f660019f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f65f80314e0 tx=0x7f65f8004360 comp rx=0 tx=0).stop 2024-05-09T09:16:07.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 shutdown_connections 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2b90 0x7f65e80a5050 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6600103430 0x7f660019f0a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6600102a70 0x7f6600198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 --2- 172.21.15.69:0/2510750559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6600101880 0x7f6600197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 >> 172.21.15.69:0/2510750559 conn(0x7f66000fce10 msgr2=0x7f6600105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 shutdown_connections 2024-05-09T09:16:07.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:07.892+0000 7f66087b6640 1 -- 172.21.15.69:0/2510750559 wait complete. 2024-05-09T09:16:07.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:07.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:07.902 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: Deploying daemon mds.f on smithi163 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6834/3790151668,v1:172.21.15.103:6835/3790151668] up:boot 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: fsmap 5 up:standby 2024-05-09T09:16:07.903 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: Deploying daemon mds.f on smithi163 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6834/3790151668,v1:172.21.15.103:6835/3790151668] up:boot 2024-05-09T09:16:08.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: fsmap 5 up:standby 2024-05-09T09:16:08.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: Deploying daemon mds.f on smithi163 2024-05-09T09:16:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:08.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6834/3790151668,v1:172.21.15.103:6835/3790151668] up:boot 2024-05-09T09:16:08.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: fsmap 5 up:standby 2024-05-09T09:16:08.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:16:09.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:08 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2510750559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:09.055 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:08 smithi163 systemd[1]: Starting Ceph mds.f for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:09.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:08 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2510750559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:09.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:08 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2510750559' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:09.395 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 podman[65199]: 2024-05-09 09:16:09.03155827 +0000 UTC m=+0.017654429 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:09.396 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 podman[65199]: 2024-05-09 09:16:09.152868715 +0000 UTC m=+0.138964858 container create aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:16:09.396 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 podman[65199]: 2024-05-09 09:16:09.397186995 +0000 UTC m=+0.383283154 container init aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD) 2024-05-09T09:16:09.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 podman[65199]: 2024-05-09 09:16:09.39984338 +0000 UTC m=+0.385939524 container start aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T09:16:09.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mds[65218]: -- 172.21.15.163:0/151649889 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55ee10123a40 con 0x55ee100f6c00 2024-05-09T09:16:09.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: starting mds.f at 2024-05-09T09:16:09.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 bash[65199]: aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 2024-05-09T09:16:09.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:16:09 smithi163 systemd[1]: Started Ceph mds.f for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:09.905 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 -- 172.21.15.69:0/185079219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904102a90 msgr2=0x7f6904102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 --2- 172.21.15.69:0/185079219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904102a90 0x7f6904102f10 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f68f8009900 tx=0x7f68f802a410 comp rx=0 tx=0).stop 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 -- 172.21.15.69:0/185079219 shutdown_connections 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 --2- 172.21.15.69:0/185079219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6904103450 0x7f6904109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 --2- 172.21.15.69:0/185079219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904102a90 0x7f6904102f10 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 --2- 172.21.15.69:0/185079219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f69041018a0 0x7f6904101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 -- 172.21.15.69:0/185079219 >> 172.21.15.69:0/185079219 conn(0x7f69040fce10 msgr2=0x7f69040ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:10.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 -- 172.21.15.69:0/185079219 shutdown_connections 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.018+0000 7f690b658640 1 -- 172.21.15.69:0/185079219 wait complete. 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 Processor -- start 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 -- start start 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69041018a0 0x7f69040755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6904102a90 0x7f6904073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d500 con 0x7f6904103450 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d380 con 0x7f69041018a0 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f690b658640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f690410d680 con 0x7f6904102a90 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57408/0 (socket says 172.21.15.69:57408) 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 -- 172.21.15.69:0/699275670 learned_addr learned my addr 172.21.15.69:0/699275670 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:10.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f69093f4640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69041018a0 0x7f69040755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6908bf3640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6904102a90 0x7f6904073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 -- 172.21.15.69:0/699275670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6904102a90 msgr2=0x7f6904073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6904102a90 0x7f6904073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 -- 172.21.15.69:0/699275670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69041018a0 msgr2=0x7f69040755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69041018a0 0x7f69040755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 -- 172.21.15.69:0/699275670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f69040749f0 con 0x7f6904103450 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.019+0000 7f6909bf5640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009fd0 tx=0x7f68fc00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.020+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc017070 con 0x7f6904103450 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.020+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f68fc003e50 con 0x7f6904103450 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.020+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f69041a1d20 con 0x7f6904103450 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.020+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f68fc0106c0 con 0x7f6904103450 2024-05-09T09:16:10.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.020+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f69041a2190 con 0x7f6904103450 2024-05-09T09:16:10.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.021+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6904108c40 con 0x7f6904103450 2024-05-09T09:16:10.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.022+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f68fc003970 con 0x7f6904103450 2024-05-09T09:16:10.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.023+0000 7f68f67fc640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:10.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.023+0000 7f69093f4640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:10.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.023+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f68fc0ce800 con 0x7f6904103450 2024-05-09T09:16:10.024 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.023+0000 7f69093f4640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 0x7f68f00a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69041028f0 tx=0x7f6900005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:10.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.027+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f68fc09b5b0 con 0x7f6904103450 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:10.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:09 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:10.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:10.227 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:10.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:10.228 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:09 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:10.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.240+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f690402d3e0 con 0x7f6904103450 2024-05-09T09:16:10.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.241+0000 7f68f67fc640 1 -- 172.21.15.69:0/699275670 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f68fc09f8b0 con 0x7f6904103450 2024-05-09T09:16:10.241 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:10.241 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T09:16:10.241 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 msgr2=0x7f68f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 0x7f68f00a5050 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f69041028f0 tx=0x7f6900005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 msgr2=0x7f6904074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009fd0 tx=0x7f68fc00b260 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 shutdown_connections 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f68f00a2b90 0x7f68f00a5050 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6904103450 0x7f6904074160 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6904102a90 0x7f6904073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 --2- 172.21.15.69:0/699275670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f69041018a0 0x7f69040755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:10.243 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 >> 172.21.15.69:0/699275670 conn(0x7f69040fce10 msgr2=0x7f6904105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 shutdown_connections 2024-05-09T09:16:10.244 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:10.243+0000 7f690b658640 1 -- 172.21.15.69:0/699275670 wait complete. 2024-05-09T09:16:10.252 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:10.253 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:09 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:10 smithi103 ceph-mon[34421]: Deploying daemon mds.c on smithi163 2024-05-09T09:16:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:10 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6834/989258568,v1:172.21.15.163:6835/989258568] up:boot 2024-05-09T09:16:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:10 smithi103 ceph-mon[34421]: fsmap 6 up:standby 2024-05-09T09:16:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:16:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:10 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/699275670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:11.220 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:10 smithi163 ceph-mon[34354]: Deploying daemon mds.c on smithi163 2024-05-09T09:16:11.221 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:10 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6834/989258568,v1:172.21.15.163:6835/989258568] up:boot 2024-05-09T09:16:11.221 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:10 smithi163 ceph-mon[34354]: fsmap 6 up:standby 2024-05-09T09:16:11.221 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:16:11.221 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:10 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/699275670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:10 smithi069 ceph-mon[30123]: Deploying daemon mds.c on smithi163 2024-05-09T09:16:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:10 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6834/989258568,v1:172.21.15.163:6835/989258568] up:boot 2024-05-09T09:16:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:10 smithi069 ceph-mon[30123]: fsmap 6 up:standby 2024-05-09T09:16:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:16:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:10 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/699275670' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:11.805 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:11 smithi163 systemd[1]: Starting Ceph mds.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:12.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:11 smithi103 ceph-mon[34421]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:12.209 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:11 smithi163 ceph-mon[34354]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:12.209 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:11 smithi163 podman[65522]: 2024-05-09 09:16:11.823197294 +0000 UTC m=+0.018159173 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:12.209 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:11 smithi163 podman[65522]: 2024-05-09 09:16:11.932756082 +0000 UTC m=+0.127717950 container create 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0) 2024-05-09T09:16:12.209 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 podman[65522]: 2024-05-09 09:16:12.210642884 +0000 UTC m=+0.405604757 container init 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:16:12.255 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:12.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:11 smithi069 ceph-mon[30123]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 -- 172.21.15.69:0/2874656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808100770 msgr2=0x7f5808100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 --2- 172.21.15.69:0/2874656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808100770 0x7f5808100b90 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f5804009900 tx=0x7f580402a410 comp rx=0 tx=0).stop 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 -- 172.21.15.69:0/2874656361 shutdown_connections 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 --2- 172.21.15.69:0/2874656361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 0x7f5808108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 --2- 172.21.15.69:0/2874656361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808101960 0x7f5808101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 --2- 172.21.15.69:0/2874656361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808100770 0x7f5808100b90 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.371+0000 7f58107db640 1 -- 172.21.15.69:0/2874656361 >> 172.21.15.69:0/2874656361 conn(0x7f58080fbce0 msgr2=0x7f58080fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- 172.21.15.69:0/2874656361 shutdown_connections 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- 172.21.15.69:0/2874656361 wait complete. 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 Processor -- start 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- start start 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808100770 0x7f58081969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 0x7f580819df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:12.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f580810c3f0 con 0x7f5808101960 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f580810c270 con 0x7f5808100770 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f58107db640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f580810c570 con 0x7f5808102320 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f580dd76640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f580dd76640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57412/0 (socket says 172.21.15.69:57412) 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f580dd76640 1 -- 172.21.15.69:0/493746802 learned_addr learned my addr 172.21.15.69:0/493746802 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:12.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f580ed78640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 0x7f580819df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.372+0000 7f580e577640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808100770 0x7f58081969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 -- 172.21.15.69:0/493746802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 msgr2=0x7f580819df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 0x7f580819df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 -- 172.21.15.69:0/493746802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808100770 msgr2=0x7f58081969d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808100770 0x7f58081969d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 -- 172.21.15.69:0/493746802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f580819e5f0 con 0x7f5808101960 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f580dd76640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f57f800cce0 tx=0x7f57f800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f800f650 con 0x7f5808101960 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57f800fcd0 con 0x7f5808101960 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57f8018c70 con 0x7f5808101960 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f580819e8e0 con 0x7f5808101960 2024-05-09T09:16:12.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.373+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f580819ee30 con 0x7f5808101960 2024-05-09T09:16:12.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.374+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f58081079c0 con 0x7f5808101960 2024-05-09T09:16:12.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.375+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f57f800f7f0 con 0x7f5808101960 2024-05-09T09:16:12.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.376+0000 7f57ff7fe640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 0x7f57f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:12.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.376+0000 7f580e577640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 0x7f57f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:12.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.376+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f57f80cefa0 con 0x7f5808101960 2024-05-09T09:16:12.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.376+0000 7f580e577640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 0x7f57f00a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5804005ab0 tx=0x7f5804000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:12.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.380+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f57f8014030 con 0x7f5808101960 2024-05-09T09:16:12.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 podman[65522]: 2024-05-09 09:16:12.213331966 +0000 UTC m=+0.408293836 container start 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD) 2024-05-09T09:16:12.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/1386002813 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x564d0dc89a40 con 0x564d0dc5c800 2024-05-09T09:16:12.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:16:12.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 bash[65522]: 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 2024-05-09T09:16:12.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:16:12 smithi163 systemd[1]: Started Ceph mds.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:12.607 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.608+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f580802d330 con 0x7f5808101960 2024-05-09T09:16:12.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.608+0000 7f57ff7fe640 1 -- 172.21.15.69:0/493746802 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) ==== 93+0+5527 (secure 0 0 0) 0x7f57f809c3a0 con 0x7f5808101960 2024-05-09T09:16:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:12.608 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7}],"filesystems":[]} 2024-05-09T09:16:12.609 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 7 2024-05-09T09:16:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 msgr2=0x7f57f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 0x7f57f00a5050 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5804005ab0 tx=0x7f5804000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 msgr2=0x7f5808196f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f57f800cce0 tx=0x7f57f800a620 comp rx=0 tx=0).stop 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 shutdown_connections 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57f00a2b90 0x7f57f00a5050 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5808102320 0x7f580819df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5808101960 0x7f5808196f10 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 --2- 172.21.15.69:0/493746802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5808100770 0x7f58081969d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.610+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 >> 172.21.15.69:0/493746802 conn(0x7f58080fbce0 msgr2=0x7f5808104b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.611+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 shutdown_connections 2024-05-09T09:16:12.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:12.611+0000 7f58107db640 1 -- 172.21.15.69:0/493746802 wait complete. 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: Deploying daemon mds.h on smithi103 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/493746802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/3224689113,v1:172.21.15.163:6837/3224689113] up:boot 2024-05-09T09:16:13.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: fsmap 7 up:standby 2024-05-09T09:16:13.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:16:13.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:13 smithi103 ceph-mon[34421]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: Deploying daemon mds.h on smithi103 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/493746802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/3224689113,v1:172.21.15.163:6837/3224689113] up:boot 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: fsmap 7 up:standby 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:16:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:13 smithi069 ceph-mon[30123]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: Deploying daemon mds.h on smithi103 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/493746802' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/3224689113,v1:172.21.15.163:6837/3224689113] up:boot 2024-05-09T09:16:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: fsmap 7 up:standby 2024-05-09T09:16:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:16:13.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:13 smithi163 ceph-mon[34354]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:14.622 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 -- 172.21.15.69:0/549886807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec100780 msgr2=0x7f4bec100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/549886807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec100780 0x7f4bec100ba0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc009900 tx=0x7f4bdc02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 -- 172.21.15.69:0/549886807 shutdown_connections 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/549886807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 0x7f4bec108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/549886807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec101970 0x7f4bec101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/549886807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec100780 0x7f4bec100ba0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 -- 172.21.15.69:0/549886807 >> 172.21.15.69:0/549886807 conn(0x7f4bec0fbd10 msgr2=0x7f4bec0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 -- 172.21.15.69:0/549886807 shutdown_connections 2024-05-09T09:16:14.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.735+0000 7f4bf43f9640 1 -- 172.21.15.69:0/549886807 wait complete. 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 Processor -- start 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 -- start start 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec100780 0x7f4bec104710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 0x7f4bec103290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10c3d0 con 0x7f4bec101970 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10c250 con 0x7f4bec102330 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf43f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bec10c550 con 0x7f4bec100780 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57436/0 (socket says 172.21.15.69:57436) 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 -- 172.21.15.69:0/897604857 learned_addr learned my addr 172.21.15.69:0/897604857 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf2996640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 0x7f4bec103290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf2195640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec100780 0x7f4bec104710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:14.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 -- 172.21.15.69:0/897604857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec100780 msgr2=0x7f4bec104710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec100780 0x7f4bec104710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 -- 172.21.15.69:0/897604857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 msgr2=0x7f4bec103290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 0x7f4bec103290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 -- 172.21.15.69:0/897604857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bec103b20 con 0x7f4bec101970 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.736+0000 7f4bf1994640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d950 tx=0x7f4be800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.737+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be8017070 con 0x7f4bec101970 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.737+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bec1a1cc0 con 0x7f4bec101970 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.737+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4be8009c30 con 0x7f4bec101970 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.737+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4be800f6b0 con 0x7f4bec101970 2024-05-09T09:16:14.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.737+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bec1a21b0 con 0x7f4bec101970 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.738+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bec1a2440 con 0x7f4bec101970 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.739+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4be80109d0 con 0x7f4bec101970 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.739+0000 7f4be37fe640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 0x7f4bd80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.739+0000 7f4bf2195640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 0x7f4bd80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.740+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f4be80ce700 con 0x7f4bec101970 2024-05-09T09:16:14.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.740+0000 7f4bf2195640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 0x7f4bd80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc005ab0 tx=0x7f4bdc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:14.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.744+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4be8009de0 con 0x7f4bec101970 2024-05-09T09:16:14.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.955+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f4bec1a27d0 con 0x7f4bec101970 2024-05-09T09:16:14.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.956+0000 7f4be37fe640 1 -- 172.21.15.69:0/897604857 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) ==== 93+0+6355 (secure 0 0 0) 0x7f4be809b4b0 con 0x7f4bec101970 2024-05-09T09:16:14.956 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:14.956 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8}],"filesystems":[]} 2024-05-09T09:16:14.956 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 8 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 msgr2=0x7f4bd80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 0x7f4bd80a5050 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f4bdc005ab0 tx=0x7f4bdc000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 msgr2=0x7f4bec102d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f4be800d950 tx=0x7f4be800de10 comp rx=0 tx=0).stop 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 shutdown_connections 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bd80a2b90 0x7f4bd80a5050 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bec102330 0x7f4bec103290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bec101970 0x7f4bec102d50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 --2- 172.21.15.69:0/897604857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bec100780 0x7f4bec104710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:14.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 >> 172.21.15.69:0/897604857 conn(0x7f4bec0fbd10 msgr2=0x7f4bec0fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:14.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 shutdown_connections 2024-05-09T09:16:14.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:14.958+0000 7f4bf43f9640 1 -- 172.21.15.69:0/897604857 wait complete. 2024-05-09T09:16:15.008 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:14 smithi103 podman[64705]: 2024-05-09 09:16:14.928974058 +0000 UTC m=+0.135137636 container create 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-09T09:16:15.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:15 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/897604857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:15.280 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:15 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/897604857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:15.281 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 podman[64705]: 2024-05-09 09:16:15.182010821 +0000 UTC m=+0.388174398 container init 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-05-09T09:16:15.281 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 podman[64705]: 2024-05-09 09:16:15.1846947 +0000 UTC m=+0.390858277 container start 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, io.buildah.version=1.33.5) 2024-05-09T09:16:15.281 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 ceph-mds[64725]: -- 172.21.15.103:0/441077150 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555922a57a40 con 0x555922a2ac00 2024-05-09T09:16:15.281 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: starting mds.h at 2024-05-09T09:16:15.281 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 bash[64705]: 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 2024-05-09T09:16:15.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:15 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/897604857' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:15.617 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:16:15 smithi103 systemd[1]: Started Ceph mds.h for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:16.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:16.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.252 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.253 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:16.253 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:16.253 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:16.970 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:17.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 -- 172.21.15.69:0/2945224202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8101970 msgr2=0x7f2cf8101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 --2- 172.21.15.69:0/2945224202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8101970 0x7f2cf8101df0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009900 tx=0x7f2ce002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 -- 172.21.15.69:0/2945224202 shutdown_connections 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 --2- 172.21.15.69:0/2945224202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8102330 0x7f2cf8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 --2- 172.21.15.69:0/2945224202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8101970 0x7f2cf8101df0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 --2- 172.21.15.69:0/2945224202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8100780 0x7f2cf8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 -- 172.21.15.69:0/2945224202 >> 172.21.15.69:0/2945224202 conn(0x7f2cf80fbd10 msgr2=0x7f2cf80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 -- 172.21.15.69:0/2945224202 shutdown_connections 2024-05-09T09:16:17.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.086+0000 7f2cfd153640 1 -- 172.21.15.69:0/2945224202 wait complete. 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 Processor -- start 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 -- start start 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8101970 0x7f2cf819c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 0x7f2cf819ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810c400 con 0x7f2cf8100780 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810c280 con 0x7f2cf8102330 2024-05-09T09:16:17.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cfd153640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf810c580 con 0x7f2cf8101970 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cf6d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cf6d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57454/0 (socket says 172.21.15.69:57454) 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cf6d9d640 1 -- 172.21.15.69:0/1144107151 learned_addr learned my addr 172.21.15.69:0/1144107151 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cf759e640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 0x7f2cf819ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.087+0000 7f2cf659c640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8101970 0x7f2cf819c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 -- 172.21.15.69:0/1144107151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8101970 msgr2=0x7f2cf819c8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8101970 0x7f2cf819c8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 -- 172.21.15.69:0/1144107151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 msgr2=0x7f2cf819ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 0x7f2cf819ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 -- 172.21.15.69:0/1144107151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf819f7e0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf759e640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 0x7f2cf819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cf6d9d640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800e9c0 tx=0x7f2ce800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce800cd80 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2ce80035f0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf819fad0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce8010690 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.088+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf819fff0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.089+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf8107aa0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.090+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2ce80037a0 con 0x7f2cf8100780 2024-05-09T09:16:17.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.091+0000 7f2ceffff640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 0x7f2ce40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:17.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.091+0000 7f2cf659c640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 0x7f2ce40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:17.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.091+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f2ce80cf0f0 con 0x7f2cf8100780 2024-05-09T09:16:17.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.091+0000 7f2cf659c640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 0x7f2ce40a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0000c00 tx=0x7f2ce00315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:17.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.095+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2ce809c4f0 con 0x7f2cf8100780 2024-05-09T09:16:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:17 smithi069 ceph-mon[30123]: Deploying daemon mds.i on smithi163 2024-05-09T09:16:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:17 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:boot 2024-05-09T09:16:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:17 smithi069 ceph-mon[30123]: fsmap 8 up:standby 2024-05-09T09:16:17.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:16:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:17 smithi163 ceph-mon[34354]: Deploying daemon mds.i on smithi163 2024-05-09T09:16:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:17 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:boot 2024-05-09T09:16:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:17 smithi163 ceph-mon[34354]: fsmap 8 up:standby 2024-05-09T09:16:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:16:17.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.311+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2cf802d1c0 con 0x7f2cf8100780 2024-05-09T09:16:17.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.312+0000 7f2ceffff640 1 -- 172.21.15.69:0/1144107151 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) ==== 93+0+7183 (secure 0 0 0) 0x7f2cf802d1c0 con 0x7f2cf8100780 2024-05-09T09:16:17.311 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:17.311 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9}],"filesystems":[]} 2024-05-09T09:16:17.311 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 9 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 msgr2=0x7f2ce40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 0x7f2ce40a5050 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0000c00 tx=0x7f2ce00315d0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 msgr2=0x7f2cf806aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f2ce800e9c0 tx=0x7f2ce800ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 shutdown_connections 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ce40a2b90 0x7f2ce40a5050 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf8102330 0x7f2cf819ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf8101970 0x7f2cf819c8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 --2- 172.21.15.69:0/1144107151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf8100780 0x7f2cf806aa60 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 >> 172.21.15.69:0/1144107151 conn(0x7f2cf80fbd10 msgr2=0x7f2cf8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 shutdown_connections 2024-05-09T09:16:17.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:17.314+0000 7f2cfd153640 1 -- 172.21.15.69:0/1144107151 wait complete. 2024-05-09T09:16:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:17 smithi103 ceph-mon[34421]: Deploying daemon mds.i on smithi163 2024-05-09T09:16:17.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:17 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:boot 2024-05-09T09:16:17.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:17 smithi103 ceph-mon[34421]: fsmap 8 up:standby 2024-05-09T09:16:17.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:16:17.735 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:17 smithi163 systemd[1]: Starting Ceph mds.i for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:17.989 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:17 smithi163 podman[65831]: 2024-05-09 09:16:17.637623326 +0000 UTC m=+0.018646799 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:17.989 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:17 smithi163 podman[65831]: 2024-05-09 09:16:17.754299874 +0000 UTC m=+0.135323329 container create 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True) 2024-05-09T09:16:17.989 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:17 smithi163 podman[65831]: 2024-05-09 09:16:17.990070173 +0000 UTC m=+0.371093627 container init 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:18 smithi163 ceph-mon[34354]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:18 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1144107151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:17 smithi163 podman[65831]: 2024-05-09 09:16:17.992797287 +0000 UTC m=+0.373820743 container start 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i, GIT_CLEAN=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5) 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: starting mds.i at 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:18 smithi163 bash[65831]: 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 2024-05-09T09:16:18.305 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:16:18 smithi163 systemd[1]: Started Ceph mds.i for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:18.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:18 smithi103 ceph-mon[34421]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:18.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:18 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1144107151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:18.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:18 smithi069 ceph-mon[30123]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:18.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:18 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1144107151' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:19.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:19.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: Deploying daemon mds.g on smithi069 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:boot 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: fsmap 9 up:standby 2024-05-09T09:16:19.350 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: Deploying daemon mds.g on smithi069 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:boot 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: fsmap 9 up:standby 2024-05-09T09:16:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:16:19.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- 172.21.15.69:0/3775503198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50100770 msgr2=0x7f3d50100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- 172.21.15.69:0/3775503198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50100770 0x7f3d50100b90 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f3d40009900 tx=0x7f3d4002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- 172.21.15.69:0/3775503198 shutdown_connections 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- 172.21.15.69:0/3775503198 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50102320 0x7f3d50108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- 172.21.15.69:0/3775503198 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- 172.21.15.69:0/3775503198 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50100770 0x7f3d50100b90 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- 172.21.15.69:0/3775503198 >> 172.21.15.69:0/3775503198 conn(0x7f3d500fbce0 msgr2=0x7f3d500fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- 172.21.15.69:0/3775503198 shutdown_connections 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- 172.21.15.69:0/3775503198 wait complete. 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 Processor -- start 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- start start 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50100770 0x7f3d50196bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:19.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50197130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5010c4c0 con 0x7f3d50102320 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5010c340 con 0x7f3d50101960 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.432+0000 7f3d55af5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3d5010c640 con 0x7f3d50100770 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34430/0 (socket says 172.21.15.69:34430) 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 -- 172.21.15.69:0/2327724257 learned_addr learned my addr 172.21.15.69:0/2327724257 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d47fff640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50197130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4f7fe640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50100770 0x7f3d50196bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:19.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 -- 172.21.15.69:0/2327724257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50100770 msgr2=0x7f3d50196bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50100770 0x7f3d50196bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 -- 172.21.15.69:0/2327724257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 msgr2=0x7f3d50197130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50197130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 -- 172.21.15.69:0/2327724257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3d5019e810 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d47fff640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4ffff640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b970 tx=0x7f3d4800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48019070 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3d5019eb00 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3d4800cb40 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3d48007610 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.433+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3d5019f050 con 0x7f3d50102320 2024-05-09T09:16:19.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.434+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3d50107b60 con 0x7f3d50102320 2024-05-09T09:16:19.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.435+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f3d4801c070 con 0x7f3d50102320 2024-05-09T09:16:19.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.436+0000 7f3d4d7fa640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 0x7f3d3c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:19.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.436+0000 7f3d4f7fe640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 0x7f3d3c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:19.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.436+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f3d480ce1c0 con 0x7f3d50102320 2024-05-09T09:16:19.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.436+0000 7f3d4f7fe640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 0x7f3d3c0a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d40006010 tx=0x7f3d40000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:19.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.440+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3d4809b5c0 con 0x7f3d50102320 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: Deploying daemon mds.g on smithi069 2024-05-09T09:16:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:boot 2024-05-09T09:16:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: fsmap 9 up:standby 2024-05-09T09:16:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:16:19.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.656+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f3d5002d1c0 con 0x7f3d50102320 2024-05-09T09:16:19.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.657+0000 7f3d4d7fa640 1 -- 172.21.15.69:0/2327724257 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) ==== 94+0+8010 (secure 0 0 0) 0x7f3d4809aec0 con 0x7f3d50102320 2024-05-09T09:16:19.656 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:19.656 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T09:16:19.657 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 10 2024-05-09T09:16:19.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 msgr2=0x7f3d3c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:19.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 0x7f3d3c0a5050 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3d40006010 tx=0x7f3d40000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 msgr2=0x7f3d5019e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800b970 tx=0x7f3d4800be30 comp rx=0 tx=0).stop 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 shutdown_connections 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3d3c0a2b90 0x7f3d3c0a5050 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3d50102320 0x7f3d5019e1a0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3d50101960 0x7f3d50197130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 --2- 172.21.15.69:0/2327724257 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3d50100770 0x7f3d50196bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:19.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 >> 172.21.15.69:0/2327724257 conn(0x7f3d500fbce0 msgr2=0x7f3d50104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:19.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 shutdown_connections 2024-05-09T09:16:19.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:19.659+0000 7f3d55af5640 1 -- 172.21.15.69:0/2327724257 wait complete. 2024-05-09T09:16:20.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:20 smithi069 ceph-mon[30123]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:20.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:20 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2327724257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:20 smithi163 ceph-mon[34354]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:20 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2327724257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:20.590 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 systemd[1]: Starting Ceph mds.g for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:20 smithi103 ceph-mon[34421]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:20 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2327724257' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:20.887 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 podman[70825]: 2024-05-09 09:16:20.494347872 +0000 UTC m=+0.017938188 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:20.887 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 podman[70825]: 2024-05-09 09:16:20.62726809 +0000 UTC m=+0.150858394 container create 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:16:20.887 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 podman[70825]: 2024-05-09 09:16:20.888461642 +0000 UTC m=+0.412051956 container init 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-09T09:16:21.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 podman[70825]: 2024-05-09 09:16:20.891493353 +0000 UTC m=+0.415083668 container start 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc) 2024-05-09T09:16:21.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 ceph-mds[70844]: -- 172.21.15.69:0/3905833173 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cb153cda40 con 0x55cb153a0800 2024-05-09T09:16:21.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g[70840]: starting mds.g at 2024-05-09T09:16:21.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:21 smithi069 bash[70825]: 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be 2024-05-09T09:16:21.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:16:21 smithi069 systemd[1]: Started Ceph mds.g for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:21.670 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:21.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 -- 172.21.15.69:0/1178872807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 msgr2=0x7f05e0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:21.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 --2- 172.21.15.69:0/1178872807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e0100bb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f05d4009900 tx=0x7f05d402a410 comp rx=0 tx=0).stop 2024-05-09T09:16:21.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 -- 172.21.15.69:0/1178872807 shutdown_connections 2024-05-09T09:16:21.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 --2- 172.21.15.69:0/1178872807 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:21.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 --2- 172.21.15.69:0/1178872807 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 --2- 172.21.15.69:0/1178872807 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e0100bb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 -- 172.21.15.69:0/1178872807 >> 172.21.15.69:0/1178872807 conn(0x7f05e00fbd20 msgr2=0x7f05e00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 -- 172.21.15.69:0/1178872807 shutdown_connections 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.778+0000 7f05e7709640 1 -- 172.21.15.69:0/1178872807 wait complete. 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 Processor -- start 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 -- start start 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e019c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e019ef80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010c460 con 0x7f05e0100790 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010c2e0 con 0x7f05e0101980 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f05e010c5e0 con 0x7f05e0102340 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:21.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34456/0 (socket says 172.21.15.69:34456) 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 -- 172.21.15.69:0/820707299 learned_addr learned my addr 172.21.15.69:0/820707299 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e5ca6640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e019ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e4ca4640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e019c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 -- 172.21.15.69:0/820707299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 msgr2=0x7f05e019ef80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e019ef80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 -- 172.21.15.69:0/820707299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 msgr2=0x7f05e019c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e019c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 -- 172.21.15.69:0/820707299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f05e019f840 con 0x7f05e0100790 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e4ca4640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e5ca6640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e54a5640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f05d4031610 tx=0x7f05d40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d403f030 con 0x7f05e0100790 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.779+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f05e019fad0 con 0x7f05e0100790 2024-05-09T09:16:21.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.780+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f05d4003ce0 con 0x7f05e0100790 2024-05-09T09:16:21.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.780+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f05d4035070 con 0x7f05e0100790 2024-05-09T09:16:21.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.780+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f05e019ffc0 con 0x7f05e0100790 2024-05-09T09:16:21.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.781+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f05e0107ad0 con 0x7f05e0100790 2024-05-09T09:16:21.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.782+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f05d4042030 con 0x7f05e0100790 2024-05-09T09:16:21.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.782+0000 7f05d27fc640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 0x7f05cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:21.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.782+0000 7f05e4ca4640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 0x7f05cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:21.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.782+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f05d40ef1f0 con 0x7f05e0100790 2024-05-09T09:16:21.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.783+0000 7f05e4ca4640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 0x7f05cc0a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f05e006a790 tx=0x7f05dc005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:21.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:21.787+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f05d40bc5f0 con 0x7f05e0100790 2024-05-09T09:16:22.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.011+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f05e002d1c0 con 0x7f05e0100790 2024-05-09T09:16:22.010 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.012+0000 7f05d27fc640 1 -- 172.21.15.69:0/820707299 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) ==== 94+0+8836 (secure 0 0 0) 0x7f05d40bbef0 con 0x7f05e0100790 2024-05-09T09:16:22.010 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:22.011 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T09:16:22.011 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 11 2024-05-09T09:16:22.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 msgr2=0x7f05cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 0x7f05cc0a5050 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f05e006a790 tx=0x7f05dc005e50 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 msgr2=0x7f05e006aa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f05d4031610 tx=0x7f05d40043b0 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 shutdown_connections 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05cc0a2b90 0x7f05cc0a5050 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f05e0102340 0x7f05e019ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f05e0101980 0x7f05e019c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 --2- 172.21.15.69:0/820707299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f05e0100790 0x7f05e006aa20 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 >> 172.21.15.69:0/820707299 conn(0x7f05e00fbd20 msgr2=0x7f05e0104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 shutdown_connections 2024-05-09T09:16:22.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:22.014+0000 7f05e7709640 1 -- 172.21.15.69:0/820707299 wait complete. 2024-05-09T09:16:22.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:22.299 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: Deploying daemon mds.a on smithi069 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.69:6838/1794172552,v1:172.21.15.69:6839/1794172552] up:boot 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: fsmap 10 up:standby 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:16:22.300 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:22 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/820707299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:22.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: Deploying daemon mds.a on smithi069 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.69:6838/1794172552,v1:172.21.15.69:6839/1794172552] up:boot 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: fsmap 10 up:standby 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:16:22.369 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:22 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/820707299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: Deploying daemon mds.a on smithi069 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.69:6838/1794172552,v1:172.21.15.69:6839/1794172552] up:boot 2024-05-09T09:16:22.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: fsmap 10 up:standby 2024-05-09T09:16:22.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-05-09T09:16:22.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:22 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/820707299' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:23.284 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 systemd[1]: Starting Ceph mds.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:23.696 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 podman[71186]: 2024-05-09 09:16:23.333965411 +0000 UTC m=+0.017915119 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:23.696 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 podman[71186]: 2024-05-09 09:16:23.449329806 +0000 UTC m=+0.133279508 container create faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 podman[71186]: 2024-05-09 09:16:23.818952452 +0000 UTC m=+0.502902164 container init faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 podman[71186]: 2024-05-09 09:16:23.821917378 +0000 UTC m=+0.505867090 container start faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True) 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 ceph-mds[71205]: -- 172.21.15.69:0/3869771871 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55940c50da40 con 0x55940c4e0800 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a[71201]: starting mds.a at 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 bash[71186]: faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 2024-05-09T09:16:24.010 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:16:23 smithi069 systemd[1]: Started Ceph mds.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:24.024 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 -- 172.21.15.69:0/3689390616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604102a70 msgr2=0x7f2604102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 --2- 172.21.15.69:0/3689390616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604102a70 0x7f2604102ef0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f2600009900 tx=0x7f260002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 -- 172.21.15.69:0/3689390616 shutdown_connections 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 --2- 172.21.15.69:0/3689390616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604103430 0x7f2604109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 --2- 172.21.15.69:0/3689390616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604102a70 0x7f2604102ef0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 --2- 172.21.15.69:0/3689390616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604101880 0x7f2604101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 -- 172.21.15.69:0/3689390616 >> 172.21.15.69:0/3689390616 conn(0x7f26040fce10 msgr2=0x7f26040ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:24.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 -- 172.21.15.69:0/3689390616 shutdown_connections 2024-05-09T09:16:24.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.131+0000 7f260c69a640 1 -- 172.21.15.69:0/3689390616 wait complete. 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 Processor -- start 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 -- start start 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604102a70 0x7f2604198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 0x7f260419f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f260410d570 con 0x7f2604101880 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f260410d3f0 con 0x7f2604102a70 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260c69a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f260410d6f0 con 0x7f2604103430 2024-05-09T09:16:24.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34488/0 (socket says 172.21.15.69:34488) 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 -- 172.21.15.69:0/2326993580 learned_addr learned my addr 172.21.15.69:0/2326993580 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f2609c35640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604102a70 0x7f2604198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260ac37640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 0x7f260419f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 -- 172.21.15.69:0/2326993580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 msgr2=0x7f260419f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 0x7f260419f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 -- 172.21.15.69:0/2326993580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604102a70 msgr2=0x7f2604198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604102a70 0x7f2604198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 -- 172.21.15.69:0/2326993580 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f260419f800 con 0x7f2604101880 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260ac37640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 0x7f260419f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f260a436640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25f800b450 tx=0x7f25f800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.132+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f8015020 con 0x7f2604101880 2024-05-09T09:16:24.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.133+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25f8011d70 con 0x7f2604101880 2024-05-09T09:16:24.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.133+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25f800f340 con 0x7f2604101880 2024-05-09T09:16:24.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.133+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f260419faf0 con 0x7f2604101880 2024-05-09T09:16:24.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.133+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f260419ffc0 con 0x7f2604101880 2024-05-09T09:16:24.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.134+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2604108ba0 con 0x7f2604101880 2024-05-09T09:16:24.135 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.135+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f25f8018020 con 0x7f2604101880 2024-05-09T09:16:24.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.135+0000 7f25f77fe640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 0x7f25f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:24.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.135+0000 7f2609c35640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 0x7f25f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:24.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.135+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f25f80ceb80 con 0x7f2604101880 2024-05-09T09:16:24.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.136+0000 7f2609c35640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 0x7f25f00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f26000314e0 tx=0x7f2600000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:24.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.139+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25f801f020 con 0x7f2604101880 2024-05-09T09:16:24.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.354+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f260402d1c0 con 0x7f2604101880 2024-05-09T09:16:24.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.355+0000 7f25f77fe640 1 -- 172.21.15.69:0/2326993580 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f25f800d040 con 0x7f2604101880 2024-05-09T09:16:24.354 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:24.354 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T09:16:24.354 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 12 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 msgr2=0x7f25f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 0x7f25f00a5050 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f26000314e0 tx=0x7f2600000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 msgr2=0x7f2604197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f25f800b450 tx=0x7f25f800b910 comp rx=0 tx=0).stop 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 shutdown_connections 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25f00a2b90 0x7f25f00a5050 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2604103430 0x7f260419f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2604102a70 0x7f2604198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 --2- 172.21.15.69:0/2326993580 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2604101880 0x7f2604197be0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:24.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 >> 172.21.15.69:0/2326993580 conn(0x7f26040fce10 msgr2=0x7f2604105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:24.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 shutdown_connections 2024-05-09T09:16:24.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:24.357+0000 7f260c69a640 1 -- 172.21.15.69:0/2326993580 wait complete. 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:24.366 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:24.367 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:24.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-05-09T09:16:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:25 smithi103 ceph-mon[34421]: Deploying daemon mds.b on smithi103 2024-05-09T09:16:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:25 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.69:6840/4006889644,v1:172.21.15.69:6841/4006889644] up:boot 2024-05-09T09:16:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:25 smithi103 ceph-mon[34421]: fsmap 11 up:standby 2024-05-09T09:16:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:16:25.330 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:25 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2326993580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:25.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:25 smithi069 ceph-mon[30123]: Deploying daemon mds.b on smithi103 2024-05-09T09:16:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:25 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.69:6840/4006889644,v1:172.21.15.69:6841/4006889644] up:boot 2024-05-09T09:16:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:25 smithi069 ceph-mon[30123]: fsmap 11 up:standby 2024-05-09T09:16:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:16:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:25 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2326993580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:25 smithi163 ceph-mon[34354]: Deploying daemon mds.b on smithi103 2024-05-09T09:16:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:25 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.69:6840/4006889644,v1:172.21.15.69:6841/4006889644] up:boot 2024-05-09T09:16:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:25 smithi163 ceph-mon[34354]: fsmap 11 up:standby 2024-05-09T09:16:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-05-09T09:16:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:25 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2326993580' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:26.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:26 smithi103 ceph-mon[34421]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:26.367 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 systemd[1]: Starting Ceph mds.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:16:26.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 -- 172.21.15.69:0/2140529946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04101060 msgr2=0x7f2a041014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 --2- 172.21.15.69:0/2140529946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04101060 0x7f2a041014e0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f29ec009900 tx=0x7f29ec02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 -- 172.21.15.69:0/2140529946 shutdown_connections 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 --2- 172.21.15.69:0/2140529946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 0x7f2a04108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 --2- 172.21.15.69:0/2140529946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a04101060 0x7f2a041014e0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 --2- 172.21.15.69:0/2140529946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a040fff10 0x7f2a04100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 -- 172.21.15.69:0/2140529946 >> 172.21.15.69:0/2140529946 conn(0x7f2a040fb4a0 msgr2=0x7f2a040fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:26.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 -- 172.21.15.69:0/2140529946 shutdown_connections 2024-05-09T09:16:26.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.483+0000 7f2a08989640 1 -- 172.21.15.69:0/2140529946 wait complete. 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 Processor -- start 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 -- start start 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a04101060 0x7f2a0419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 0x7f2a0419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0410c490 con 0x7f2a040fff10 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0410c310 con 0x7f2a04101a20 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a08989640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2a0410c610 con 0x7f2a04101060 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34518/0 (socket says 172.21.15.69:34518) 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 -- 172.21.15.69:0/2536966840 learned_addr learned my addr 172.21.15.69:0/2536966840 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a02d9d640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 0x7f2a0419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 -- 172.21.15.69:0/2536966840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a04101060 msgr2=0x7f2a0419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:26.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a04101060 0x7f2a0419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 -- 172.21.15.69:0/2536966840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 msgr2=0x7f2a0419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 0x7f2a0419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 -- 172.21.15.69:0/2536966840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2a0419f870 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.484+0000 7f2a0259c640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29f400b4a0 tx=0x7f29f400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.485+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f4014070 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.485+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2a0419fb60 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.485+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29f4010af0 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.485+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29f4017d10 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.485+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2a041a0060 con 0x7f2a040fff10 2024-05-09T09:16:26.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.486+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2a041072f0 con 0x7f2a040fff10 2024-05-09T09:16:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.487+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f29f4017070 con 0x7f2a040fff10 2024-05-09T09:16:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.488+0000 7f29fb7fe640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 0x7f29f00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.488+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f29f40ce090 con 0x7f2a040fff10 2024-05-09T09:16:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.488+0000 7f29fbfff640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 0x7f29f00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:26.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.488+0000 7f29fbfff640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 0x7f29f00a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29ec02a930 tx=0x7f29ec031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:26.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.492+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f29f409b490 con 0x7f2a040fff10 2024-05-09T09:16:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:26 smithi069 ceph-mon[30123]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:26.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:26 smithi163 ceph-mon[34354]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:26.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.709+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2a0402d1c0 con 0x7f2a040fff10 2024-05-09T09:16:26.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.709+0000 7f29fb7fe640 1 -- 172.21.15.69:0/2536966840 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) ==== 94+0+9662 (secure 0 0 0) 0x7f29f409ad90 con 0x7f2a040fff10 2024-05-09T09:16:26.709 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:26.709 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[]} 2024-05-09T09:16:26.710 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 12 2024-05-09T09:16:26.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 msgr2=0x7f29f00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 0x7f29f00a5050 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f29ec02a930 tx=0x7f29ec031400 comp rx=0 tx=0).stop 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 msgr2=0x7f2a04109870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f29f400b4a0 tx=0x7f29f400b960 comp rx=0 tx=0).stop 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 shutdown_connections 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29f00a2b90 0x7f29f00a5050 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2a04101a20 0x7f2a0419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2a04101060 0x7f2a0419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 --2- 172.21.15.69:0/2536966840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2a040fff10 0x7f2a04109870 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 >> 172.21.15.69:0/2536966840 conn(0x7f2a040fb4a0 msgr2=0x7f2a041044c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 shutdown_connections 2024-05-09T09:16:26.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:26.712+0000 7f2a08989640 1 -- 172.21.15.69:0/2536966840 wait complete. 2024-05-09T09:16:26.714 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 podman[65017]: 2024-05-09 09:16:26.344504223 +0000 UTC m=+0.018406681 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:16:26.714 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 podman[65017]: 2024-05-09 09:16:26.460025748 +0000 UTC m=+0.133928197 container create 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-05-09T09:16:26.714 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 podman[65017]: 2024-05-09 09:16:26.713159559 +0000 UTC m=+0.387062008 container init 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS) 2024-05-09T09:16:27.117 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 podman[65017]: 2024-05-09 09:16:26.715880615 +0000 UTC m=+0.389783071 container start 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:16:27.117 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 ceph-mds[65037]: -- 172.21.15.103:0/604637084 <== mon.2 v2:172.21.15.103:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56492c429a40 con 0x56492c3fc800 2024-05-09T09:16:27.117 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: starting mds.b at 2024-05-09T09:16:27.117 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 bash[65017]: 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 2024-05-09T09:16:27.117 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:26 smithi103 systemd[1]: Started Ceph mds.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2536966840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:27.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2536966840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2536966840' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:28 smithi069 ceph-mon[30123]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:28 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:boot 2024-05-09T09:16:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:28 smithi069 ceph-mon[30123]: fsmap 12 up:standby 2024-05-09T09:16:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:16:28.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:28 smithi163 ceph-mon[34354]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:28 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:boot 2024-05-09T09:16:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:28 smithi163 ceph-mon[34354]: fsmap 12 up:standby 2024-05-09T09:16:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:16:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:28 smithi103 ceph-mon[34421]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:28 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:boot 2024-05-09T09:16:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:28 smithi103 ceph-mon[34421]: fsmap 12 up:standby 2024-05-09T09:16:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:16:28.723 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 -- 172.21.15.69:0/2842881901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 msgr2=0x7fabd8107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/2842881901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd8107070 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fabcc009900 tx=0x7fabcc02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 -- 172.21.15.69:0/2842881901 shutdown_connections 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/2842881901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd81099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/2842881901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd8107070 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/2842881901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:28.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 -- 172.21.15.69:0/2842881901 >> 172.21.15.69:0/2842881901 conn(0x7fabd80fbce0 msgr2=0x7fabd80fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 -- 172.21.15.69:0/2842881901 shutdown_connections 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.836+0000 7fabdd3c8640 1 -- 172.21.15.69:0/2842881901 wait complete. 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 Processor -- start 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 -- start start 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabd810d4e0 con 0x7fabd8104c80 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabd810d360 con 0x7fabd81075b0 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.837+0000 7fabdd3c8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fabd810d660 con 0x7fabd8102350 2024-05-09T09:16:28.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46276/0 (socket says 172.21.15.69:46276) 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 -- 172.21.15.69:0/3222766491 learned_addr learned my addr 172.21.15.69:0/3222766491 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd6ffd640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 -- 172.21.15.69:0/3222766491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 msgr2=0x7fabd8197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd77fe640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 -- 172.21.15.69:0/3222766491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 msgr2=0x7fabd819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 -- 172.21.15.69:0/3222766491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabd819f7d0 con 0x7fabd8104c80 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd6ffd640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd77fe640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabd67fc640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fabcc031610 tx=0x7fabcc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:28.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc03f030 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabcc003ad0 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabcc035070 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabd819fa60 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.838+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabd819ff50 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.839+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabd80fd7d0 con 0x7fabd8104c80 2024-05-09T09:16:28.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.840+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fabcc042030 con 0x7fabd8104c80 2024-05-09T09:16:28.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.841+0000 7fabbffff640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 0x7fabc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:28.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.841+0000 7fabd6ffd640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 0x7fabc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:28.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.841+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fabcc0ef350 con 0x7fabd8104c80 2024-05-09T09:16:28.843 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.842+0000 7fabd6ffd640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 0x7fabc00a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009c20 tx=0x7fabc8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:28.844 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:28.846+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fabcc0bc6b0 con 0x7fabd8104c80 2024-05-09T09:16:29.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.061+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabd8062430 con 0x7fabd8104c80 2024-05-09T09:16:29.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.062+0000 7fabbffff640 1 -- 172.21.15.69:0/3222766491 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) ==== 94+0+10488 (secure 0 0 0) 0x7fabcc0bbfb0 con 0x7fabd8104c80 2024-05-09T09:16:29.061 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:29.061 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10},{"gid":24527,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":13}],"filesystems":[]} 2024-05-09T09:16:29.062 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 13 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 msgr2=0x7fabc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 0x7fabc00a5050 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fabc8009c20 tx=0x7fabc8005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 msgr2=0x7fabd81980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fabcc031610 tx=0x7fabcc004270 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 shutdown_connections 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabc00a2b90 0x7fabc00a5050 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabd81075b0 0x7fabd819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabd8104c80 0x7fabd81980f0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 --2- 172.21.15.69:0/3222766491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabd8102350 0x7fabd8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 >> 172.21.15.69:0/3222766491 conn(0x7fabd80fbce0 msgr2=0x7fabd8102dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.064 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 shutdown_connections 2024-05-09T09:16:29.065 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.064+0000 7fabdd3c8640 1 -- 172.21.15.69:0/3222766491 wait complete. 2024-05-09T09:16:29.076 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 -- 172.21.15.69:0/2037897110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 msgr2=0x7f9d30101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/2037897110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d30101df0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009900 tx=0x7f9d1c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 -- 172.21.15.69:0/2037897110 shutdown_connections 2024-05-09T09:16:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/2037897110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30102330 0x7f9d30108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/2037897110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d30101df0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/2037897110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d30100780 0x7f9d30100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 -- 172.21.15.69:0/2037897110 >> 172.21.15.69:0/2037897110 conn(0x7f9d300fbd30 msgr2=0x7f9d300fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 -- 172.21.15.69:0/2037897110 shutdown_connections 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.179+0000 7f9d36aa5640 1 -- 172.21.15.69:0/2037897110 wait complete. 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 Processor -- start 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 -- start start 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 0x7f9d3006aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d30102330 0x7f9d301a11f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3010c400 con 0x7f9d30101970 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3010c280 con 0x7f9d30102330 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9d3010c580 con 0x7f9d30100780 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46288/0 (socket says 172.21.15.69:46288) 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 -- 172.21.15.69:0/1816392896 learned_addr learned my addr 172.21.15.69:0/1816392896 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d34841640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 0x7f9d3006aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 -- 172.21.15.69:0/1816392896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 msgr2=0x7f9d3006aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 0x7f9d3006aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 -- 172.21.15.69:0/1816392896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d30102330 msgr2=0x7f9d301a11f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d30102330 0x7f9d301a11f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 -- 172.21.15.69:0/1816392896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d301a19e0 con 0x7f9d30101970 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d34841640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 0x7f9d3006aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:29.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2ffff640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009230 tx=0x7f9d1c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c03f030 con 0x7f9d30101970 2024-05-09T09:16:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d1c003ce0 con 0x7f9d30101970 2024-05-09T09:16:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d301a1c10 con 0x7f9d30101970 2024-05-09T09:16:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.180+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d1c035070 con 0x7f9d30101970 2024-05-09T09:16:29.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.181+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d301a2150 con 0x7f9d30101970 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.182+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d30107aa0 con 0x7f9d30101970 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.183+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9d1c042030 con 0x7f9d30101970 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.184+0000 7f9d2dffb640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 0x7f9d200a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.184+0000 7f9d34841640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 0x7f9d200a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.184+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7f9d1c0ef0f0 con 0x7f9d30101970 2024-05-09T09:16:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.184+0000 7f9d34841640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 0x7f9d200a5100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d301017d0 tx=0x7f9d24005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.188+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9d1c0bc4f0 con 0x7f9d30101970 2024-05-09T09:16:29.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.368+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9d3002d420 con 0x7f9d30101970 2024-05-09T09:16:29.369 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.370+0000 7f9d2dffb640 1 -- 172.21.15.69:0/1816392896 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) ==== 74+0+18444 (secure 0 0 0) 0x7f9d1c0bbdf0 con 0x7f9d30101970 2024-05-09T09:16:29.369 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:29.369 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":81,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:15:43.727619+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 msgr2=0x7f9d200a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 0x7f9d200a5100 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f9d301017d0 tx=0x7f9d24005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 msgr2=0x7f9d3019eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9d1c009230 tx=0x7f9d1c0042f0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 shutdown_connections 2024-05-09T09:16:29.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9d200a2c40 0x7f9d200a5100 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d30102330 0x7f9d301a11f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d30101970 0x7f9d3019eca0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 --2- 172.21.15.69:0/1816392896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d30100780 0x7f9d3006aa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 >> 172.21.15.69:0/1816392896 conn(0x7f9d300fbd30 msgr2=0x7f9d30104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 shutdown_connections 2024-05-09T09:16:29.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.372+0000 7f9d36aa5640 1 -- 172.21.15.69:0/1816392896 wait complete. 2024-05-09T09:16:29.383 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-05-09T09:16:29.383 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 -- 172.21.15.69:0/3544392553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde4101a50 msgr2=0x7fbde4108370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3544392553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde4101a50 0x7fbde4108370 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fbddc009900 tx=0x7fbddc02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 -- 172.21.15.69:0/3544392553 shutdown_connections 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3544392553 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde4101a50 0x7fbde4108370 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3544392553 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101090 0x7fbde4101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3544392553 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde40ffeb0 0x7fbde41002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 -- 172.21.15.69:0/3544392553 >> 172.21.15.69:0/3544392553 conn(0x7fbde40fb470 msgr2=0x7fbde40fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 -- 172.21.15.69:0/3544392553 shutdown_connections 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 -- 172.21.15.69:0/3544392553 wait complete. 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.492+0000 7fbdebb34640 1 Processor -- start 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- start start 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde4101090 0x7fbde419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101a50 0x7fbde419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbde410bba0 con 0x7fbde40ffeb0 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbde410ba20 con 0x7fbde4101a50 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbde410bd20 con 0x7fbde4101090 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46298/0 (socket says 172.21.15.69:46298) 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 -- 172.21.15.69:0/3042483694 learned_addr learned my addr 172.21.15.69:0/3042483694 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde90cf640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde4101090 0x7fbde419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdea0d1640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101a50 0x7fbde419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 -- 172.21.15.69:0/3042483694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde4101090 msgr2=0x7fbde419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde4101090 0x7fbde419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 -- 172.21.15.69:0/3042483694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101a50 msgr2=0x7fbde419eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101a50 0x7fbde419eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 -- 172.21.15.69:0/3042483694 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbde419f800 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde98d0640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400eb70 tx=0x7fbdd400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4003b70 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbde419fa90 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbdd400ee70 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbdd4010830 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.493+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbde419ff90 con 0x7fbde40ffeb0 2024-05-09T09:16:29.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.494+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbde4107620 con 0x7fbde40ffeb0 2024-05-09T09:16:29.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.495+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fbdd4004020 con 0x7fbde40ffeb0 2024-05-09T09:16:29.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.496+0000 7fbde2ffd640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 0x7fbdcc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.496+0000 7fbde90cf640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 0x7fbdcc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.496+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fbdd40cf230 con 0x7fbde40ffeb0 2024-05-09T09:16:29.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.496+0000 7fbde90cf640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 0x7fbdcc0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbde40fd2b0 tx=0x7fbdd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.500+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbdd409c630 con 0x7fbde40ffeb0 2024-05-09T09:16:29.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.711+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fbde4062430 con 0x7fbde40ffeb0 2024-05-09T09:16:29.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.712+0000 7fbde2ffd640 1 -- 172.21.15.69:0/3042483694 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) ==== 93+0+10 (secure 0 0 0) 0x7fbdd409bf30 con 0x7fbde40ffeb0 2024-05-09T09:16:29.710 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:16:29.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.713+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 msgr2=0x7fbdcc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.713+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 0x7fbdcc0a5050 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fbde40fd2b0 tx=0x7fbdd8005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.713+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 msgr2=0x7fbde40fd540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.713+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fbdd400eb70 tx=0x7fbdd400c6a0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 shutdown_connections 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbdcc0a2b90 0x7fbdcc0a5050 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbde4101a50 0x7fbde419eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbde4101090 0x7fbde419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 --2- 172.21.15.69:0/3042483694 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbde40ffeb0 0x7fbde40fd540 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 >> 172.21.15.69:0/3042483694 conn(0x7fbde40fb470 msgr2=0x7fbde41047f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 shutdown_connections 2024-05-09T09:16:29.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.714+0000 7fbdebb34640 1 -- 172.21.15.69:0/3042483694 wait complete. 2024-05-09T09:16:29.724 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 -- 172.21.15.69:0/3578757310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0100140 msgr2=0x7fa6e0100560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 --2- 172.21.15.69:0/3578757310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0100140 0x7fa6e0100560 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa6d0009900 tx=0x7fa6d002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 -- 172.21.15.69:0/3578757310 shutdown_connections 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 --2- 172.21.15.69:0/3578757310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0101440 0x7fa6e01018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 --2- 172.21.15.69:0/3578757310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e0100f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 --2- 172.21.15.69:0/3578757310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0100140 0x7fa6e0100560 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 -- 172.21.15.69:0/3578757310 >> 172.21.15.69:0/3578757310 conn(0x7fa6e00fbce0 msgr2=0x7fa6e00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 -- 172.21.15.69:0/3578757310 shutdown_connections 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.830+0000 7fa6e7710640 1 -- 172.21.15.69:0/3578757310 wait complete. 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 Processor -- start 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 -- start start 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f6b0 con 0x7fa6e0101440 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f530 con 0x7fa6e0100aa0 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e7710640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa6e010f830 con 0x7fa6e0100140 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e5cad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e54ac640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e5cad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46312/0 (socket says 172.21.15.69:46312) 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e54ac640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:58596/0 (socket says 172.21.15.69:58596) 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e5cad640 1 -- 172.21.15.69:0/1477330070 learned_addr learned my addr 172.21.15.69:0/1477330070 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.831+0000 7fa6e4cab640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 -- 172.21.15.69:0/1477330070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 msgr2=0x7fa6e0199da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 -- 172.21.15.69:0/1477330070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 msgr2=0x7fa6e019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 -- 172.21.15.69:0/1477330070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa6e01a19b0 con 0x7fa6e0101440 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e54ac640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e4cab640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e5cad640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800eb10 tx=0x7fa6d800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8004b20 con 0x7fa6e0101440 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa6e01a1ca0 con 0x7fa6e0101440 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa6d8004cc0 con 0x7fa6e0101440 2024-05-09T09:16:29.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa6d8010760 con 0x7fa6e0101440 2024-05-09T09:16:29.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.832+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa6e01a21a0 con 0x7fa6e0101440 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.833+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa6e010b090 con 0x7fa6e0101440 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.835+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa6d8005020 con 0x7fa6e0101440 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.836+0000 7fa6d67fc640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 0x7fa6cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.836+0000 7fa6e54ac640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 0x7fa6cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.836+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fa6d80cf1e0 con 0x7fa6e0101440 2024-05-09T09:16:29.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.836+0000 7fa6e54ac640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 0x7fa6cc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6e019b270 tx=0x7fa6d0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:29.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:29.840+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa6d809c5e0 con 0x7fa6e0101440 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3222766491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1816392896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:30.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:30.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:29 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3042483694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:30.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.021+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) -- 0x7fa6e002d410 con 0x7fa6e0101440 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.021+0000 7fa6d67fc640 1 -- 172.21.15.69:0/1477330070 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) ==== 84+0+65 (secure 0 0 0) 0x7fa6d809bee0 con 0x7fa6e0101440 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout: "poolnum": 1, 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout: "poolname": ".mgr" 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:16:30.020 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 msgr2=0x7fa6cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 0x7fa6cc0a5050 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fa6e019b270 tx=0x7fa6d0000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 msgr2=0x7fa6e01a1310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fa6d800eb10 tx=0x7fa6d800c2d0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 shutdown_connections 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa6cc0a2b90 0x7fa6cc0a5050 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa6e0101440 0x7fa6e01a1310 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.022 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa6e0100aa0 0x7fa6e019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 --2- 172.21.15.69:0/1477330070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa6e0100140 0x7fa6e0199da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 >> 172.21.15.69:0/1477330070 conn(0x7fa6e00fbce0 msgr2=0x7fa6e00fd430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:30.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 shutdown_connections 2024-05-09T09:16:30.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.023+0000 7fa6e7710640 1 -- 172.21.15.69:0/1477330070 wait complete. 2024-05-09T09:16:30.033 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-05-09T09:16:30.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3222766491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:30.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1816392896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:29 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3042483694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:30.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 -- 172.21.15.69:0/2075609919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258103440 msgr2=0x7fc258109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 --2- 172.21.15.69:0/2075609919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258103440 0x7fc258109cd0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc250009ca0 tx=0x7fc25002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 -- 172.21.15.69:0/2075609919 shutdown_connections 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 --2- 172.21.15.69:0/2075609919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258103440 0x7fc258109cd0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 --2- 172.21.15.69:0/2075609919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258102a80 0x7fc258102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 --2- 172.21.15.69:0/2075609919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 0x7fc258101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 -- 172.21.15.69:0/2075609919 >> 172.21.15.69:0/2075609919 conn(0x7fc2580fce00 msgr2=0x7fc2580ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 -- 172.21.15.69:0/2075609919 shutdown_connections 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.148+0000 7fc2603c1640 1 -- 172.21.15.69:0/2075609919 wait complete. 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 Processor -- start 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 -- start start 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 0x7fc258197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 0x7fc25819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d510 con 0x7fc258102a80 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d390 con 0x7fc258103440 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc2603c1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc25810d690 con 0x7fc258101890 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc25d95c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc25d95c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46340/0 (socket says 172.21.15.69:46340) 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc25d95c640 1 -- 172.21.15.69:0/565050168 learned_addr learned my addr 172.21.15.69:0/565050168 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc25e15d640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 0x7fc258197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.149+0000 7fc25e95e640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 0x7fc25819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 -- 172.21.15.69:0/565050168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 msgr2=0x7fc258197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 0x7fc258197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 -- 172.21.15.69:0/565050168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 msgr2=0x7fc25819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 0x7fc25819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 -- 172.21.15.69:0/565050168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc25819f770 con 0x7fc258102a80 2024-05-09T09:16:30.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25e95e640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 0x7fc25819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc25d95c640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc24800e9f0 tx=0x7fc24800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc24800cd80 con 0x7fc258102a80 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc248010910 con 0x7fc258102a80 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc25819fa60 con 0x7fc258102a80 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc248018990 con 0x7fc258102a80 2024-05-09T09:16:30.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.150+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc25819ffb0 con 0x7fc258102a80 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.151+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fc258108bb0 con 0x7fc258102a80 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.152+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fc248010ac0 con 0x7fc258102a80 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.153+0000 7fc24f7fe640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 0x7fc2440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.153+0000 7fc25e15d640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 0x7fc2440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.153+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(81..81 src has 1..81) ==== 7127+0+0 (secure 0 0 0) 0x7fc2480ceeb0 con 0x7fc258102a80 2024-05-09T09:16:30.153 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.154+0000 7fc25e15d640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 0x7fc2440a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc2581028e0 tx=0x7fc254005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:30.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.158+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fc24809bc60 con 0x7fc258102a80 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3222766491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1816392896' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:16:30.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:29 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3042483694' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:30.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.358+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) -- 0x7fc25802d400 con 0x7fc258102a80 2024-05-09T09:16:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.849+0000 7fc24f7fe640 1 -- 172.21.15.69:0/565050168 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) ==== 138+0+0 (secure 0 0 0) 0x7fc24809ff60 con 0x7fc258102a80 2024-05-09T09:16:30.848 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' created 2024-05-09T09:16:30.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 msgr2=0x7fc2440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 0x7fc2440a5050 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fc2581028e0 tx=0x7fc254005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:30.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 msgr2=0x7fc258198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fc24800e9f0 tx=0x7fc24800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 shutdown_connections 2024-05-09T09:16:30.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc2440a2b90 0x7fc2440a5050 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc258103440 0x7fc25819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc258102a80 0x7fc258198090 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 --2- 172.21.15.69:0/565050168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc258101890 0x7fc258197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 >> 172.21.15.69:0/565050168 conn(0x7fc2580fce00 msgr2=0x7fc258105d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.852+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 shutdown_connections 2024-05-09T09:16:30.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.853+0000 7fc2603c1640 1 -- 172.21.15.69:0/565050168 wait complete. 2024-05-09T09:16:30.864 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-05-09T09:16:30.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 -- 172.21.15.69:0/3684267400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1034b0 msgr2=0x7efe0c109d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 --2- 172.21.15.69:0/3684267400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1034b0 0x7efe0c109d40 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7efe04009cd0 tx=0x7efe0402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 -- 172.21.15.69:0/3684267400 shutdown_connections 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 --2- 172.21.15.69:0/3684267400 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c1034b0 0x7efe0c109d40 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 --2- 172.21.15.69:0/3684267400 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c102af0 0x7efe0c102f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 --2- 172.21.15.69:0/3684267400 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c0ffb20 0x7efe0c0fff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.976+0000 7efe119c1640 1 -- 172.21.15.69:0/3684267400 >> 172.21.15.69:0/3684267400 conn(0x7efe0c0fb450 msgr2=0x7efe0c0fd890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- 172.21.15.69:0/3684267400 shutdown_connections 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- 172.21.15.69:0/3684267400 wait complete. 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 Processor -- start 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- start start 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 0x7efe0c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c1034b0 0x7efe0c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10d4d0 con 0x7efe0c102af0 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10d350 con 0x7efe0c1034b0 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe119c1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efe0c10d650 con 0x7efe0c0ffb20 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe03fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe03fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46354/0 (socket says 172.21.15.69:46354) 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe03fff640 1 -- 172.21.15.69:0/1637227572 learned_addr learned my addr 172.21.15.69:0/1637227572 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe0affd640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 0x7efe0c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.977+0000 7efe0b7fe640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c1034b0 0x7efe0c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 -- 172.21.15.69:0/1637227572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c1034b0 msgr2=0x7efe0c19f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c1034b0 0x7efe0c19f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 -- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 msgr2=0x7efe0c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 -- 172.21.15.69:0/1637227572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe0c19f710 con 0x7efe0c0ffb20 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe03fff640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:30.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe0affd640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 0x7efe0c197af0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00e9c0 tx=0x7efdfc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:30.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc00cd80 con 0x7efe0c0ffb20 2024-05-09T09:16:30.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe0c19fa00 con 0x7efe0c0ffb20 2024-05-09T09:16:30.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efdfc010910 con 0x7efe0c0ffb20 2024-05-09T09:16:30.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efdfc018980 con 0x7efe0c0ffb20 2024-05-09T09:16:30.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.978+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe0c19ff50 con 0x7efe0c0ffb20 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.979+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe0c067dc0 con 0x7efe0c0ffb20 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.980+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7efdfc01f020 con 0x7efe0c0ffb20 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.981+0000 7efe109bf640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 0x7efdf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.981+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(82..82 src has 1..82) ==== 7497+0+0 (secure 0 0 0) 0x7efdfc0ced00 con 0x7efe0c0ffb20 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.981+0000 7efe03fff640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 0x7efdf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:30.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.981+0000 7efe03fff640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 0x7efdf80a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efe0c199000 tx=0x7efdf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:30.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:30.986+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efdfc09bf70 con 0x7efe0c0ffb20 2024-05-09T09:16:31.116 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:30 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1477330070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T09:16:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:30 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T09:16:31.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.222+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) -- 0x7efe0c062430 con 0x7efe0c0ffb20 2024-05-09T09:16:31.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:30 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1477330070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T09:16:31.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:30 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T09:16:31.304 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:30 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1477330070' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-05-09T09:16:31.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:30 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-05-09T09:16:31.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.863+0000 7efe109bf640 1 -- 172.21.15.69:0/1637227572 <== mon.1 v2:172.21.15.163:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) ==== 171+0+0 (secure 0 0 0) 0x7efdfc09b870 con 0x7efe0c0ffb20 2024-05-09T09:16:31.862 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' created 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 msgr2=0x7efdf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 0x7efdf80a5050 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7efe0c199000 tx=0x7efdf4005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 msgr2=0x7efe0c197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 0x7efe0c197af0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7efdfc00e9c0 tx=0x7efdfc00ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 shutdown_connections 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efdf80a2b90 0x7efdf80a5050 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe0c1034b0 0x7efe0c19f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe0c102af0 0x7efe0c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 --2- 172.21.15.69:0/1637227572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe0c0ffb20 0x7efe0c197af0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 >> 172.21.15.69:0/1637227572 conn(0x7efe0c0fb450 msgr2=0x7efe0c105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:31.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 shutdown_connections 2024-05-09T09:16:31.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.866+0000 7efe119c1640 1 -- 172.21.15.69:0/1637227572 wait complete. 2024-05-09T09:16:31.882 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-05-09T09:16:31.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 -- 172.21.15.69:0/1890387726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984101890 msgr2=0x7ff984101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 --2- 172.21.15.69:0/1890387726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984101890 0x7ff984101cb0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff974009900 tx=0x7ff97402a410 comp rx=0 tx=0).stop 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 -- 172.21.15.69:0/1890387726 shutdown_connections 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 --2- 172.21.15.69:0/1890387726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984103440 0x7ff984109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 --2- 172.21.15.69:0/1890387726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984102a80 0x7ff984102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 --2- 172.21.15.69:0/1890387726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984101890 0x7ff984101cb0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 -- 172.21.15.69:0/1890387726 >> 172.21.15.69:0/1890387726 conn(0x7ff9840fce00 msgr2=0x7ff9840ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 -- 172.21.15.69:0/1890387726 shutdown_connections 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.987+0000 7ff98a529640 1 -- 172.21.15.69:0/1890387726 wait complete. 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 Processor -- start 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 -- start start 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984101890 0x7ff984197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:31.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984102a80 0x7ff984198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410d510 con 0x7ff984103440 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410d390 con 0x7ff984101890 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff98a529640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff98410d690 con 0x7ff984102a80 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff988ac6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff988ac6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46374/0 (socket says 172.21.15.69:46374) 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff988ac6640 1 -- 172.21.15.69:0/2575459282 learned_addr learned my addr 172.21.15.69:0/2575459282 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff983fff640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984101890 0x7ff984197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.988+0000 7ff97bfff640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984102a80 0x7ff984198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 -- 172.21.15.69:0/2575459282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984102a80 msgr2=0x7ff984198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984102a80 0x7ff984198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 -- 172.21.15.69:0/2575459282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984101890 msgr2=0x7ff984197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984101890 0x7ff984197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:31.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 -- 172.21.15.69:0/2575459282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff98419f770 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff988ac6640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00eb10 tx=0x7ff97c00ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c00ce20 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff97c005020 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff98419fa60 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff97c010690 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.989+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff98419ffb0 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.990+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff984108b50 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.991+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff97c005540 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.992+0000 7ff981ffb640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 0x7ff9700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.992+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(83..83 src has 1..83) ==== 7879+0+0 (secure 0 0 0) 0x7ff97c0cf900 con 0x7ff984103440 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.992+0000 7ff983fff640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 0x7ff9700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:31.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.992+0000 7ff983fff640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 0x7ff9700a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff974005ab0 tx=0x7ff974000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:31.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:31.995+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff97c09ca10 con 0x7ff984103440 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: pgmap v283: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1637227572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.209+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) -- 0x7ff98402d1c0 con 0x7ff984103440 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: pgmap v283: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1637227572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/565050168' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: osdmap e82: 12 total, 12 up, 12 in 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: pgmap v283: 33 pgs: 2 creating+peering, 1 active+clean, 30 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1637227572' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:16:32.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:16:32.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:16:32.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.856+0000 7ff981ffb640 1 -- 172.21.15.69:0/2575459282 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) ==== 172+0+0 (secure 0 0 0) 0x7ff97c09c310 con 0x7ff984103440 2024-05-09T09:16:32.855 INFO:teuthology.orchestra.run.smithi069.stderr:new fs with metadata pool 2 and data pool 3 2024-05-09T09:16:32.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 msgr2=0x7ff9700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:32.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 0x7ff9700a5050 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff974005ab0 tx=0x7ff974000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 msgr2=0x7ff98419f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:32.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ff97c00eb10 tx=0x7ff97c00ef10 comp rx=0 tx=0).stop 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 shutdown_connections 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff9700a2b90 0x7ff9700a5050 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff984103440 0x7ff98419f100 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff984102a80 0x7ff984198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 --2- 172.21.15.69:0/2575459282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff984101890 0x7ff984197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 >> 172.21.15.69:0/2575459282 conn(0x7ff9840fce00 msgr2=0x7ff984105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:32.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 shutdown_connections 2024-05-09T09:16:32.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.859+0000 7ff98a529640 1 -- 172.21.15.69:0/2575459282 wait complete. 2024-05-09T09:16:32.880 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-05-09T09:16:32.880 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 -- 172.21.15.69:0/1844246297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c102320 msgr2=0x7f541c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 --2- 172.21.15.69:0/1844246297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c102320 0x7f541c108bb0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f5414009ca0 tx=0x7f541402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 -- 172.21.15.69:0/1844246297 shutdown_connections 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 --2- 172.21.15.69:0/1844246297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c102320 0x7f541c108bb0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 --2- 172.21.15.69:0/1844246297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c101960 0x7f541c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 --2- 172.21.15.69:0/1844246297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c100770 0x7f541c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:32.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 -- 172.21.15.69:0/1844246297 >> 172.21.15.69:0/1844246297 conn(0x7f541c0fbce0 msgr2=0x7f541c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 -- 172.21.15.69:0/1844246297 shutdown_connections 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.997+0000 7f5420d6a640 1 -- 172.21.15.69:0/1844246297 wait complete. 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 Processor -- start 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 -- start start 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c100770 0x7f541c1047d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 0x7f541c103350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10c3f0 con 0x7f541c101960 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10c270 con 0x7f541c102320 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5420d6a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f541c10c570 con 0x7f541c100770 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5419d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5419d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46400/0 (socket says 172.21.15.69:46400) 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f5419d9b640 1 -- 172.21.15.69:0/347916492 learned_addr learned my addr 172.21.15.69:0/347916492 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f541ad9d640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 0x7f541c103350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:33.000 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.998+0000 7f541a59c640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c100770 0x7f541c1047d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 -- 172.21.15.69:0/347916492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c100770 msgr2=0x7f541c1047d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c100770 0x7f541c1047d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 -- 172.21.15.69:0/347916492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 msgr2=0x7f541c103350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 0x7f541c103350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 -- 172.21.15.69:0/347916492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f541c103be0 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f541ad9d640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 0x7f541c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5419d9b640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f541000d770 tx=0x7f541000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410017070 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5410005500 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f541c1a1c60 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5410004e00 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:32.999+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f541c1a21b0 con 0x7f541c101960 2024-05-09T09:16:33.001 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.000+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f541c1a2380 con 0x7f541c101960 2024-05-09T09:16:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.001+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5410005000 con 0x7f541c101960 2024-05-09T09:16:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.002+0000 7f540b7fe640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 0x7f54040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:33.002 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.002+0000 7f541a59c640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 0x7f54040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.002+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(84..84 src has 1..84) ==== 7979+0+0 (secure 0 0 0) 0x7f54100ce990 con 0x7f541c101960 2024-05-09T09:16:33.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.002+0000 7f541a59c640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 0x7f54040a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f541c1017c0 tx=0x7f540c005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:33.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.006+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f541009ba40 con 0x7f541c101960 2024-05-09T09:16:33.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.192+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) -- 0x7f541c1a2710 con 0x7f541c101960 2024-05-09T09:16:33.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:32 smithi069 ceph-mon[30123]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:32 smithi069 ceph-mon[30123]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T09:16:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:32 smithi069 ceph-mon[30123]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T09:16:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:32 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T09:16:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:32 smithi069 ceph-mon[30123]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T09:16:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:32 smithi163 ceph-mon[34354]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:32 smithi163 ceph-mon[34354]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T09:16:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:32 smithi163 ceph-mon[34354]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T09:16:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:32 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T09:16:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:32 smithi163 ceph-mon[34354]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T09:16:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:32 smithi103 ceph-mon[34421]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:32 smithi103 ceph-mon[34421]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-05-09T09:16:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:32 smithi103 ceph-mon[34421]: osdmap e83: 12 total, 12 up, 12 in 2024-05-09T09:16:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:32 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-05-09T09:16:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:32 smithi103 ceph-mon[34421]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-05-09T09:16:33.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.875+0000 7f540b7fe640 1 -- 172.21.15.69:0/347916492 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) ==== 157+0+0 (secure 0 0 0) 0x7f541009b340 con 0x7f541c101960 2024-05-09T09:16:33.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 msgr2=0x7f54040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:33.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 0x7f54040a50d0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f541c1017c0 tx=0x7f540c005c60 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 msgr2=0x7f541c102e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f541000d770 tx=0x7f541000dc30 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 shutdown_connections 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54040a2c10 0x7f54040a50d0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f541c102320 0x7f541c103350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f541c101960 0x7f541c102e10 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 --2- 172.21.15.69:0/347916492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f541c100770 0x7f541c1047d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.878+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 >> 172.21.15.69:0/347916492 conn(0x7f541c0fbce0 msgr2=0x7f541c069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.879+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 shutdown_connections 2024-05-09T09:16:33.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:33.879+0000 7f5420d6a640 1 -- 172.21.15.69:0/347916492 wait complete. 2024-05-09T09:16:33.890 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-05-09T09:16:34.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 -- 172.21.15.69:0/1060267994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc102360 msgr2=0x7fccdc108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:34.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 --2- 172.21.15.69:0/1060267994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc102360 0x7fccdc108bf0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fccd4009ca0 tx=0x7fccd402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 -- 172.21.15.69:0/1060267994 shutdown_connections 2024-05-09T09:16:34.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 --2- 172.21.15.69:0/1060267994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc102360 0x7fccdc108bf0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 --2- 172.21.15.69:0/1060267994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1019a0 0x7fccdc101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 --2- 172.21.15.69:0/1060267994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc1007b0 0x7fccdc100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 -- 172.21.15.69:0/1060267994 >> 172.21.15.69:0/1060267994 conn(0x7fccdc0fbd20 msgr2=0x7fccdc0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 -- 172.21.15.69:0/1060267994 shutdown_connections 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 -- 172.21.15.69:0/1060267994 wait complete. 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.004+0000 7fcce25ed640 1 Processor -- start 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- start start 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 0x7fccdc06a730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc102360 0x7fccdc1a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10c430 con 0x7fccdc1019a0 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10c2b0 con 0x7fccdc1007b0 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fccdc10c5b0 con 0x7fccdc102360 2024-05-09T09:16:34.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46418/0 (socket says 172.21.15.69:46418) 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 -- 172.21.15.69:0/3668419435 learned_addr learned my addr 172.21.15.69:0/3668419435 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccdbfff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 0x7fccdc06a730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce0b8a640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc102360 0x7fccdc1a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 -- 172.21.15.69:0/3668419435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc102360 msgr2=0x7fccdc1a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc102360 0x7fccdc1a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 -- 172.21.15.69:0/3668419435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 msgr2=0x7fccdc06a730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 0x7fccdc06a730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 -- 172.21.15.69:0/3668419435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fccdc1a1a30 con 0x7fccdc1019a0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccdbfff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 0x7fccdc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd3fff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d950 tx=0x7fccc400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc4017030 con 0x7fccdc1019a0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fccc4004020 con 0x7fccdc1019a0 2024-05-09T09:16:34.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fccdc1a1cc0 con 0x7fccdc1019a0 2024-05-09T09:16:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fccc40047a0 con 0x7fccdc1019a0 2024-05-09T09:16:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.005+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fccdc1a21c0 con 0x7fccdc1019a0 2024-05-09T09:16:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.006+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fccdc107ad0 con 0x7fccdc1019a0 2024-05-09T09:16:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.007+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fccc40041d0 con 0x7fccdc1019a0 2024-05-09T09:16:34.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.008+0000 7fccd9ffb640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 0x7fccc80a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.008+0000 7fccdbfff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 0x7fccc80a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.008+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(85..85 src has 1..85) ==== 8277+0+0 (secure 0 0 0) 0x7fccc40ceb20 con 0x7fccdc1019a0 2024-05-09T09:16:34.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.008+0000 7fccdbfff640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 0x7fccc80a5080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fccdc101800 tx=0x7fcccc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:34.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.012+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fccc409ba50 con 0x7fccdc1019a0 2024-05-09T09:16:34.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:34.190+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) -- 0x7fccdc062460 con 0x7fccdc1019a0 2024-05-09T09:16:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T09:16:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T09:16:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T09:16:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T09:16:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: pgmap v286: 97 pgs: 15 creating+peering, 4 active+clean, 78 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:34.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:33 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: pgmap v286: 97 pgs: 15 creating+peering, 4 active+clean, 78 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:33 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2575459282' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: osdmap e84: 12 total, 12 up, 12 in 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: pgmap v286: 97 pgs: 15 creating+peering, 4 active+clean, 78 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-05-09T09:16:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:33 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: Cluster is now healthy 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:active 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:34 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: Cluster is now healthy 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:active 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:34 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T09:16:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: Cluster is now healthy 2024-05-09T09:16:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/347916492' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-05-09T09:16:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: osdmap e85: 12 total, 12 up, 12 in 2024-05-09T09:16:35.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] up:active 2024-05-09T09:16:35.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:35.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:34 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-05-09T09:16:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.890+0000 7fccd9ffb640 1 -- 172.21.15.69:0/3668419435 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) ==== 228+0+0 (secure 0 0 0) 0x7fccc409b350 con 0x7fccdc1019a0 2024-05-09T09:16:35.889 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data_ec' created 2024-05-09T09:16:35.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 msgr2=0x7fccc80a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 0x7fccc80a5080 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fccdc101800 tx=0x7fcccc005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 msgr2=0x7fccdc19ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fccc400d950 tx=0x7fccc400de10 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 shutdown_connections 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fccc80a2bc0 0x7fccc80a5080 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fccdc102360 0x7fccdc1a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fccdc1019a0 0x7fccdc19ecf0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 --2- 172.21.15.69:0/3668419435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fccdc1007b0 0x7fccdc06a730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.894+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 >> 172.21.15.69:0/3668419435 conn(0x7fccdc0fbd20 msgr2=0x7fccdc104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:35.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.895+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 shutdown_connections 2024-05-09T09:16:35.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:35.895+0000 7fcce25ed640 1 -- 172.21.15.69:0/3668419435 wait complete. 2024-05-09T09:16:35.913 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-05-09T09:16:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- 172.21.15.69:0/317541616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 msgr2=0x7fe278101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- 172.21.15.69:0/317541616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278101de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009900 tx=0x7fe26c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- 172.21.15.69:0/317541616 shutdown_connections 2024-05-09T09:16:36.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- 172.21.15.69:0/317541616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe278102320 0x7fe278108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- 172.21.15.69:0/317541616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278101de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- 172.21.15.69:0/317541616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe278100770 0x7fe278100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- 172.21.15.69:0/317541616 >> 172.21.15.69:0/317541616 conn(0x7fe2780fbce0 msgr2=0x7fe2780fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- 172.21.15.69:0/317541616 shutdown_connections 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- 172.21.15.69:0/317541616 wait complete. 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 Processor -- start 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- start start 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe278100770 0x7fe27810b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:36.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe278102320 0x7fe278109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe27810d510 con 0x7fe278101960 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe27810d390 con 0x7fe278102320 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27fb63640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe27810d690 con 0x7fe278100770 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.024+0000 7fe27d0fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46432/0 (socket says 172.21.15.69:46432) 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 -- 172.21.15.69:0/1714171055 learned_addr learned my addr 172.21.15.69:0/1714171055 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 -- 172.21.15.69:0/1714171055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe278100770 msgr2=0x7fe27810b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe278100770 0x7fe27810b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 -- 172.21.15.69:0/1714171055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe278102320 msgr2=0x7fe278109c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:36.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe278102320 0x7fe278109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 -- 172.21.15.69:0/1714171055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe27810a500 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27d0fe640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009230 tx=0x7fe26c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c03f030 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe26c003ce0 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe2781a1bd0 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe26c035070 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.025+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe2781a2120 con 0x7fe278101960 2024-05-09T09:16:36.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.026+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe278107a90 con 0x7fe278101960 2024-05-09T09:16:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.027+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fe26c042030 con 0x7fe278101960 2024-05-09T09:16:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.028+0000 7fe26affd640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 0x7fe2640a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.028+0000 7fe27d8ff640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 0x7fe2640a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.028+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(87..87 src has 1..87) ==== 8770+0+0 (secure 0 0 0) 0x7fe26c0efd00 con 0x7fe278101960 2024-05-09T09:16:36.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.028+0000 7fe27d8ff640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 0x7fe2640a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2781017c0 tx=0x7fe274005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:36.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.033+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe26c0bca40 con 0x7fe278101960 2024-05-09T09:16:36.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.209+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) -- 0x7fe2780624a0 con 0x7fe278101960 2024-05-09T09:16:36.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:35 smithi069 ceph-mon[30123]: pgmap v289: 97 pgs: 15 creating+peering, 56 active+clean, 26 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-05-09T09:16:36.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:35 smithi069 ceph-mon[30123]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T09:16:36.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:35 smithi163 ceph-mon[34354]: pgmap v289: 97 pgs: 15 creating+peering, 56 active+clean, 26 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-05-09T09:16:36.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:35 smithi163 ceph-mon[34354]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T09:16:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:35 smithi103 ceph-mon[34421]: pgmap v289: 97 pgs: 15 creating+peering, 56 active+clean, 26 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 5 op/s 2024-05-09T09:16:36.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:35 smithi103 ceph-mon[34421]: osdmap e86: 12 total, 12 up, 12 in 2024-05-09T09:16:36.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.891+0000 7fe26affd640 1 -- 172.21.15.69:0/1714171055 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) ==== 169+0+0 (secure 0 0 0) 0x7fe26c0bc340 con 0x7fe278101960 2024-05-09T09:16:36.891 INFO:teuthology.orchestra.run.smithi069.stderr:set pool 4 allow_ec_overwrites to true 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.894+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 msgr2=0x7fe2640a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.894+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 0x7fe2640a5050 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fe2781017c0 tx=0x7fe274005c90 comp rx=0 tx=0).stop 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.894+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 msgr2=0x7fe278109730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.894+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009230 tx=0x7fe26c004360 comp rx=0 tx=0).stop 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 shutdown_connections 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe2640a2b90 0x7fe2640a5050 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe278102320 0x7fe278109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe278101960 0x7fe278109730 secure :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe26c009230 tx=0x7fe26c004360 comp rx=0 tx=0).stop 2024-05-09T09:16:36.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 --2- 172.21.15.69:0/1714171055 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe278100770 0x7fe27810b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:36.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.895+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 >> 172.21.15.69:0/1714171055 conn(0x7fe2780fbce0 msgr2=0x7fe278104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:36.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.896+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 shutdown_connections 2024-05-09T09:16:36.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:36.896+0000 7fe27fb63640 1 -- 172.21.15.69:0/1714171055 wait complete. 2024-05-09T09:16:36.920 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 -- 172.21.15.69:0/394744387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8103420 msgr2=0x7f41e8109cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 --2- 172.21.15.69:0/394744387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8103420 0x7f41e8109cb0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009ca0 tx=0x7f41e002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 -- 172.21.15.69:0/394744387 shutdown_connections 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 --2- 172.21.15.69:0/394744387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8103420 0x7f41e8109cb0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 --2- 172.21.15.69:0/394744387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 0x7f41e8102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 --2- 172.21.15.69:0/394744387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8101870 0x7f41e8101c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 -- 172.21.15.69:0/394744387 >> 172.21.15.69:0/394744387 conn(0x7f41e80fcde0 msgr2=0x7f41e80ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 -- 172.21.15.69:0/394744387 shutdown_connections 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 -- 172.21.15.69:0/394744387 wait complete. 2024-05-09T09:16:37.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.028+0000 7f41ee495640 1 Processor -- start 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 -- start start 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 0x7f41e8073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 0x7f41e8074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41e810d4f0 con 0x7f41e8101870 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41e810d370 con 0x7f41e8103420 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41ee495640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41e810d670 con 0x7f41e8102a60 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46450/0 (socket says 172.21.15.69:46450) 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 -- 172.21.15.69:0/2604049941 learned_addr learned my addr 172.21.15.69:0/2604049941 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41eca32640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 0x7f41e8074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e77fe640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 0x7f41e8073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 -- 172.21.15.69:0/2604049941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 msgr2=0x7f41e8073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:37.032 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 0x7f41e8073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 -- 172.21.15.69:0/2604049941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 msgr2=0x7f41e8074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 0x7f41e8074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 -- 172.21.15.69:0/2604049941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41e80749f0 con 0x7f41e8101870 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41eca32640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 0x7f41e8074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e7fff640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e9c0 tx=0x7f41d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.029+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d800cd80 con 0x7f41e8101870 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.030+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41d80035f0 con 0x7f41e8101870 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.030+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41e81a1c50 con 0x7f41e8101870 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.030+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41d8010690 con 0x7f41e8101870 2024-05-09T09:16:37.033 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.030+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41e81a21a0 con 0x7f41e8101870 2024-05-09T09:16:37.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.031+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41e8108b90 con 0x7f41e8101870 2024-05-09T09:16:37.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.032+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f41d80037a0 con 0x7f41e8101870 2024-05-09T09:16:37.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.033+0000 7f41e57fa640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 0x7f41d00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.033+0000 7f41e77fe640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 0x7f41d00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.033+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(88..88 src has 1..88) ==== 8770+0+0 (secure 0 0 0) 0x7f41d80cf830 con 0x7f41e8101870 2024-05-09T09:16:37.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.033+0000 7f41e77fe640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 0x7f41d00a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f41dc005fd0 tx=0x7f41dc005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:37.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.037+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41d809c570 con 0x7f41e8101870 2024-05-09T09:16:37.181 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:36 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T09:16:37.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:36 smithi103 ceph-mon[34421]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T09:16:37.182 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:36 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T09:16:37.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.252+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) -- 0x7f41e802d420 con 0x7f41e8101870 2024-05-09T09:16:37.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:36 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T09:16:37.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:36 smithi069 ceph-mon[30123]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T09:16:37.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:36 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T09:16:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:36 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3668419435' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-05-09T09:16:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:36 smithi163 ceph-mon[34354]: osdmap e87: 12 total, 12 up, 12 in 2024-05-09T09:16:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:36 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-05-09T09:16:37.617 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:16:37 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: 2024-05-09T09:16:37.181+0000 7f72ba5e3700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-05-09T09:16:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.907+0000 7f41e57fa640 1 -- 172.21.15.69:0/2604049941 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v17) ==== 137+0+0 (secure 0 0 0) 0x7f41d809be70 con 0x7f41e8101870 2024-05-09T09:16:37.907 INFO:teuthology.orchestra.run.smithi069.stderr:added data pool 4 to fsmap 2024-05-09T09:16:37.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 msgr2=0x7f41d00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:37.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 0x7f41d00a5050 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f41dc005fd0 tx=0x7f41dc005850 comp rx=0 tx=0).stop 2024-05-09T09:16:37.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 msgr2=0x7f41e80755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:37.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f41d800e9c0 tx=0x7f41d800ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 shutdown_connections 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41d00a2b90 0x7f41d00a5050 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41e8103420 0x7f41e8074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41e8102a60 0x7f41e8073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 --2- 172.21.15.69:0/2604049941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41e8101870 0x7f41e80755e0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 >> 172.21.15.69:0/2604049941 conn(0x7f41e80fcde0 msgr2=0x7f41e8105d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 shutdown_connections 2024-05-09T09:16:37.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:37.909+0000 7f41ee495640 1 -- 172.21.15.69:0/2604049941 wait complete. 2024-05-09T09:16:37.921 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:38.025 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- 172.21.15.69:0/777920885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf8103570 msgr2=0x7fadf8109e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 --2- 172.21.15.69:0/777920885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf8103570 0x7fadf8109e00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fade0009930 tx=0x7fade002a420 comp rx=0 tx=0).stop 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- 172.21.15.69:0/777920885 shutdown_connections 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 --2- 172.21.15.69:0/777920885 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf8103570 0x7fadf8109e00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 --2- 172.21.15.69:0/777920885 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadf8102bb0 0x7fadf8103030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 --2- 172.21.15.69:0/777920885 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf80ffbe0 0x7fadf8100000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- 172.21.15.69:0/777920885 >> 172.21.15.69:0/777920885 conn(0x7fadf80fb510 msgr2=0x7fadf80fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- 172.21.15.69:0/777920885 shutdown_connections 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- 172.21.15.69:0/777920885 wait complete. 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 Processor -- start 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.023+0000 7fadfef21640 1 -- start start 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 0x7fadf8198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadf8103570 0x7fadf819f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810d630 con 0x7fadf80ffbe0 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810d4b0 con 0x7fadf8103570 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fadf810d7b0 con 0x7fadf8102bb0 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38682/0 (socket says 172.21.15.69:38682) 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 -- 172.21.15.69:0/2426229617 learned_addr learned my addr 172.21.15.69:0/2426229617 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadf7fff640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 0x7fadf8198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 -- 172.21.15.69:0/2426229617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 msgr2=0x7fadf8198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 0x7fadf8198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 -- 172.21.15.69:0/2426229617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadf8103570 msgr2=0x7fadf819f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadf8103570 0x7fadf819f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 -- 172.21.15.69:0/2426229617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fadf819f810 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadf7fff640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 0x7fadf8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfccbd640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b450 tx=0x7fadf000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf0015020 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fadf0011d70 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fadf819fb00 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.024+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fadf000f340 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.025+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fadf81a0050 con 0x7fadf80ffbe0 2024-05-09T09:16:38.028 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.026+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fadf8108c80 con 0x7fadf80ffbe0 2024-05-09T09:16:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.026+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fadf0018020 con 0x7fadf80ffbe0 2024-05-09T09:16:38.029 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.027+0000 7fadf5ffb640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 0x7fade40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.027+0000 7fadf7fff640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 0x7fade40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.027+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fadf00ced80 con 0x7fadf80ffbe0 2024-05-09T09:16:38.030 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.028+0000 7fadf7fff640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 0x7fade40a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadf8199100 tx=0x7fade8008350 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.031 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.032+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fadf001f020 con 0x7fadf80ffbe0 2024-05-09T09:16:38.247 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.248+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fadf8062430 con 0x7fadf80ffbe0 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:37 smithi103 ceph-mon[34421]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:37 smithi103 ceph-mon[34421]: pgmap v292: 129 pgs: 12 creating+peering, 95 active+clean, 22 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:37 smithi163 ceph-mon[34354]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T09:16:38.370 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:37 smithi163 ceph-mon[34354]: pgmap v292: 129 pgs: 12 creating+peering, 95 active+clean, 22 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-09T09:16:38.371 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T09:16:38.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.248+0000 7fadf5ffb640 1 -- 172.21.15.69:0/2426229617 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11806 (secure 0 0 0) 0x7fadf000d040 con 0x7fadf80ffbe0 2024-05-09T09:16:38.371 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:38.371 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:37.254629+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-09T09:16:38.372 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 17 2024-05-09T09:16:38.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 msgr2=0x7fade40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 0x7fade40a50d0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fadf8199100 tx=0x7fade8008350 comp rx=0 tx=0).stop 2024-05-09T09:16:38.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 msgr2=0x7fadf8197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fadf000b450 tx=0x7fadf000b910 comp rx=0 tx=0).stop 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 shutdown_connections 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fade40a2c10 0x7fade40a50d0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fadf8103570 0x7fadf819f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fadf8102bb0 0x7fadf8198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 --2- 172.21.15.69:0/2426229617 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fadf80ffbe0 0x7fadf8197bf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 >> 172.21.15.69:0/2426229617 conn(0x7fadf80fb510 msgr2=0x7fadf8105e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.250+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 shutdown_connections 2024-05-09T09:16:38.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.251+0000 7fadfef21640 1 -- 172.21.15.69:0/2426229617 wait complete. 2024-05-09T09:16:38.373 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1714171055' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-05-09T09:16:38.374 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:37 smithi069 ceph-mon[30123]: osdmap e88: 12 total, 12 up, 12 in 2024-05-09T09:16:38.374 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:37 smithi069 ceph-mon[30123]: pgmap v292: 129 pgs: 12 creating+peering, 95 active+clean, 22 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s wr, 10 op/s 2024-05-09T09:16:38.374 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-05-09T09:16:38.375 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:38.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 -- 172.21.15.69:0/2173283495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4103440 msgr2=0x7fede4109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 --2- 172.21.15.69:0/2173283495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4103440 0x7fede4109cd0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7feddc009ca0 tx=0x7feddc02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 -- 172.21.15.69:0/2173283495 shutdown_connections 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 --2- 172.21.15.69:0/2173283495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4103440 0x7fede4109cd0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 --2- 172.21.15.69:0/2173283495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4102a80 0x7fede4102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 --2- 172.21.15.69:0/2173283495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4101890 0x7fede4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.475+0000 7fedec229640 1 -- 172.21.15.69:0/2173283495 >> 172.21.15.69:0/2173283495 conn(0x7fede40fcde0 msgr2=0x7fede40ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- 172.21.15.69:0/2173283495 shutdown_connections 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- 172.21.15.69:0/2173283495 wait complete. 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 Processor -- start 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- start start 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 0x7fede4197cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 0x7fede419f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fede410d600 con 0x7fede4102a80 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fede410d480 con 0x7fede4101890 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fedec229640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fede410d780 con 0x7fede4103440 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fede97c4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fede97c4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38704/0 (socket says 172.21.15.69:38704) 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fede97c4640 1 -- 172.21.15.69:0/1898749796 learned_addr learned my addr 172.21.15.69:0/1898749796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.476+0000 7fede9fc5640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 0x7fede4197cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 -- 172.21.15.69:0/1898749796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 msgr2=0x7fede419f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fedea7c6640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 0x7fede419f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 0x7fede419f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 -- 172.21.15.69:0/1898749796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 msgr2=0x7fede4197cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 0x7fede4197cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 -- 172.21.15.69:0/1898749796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fede419f900 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede9fc5640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 0x7fede4197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fedea7c6640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 0x7fede419f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fede97c4640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fedd400b4a0 tx=0x7fedd400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4014070 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedd4010af0 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fede419fbf0 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedd4017d10 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.477+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fede41a0140 con 0x7fede4102a80 2024-05-09T09:16:38.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.478+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fede4108c00 con 0x7fede4102a80 2024-05-09T09:16:38.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.479+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fedd4017070 con 0x7fede4102a80 2024-05-09T09:16:38.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.480+0000 7feddaffd640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 0x7fedd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.480+0000 7fede9fc5640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 0x7fedd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.480+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fedd40ceb20 con 0x7fede4102a80 2024-05-09T09:16:38.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.480+0000 7fede9fc5640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 0x7fedd00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fede41991c0 tx=0x7fede0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.483+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fedd40cee20 con 0x7fede4102a80 2024-05-09T09:16:38.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.659+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fede402d410 con 0x7fede4102a80 2024-05-09T09:16:38.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.660+0000 7feddaffd640 1 -- 172.21.15.69:0/1898749796 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7fedd409b840 con 0x7fede4102a80 2024-05-09T09:16:38.660 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:38.661 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 msgr2=0x7fedd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 0x7fedd00a5050 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fede41991c0 tx=0x7fede0008040 comp rx=0 tx=0).stop 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 msgr2=0x7fede4198230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fedd400b4a0 tx=0x7fedd400b960 comp rx=0 tx=0).stop 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 shutdown_connections 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedd00a2b90 0x7fedd00a5050 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.662 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fede4103440 0x7fede419f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fede4102a80 0x7fede4198230 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 --2- 172.21.15.69:0/1898749796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fede4101890 0x7fede4197cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 >> 172.21.15.69:0/1898749796 conn(0x7fede40fcde0 msgr2=0x7fede4105dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 shutdown_connections 2024-05-09T09:16:38.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.662+0000 7fedec229640 1 -- 172.21.15.69:0/1898749796 wait complete. 2024-05-09T09:16:38.674 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:38.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 -- 172.21.15.69:0/88951773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640101880 msgr2=0x7ff640101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 --2- 172.21.15.69:0/88951773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640101880 0x7ff640101ca0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7ff630009ca0 tx=0x7ff63002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 -- 172.21.15.69:0/88951773 shutdown_connections 2024-05-09T09:16:38.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 --2- 172.21.15.69:0/88951773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640103430 0x7ff640109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 --2- 172.21.15.69:0/88951773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640102a70 0x7ff640102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 --2- 172.21.15.69:0/88951773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640101880 0x7ff640101ca0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 -- 172.21.15.69:0/88951773 >> 172.21.15.69:0/88951773 conn(0x7ff6400fce30 msgr2=0x7ff6400ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 -- 172.21.15.69:0/88951773 shutdown_connections 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.774+0000 7ff6454b3640 1 -- 172.21.15.69:0/88951773 wait complete. 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 Processor -- start 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 -- start start 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640101880 0x7ff640197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 0x7ff6401980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.778 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff64010d500 con 0x7ff640103430 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff64010d380 con 0x7ff640101880 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff6454b3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff64010d680 con 0x7ff640102a70 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff63f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff63f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38720/0 (socket says 172.21.15.69:38720) 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff63f7fe640 1 -- 172.21.15.69:0/359748535 learned_addr learned my addr 172.21.15.69:0/359748535 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.775+0000 7ff63effd640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640101880 0x7ff640197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63e7fc640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 0x7ff6401980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 -- 172.21.15.69:0/359748535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 msgr2=0x7ff6401980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 0x7ff6401980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 -- 172.21.15.69:0/359748535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640101880 msgr2=0x7ff640197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640101880 0x7ff640197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 -- 172.21.15.69:0/359748535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff64019f7a0 con 0x7ff640103430 2024-05-09T09:16:38.779 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63e7fc640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 0x7ff6401980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff63f7fe640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff63800e970 tx=0x7ff63800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff63800ccc0 con 0x7ff640103430 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff63800ce60 con 0x7ff640103430 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff64019fa90 con 0x7ff640103430 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff638010490 con 0x7ff640103430 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.776+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff64019ffe0 con 0x7ff640103430 2024-05-09T09:16:38.780 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.777+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff640108ba0 con 0x7ff640103430 2024-05-09T09:16:38.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.778+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7ff638005020 con 0x7ff640103430 2024-05-09T09:16:38.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.779+0000 7ff637fff640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 0x7ff62c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:38.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.779+0000 7ff63effd640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 0x7ff62c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:38.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.780+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff6380cf6c0 con 0x7ff640103430 2024-05-09T09:16:38.781 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.780+0000 7ff63effd640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 0x7ff62c0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff630006010 tx=0x7ff630000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:38.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.784+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff63809c3e0 con 0x7ff640103430 2024-05-09T09:16:38.959 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.960+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7ff640062430 con 0x7ff640103430 2024-05-09T09:16:38.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.961+0000 7ff637fff640 1 -- 172.21.15.69:0/359748535 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7ff63809bce0 con 0x7ff640103430 2024-05-09T09:16:38.962 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:38.962 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:38.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 msgr2=0x7ff62c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 0x7ff62c0a5050 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff630006010 tx=0x7ff630000f40 comp rx=0 tx=0).stop 2024-05-09T09:16:38.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 msgr2=0x7ff64019f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7ff63800e970 tx=0x7ff63800ed70 comp rx=0 tx=0).stop 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 shutdown_connections 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff62c0a2b90 0x7ff62c0a5050 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff640103430 0x7ff64019f130 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff640102a70 0x7ff6401980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 --2- 172.21.15.69:0/359748535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff640101880 0x7ff640197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 >> 172.21.15.69:0/359748535 conn(0x7ff6400fce30 msgr2=0x7ff640105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 shutdown_connections 2024-05-09T09:16:38.964 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:38.963+0000 7ff6454b3640 1 -- 172.21.15.69:0/359748535 wait complete. 2024-05-09T09:16:38.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:38.973 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T09:16:38.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T09:16:38.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:38.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2426229617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:38.974 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1898749796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:38.991 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:39.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 -- 172.21.15.69:0/2270150229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 msgr2=0x7f7b98100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2270150229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100f00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b80009900 tx=0x7f7b8002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 -- 172.21.15.69:0/2270150229 shutdown_connections 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2270150229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b98108ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2270150229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100f00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2270150229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 0x7f7b98102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 -- 172.21.15.69:0/2270150229 >> 172.21.15.69:0/2270150229 conn(0x7f7b980fb510 msgr2=0x7f7b980fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 -- 172.21.15.69:0/2270150229 shutdown_connections 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.092+0000 7f7b9caee640 1 -- 172.21.15.69:0/2270150229 wait complete. 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 Processor -- start 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 -- start start 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b9819c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 0x7f7b9819ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810c490 con 0x7f7b98100a80 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810c310 con 0x7f7b98101e30 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9caee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7b9810c610 con 0x7f7b98101440 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38744/0 (socket says 172.21.15.69:38744) 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.093+0000 7f7b9659c640 1 -- 172.21.15.69:0/2197787948 learned_addr learned my addr 172.21.15.69:0/2197787948 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b8dd9b640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b9819c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b96d9d640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 0x7f7b9819ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 -- 172.21.15.69:0/2197787948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 msgr2=0x7f7b9819c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b9819c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 -- 172.21.15.69:0/2197787948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 msgr2=0x7f7b9819ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 0x7f7b9819ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 -- 172.21.15.69:0/2197787948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7b9819f820 con 0x7f7b98100a80 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b8dd9b640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b9819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:39.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9659c640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800e9c0 tx=0x7f7b8800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b8800cd80 con 0x7f7b98100a80 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7b9819fb10 con 0x7f7b98100a80 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7b880035f0 con 0x7f7b98100a80 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7b88010690 con 0x7f7b98100a80 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.094+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7b981a0030 con 0x7f7b98100a80 2024-05-09T09:16:39.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.095+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7b98068aa0 con 0x7f7b98100a80 2024-05-09T09:16:39.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.096+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f7b880037a0 con 0x7f7b98100a80 2024-05-09T09:16:39.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.097+0000 7f7b8ffff640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.097+0000 7f7b8dd9b640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.097+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7b880cf7c0 con 0x7f7b98100a80 2024-05-09T09:16:39.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.097+0000 7f7b8dd9b640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 0x7f7b840a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006010 tx=0x7f7b80032cb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.101+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7b8809c4e0 con 0x7f7b98100a80 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2426229617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1898749796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:39.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.316+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f7b98107bc0 con 0x7f7b98100a80 2024-05-09T09:16:39.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.317+0000 7f7b8ffff640 1 -- 172.21.15.69:0/2197787948 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) ==== 94+0+11806 (secure 0 0 0) 0x7f7b8809bde0 con 0x7f7b98100a80 2024-05-09T09:16:39.317 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:39.317 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":17,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3},{"gid":24458,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":6},{"gid":24478,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":7},{"gid":24490,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":8},{"gid":24491,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":9},{"gid":24511,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":17,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:37.254629+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":1,"in":[0],"up":{"mds_0":24527},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1,"qdb_leader":24527,"qdb_cluster":[24527]},"id":1}]} 2024-05-09T09:16:39.317 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 17 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 msgr2=0x7f7b840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 0x7f7b840a5050 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f7b80006010 tx=0x7f7b80032cb0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 msgr2=0x7f7b98100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f7b8800e9c0 tx=0x7f7b8800ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 shutdown_connections 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7b840a2b90 0x7f7b840a5050 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7b98101e30 0x7f7b9819ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7b98101440 0x7f7b9819c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 --2- 172.21.15.69:0/2197787948 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7b98100a80 0x7f7b98100540 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 >> 172.21.15.69:0/2197787948 conn(0x7f7b980fb510 msgr2=0x7f7b98104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 shutdown_connections 2024-05-09T09:16:39.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.319+0000 7f7b9caee640 1 -- 172.21.15.69:0/2197787948 wait complete. 2024-05-09T09:16:39.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: osdmap e89: 12 total, 12 up, 12 in 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2604049941' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2426229617' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1898749796' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 -- 172.21.15.69:0/929132406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688100770 msgr2=0x7f6688100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 --2- 172.21.15.69:0/929132406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688100770 0x7f6688100b90 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f6678009900 tx=0x7f667802a410 comp rx=0 tx=0).stop 2024-05-09T09:16:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 -- 172.21.15.69:0/929132406 shutdown_connections 2024-05-09T09:16:39.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 --2- 172.21.15.69:0/929132406 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6688102320 0x7f6688108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 --2- 172.21.15.69:0/929132406 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688101960 0x7f6688101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 --2- 172.21.15.69:0/929132406 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688100770 0x7f6688100b90 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 -- 172.21.15.69:0/929132406 >> 172.21.15.69:0/929132406 conn(0x7f66880fbce0 msgr2=0x7f66880fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 -- 172.21.15.69:0/929132406 shutdown_connections 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.431+0000 7f668d163640 1 -- 172.21.15.69:0/929132406 wait complete. 2024-05-09T09:16:39.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 Processor -- start 2024-05-09T09:16:39.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 -- start start 2024-05-09T09:16:39.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6688100770 0x7f6688196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 0x7f668819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c410 con 0x7f6688101960 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c290 con 0x7f6688102320 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668d163640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f668810c590 con 0x7f6688100770 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38762/0 (socket says 172.21.15.69:38762) 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668659c640 1 -- 172.21.15.69:0/3143582119 learned_addr learned my addr 172.21.15.69:0/3143582119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.432+0000 7f668759e640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 0x7f668819e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 -- 172.21.15.69:0/3143582119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6688100770 msgr2=0x7f6688196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6688100770 0x7f6688196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 -- 172.21.15.69:0/3143582119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 msgr2=0x7f668819e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 0x7f668819e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 -- 172.21.15.69:0/3143582119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f668819e6a0 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668759e640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 0x7f668819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668659c640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e9f0 tx=0x7f667c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c00cd80 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f668819e990 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f667c010910 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f667c0188f0 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.433+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f668819eee0 con 0x7f6688101960 2024-05-09T09:16:39.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.434+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6688107a90 con 0x7f6688101960 2024-05-09T09:16:39.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.435+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f667c010ac0 con 0x7f6688101960 2024-05-09T09:16:39.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.436+0000 7f666ffff640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 0x7f66700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.436+0000 7f6686d9d640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 0x7f66700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.436+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f667c0cf790 con 0x7f6688101960 2024-05-09T09:16:39.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.437+0000 7f6686d9d640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 0x7f66700a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f66780057c0 tx=0x7f6678031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.441+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f667c09c4b0 con 0x7f6688101960 2024-05-09T09:16:39.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.618+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6688062460 con 0x7f6688101960 2024-05-09T09:16:39.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.619+0000 7f666ffff640 1 -- 172.21.15.69:0/3143582119 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f667c09bdb0 con 0x7f6688101960 2024-05-09T09:16:39.619 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:39.620 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:39.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 msgr2=0x7f66700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 0x7f66700a5050 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f66780057c0 tx=0x7f6678031400 comp rx=0 tx=0).stop 2024-05-09T09:16:39.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 msgr2=0x7f6688196fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f667c00e9f0 tx=0x7f667c00eeb0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 shutdown_connections 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f66700a2b90 0x7f66700a5050 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6688102320 0x7f668819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6688101960 0x7f6688196fc0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 --2- 172.21.15.69:0/3143582119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6688100770 0x7f6688196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.621+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 >> 172.21.15.69:0/3143582119 conn(0x7f66880fbce0 msgr2=0x7f6688104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.622+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 shutdown_connections 2024-05-09T09:16:39.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.622+0000 7f668d163640 1 -- 172.21.15.69:0/3143582119 wait complete. 2024-05-09T09:16:39.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 17, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-05-09T09:16:32.211035+0000', 'modified': '2024-05-09T09:16:37.254629+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24527}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/189860875', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 189860875}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 189860875}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1, 'qdb_leader': 24527, 'qdb_cluster': [24527]} 2024-05-09T09:16:39.633 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-05-09T09:16:39.634 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 -- 172.21.15.69:0/2584327374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 msgr2=0x7f25581099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 --2- 172.21.15.69:0/2584327374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f25581099c0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f2550009ca0 tx=0x7f255002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 -- 172.21.15.69:0/2584327374 shutdown_connections 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 --2- 172.21.15.69:0/2584327374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f25581099c0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 --2- 172.21.15.69:0/2584327374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558104c40 0x7f2558107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 --2- 172.21.15.69:0/2584327374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 0x7f2558104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 -- 172.21.15.69:0/2584327374 >> 172.21.15.69:0/2584327374 conn(0x7f25580fbd20 msgr2=0x7f25580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 -- 172.21.15.69:0/2584327374 shutdown_connections 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.735+0000 7f255dbb6640 1 -- 172.21.15.69:0/2584327374 wait complete. 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 Processor -- start 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 -- start start 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 0x7f2558197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558104c40 0x7f25581980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f255810d530 con 0x7f2558107570 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f255810d3b0 con 0x7f2558102310 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f255dbb6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f255810d6b0 con 0x7f2558104c40 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38772/0 (socket says 172.21.15.69:38772) 2024-05-09T09:16:39.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 -- 172.21.15.69:0/1620564835 learned_addr learned my addr 172.21.15.69:0/1620564835 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f25577fe640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 0x7f2558197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 -- 172.21.15.69:0/1620564835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558104c40 msgr2=0x7f25581980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558104c40 0x7f25581980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 -- 172.21.15.69:0/1620564835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 msgr2=0x7f2558197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 0x7f2558197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.736+0000 7f2557fff640 1 -- 172.21.15.69:0/1620564835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f255819f790 con 0x7f2558107570 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f2557fff640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2550031ae0 tx=0x7f2550031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f2554ff9640 1 -- 172.21.15.69:0/1620564835 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2550003c30 con 0x7f2558107570 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f2554ff9640 1 -- 172.21.15.69:0/1620564835 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2550003dd0 con 0x7f2558107570 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f255819fa20 con 0x7f2558107570 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f2554ff9640 1 -- 172.21.15.69:0/1620564835 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f255002fcb0 con 0x7f2558107570 2024-05-09T09:16:39.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.737+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f255819ff70 con 0x7f2558107570 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.739+0000 7f2554ff9640 1 -- 172.21.15.69:0/1620564835 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2550004050 con 0x7f2558107570 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.739+0000 7f256099b740 1 Processor -- start 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.739+0000 7f256099b740 1 -- start start 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fede8c0 0x56526fedece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 0x56526fee3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56526fe424d0 con 0x56526fee4420 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56526fdea3e0 con 0x56526fede8c0 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f256099b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56526fe3e490 con 0x56526fedf2b0 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38778/0 (socket says 172.21.15.69:38778) 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 -- 172.21.15.69:0/1510307003 learned_addr learned my addr 172.21.15.69:0/1510307003 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f25577fe640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 0x56526fee3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fede8c0 0x56526fedece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 -- 172.21.15.69:0/1510307003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 msgr2=0x56526fee3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2554ff9640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 0x7f25440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 0x56526fee3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 -- 172.21.15.69:0/1510307003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fede8c0 msgr2=0x56526fedece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fede8c0 0x56526fedece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 -- 172.21.15.69:0/1510307003 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56526fb562d0 con 0x56526fee4420 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f25577fe640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 0x56526fee3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2557fff640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 0x7f25440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2556ffd640 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2540005d00 tx=0x7f254000d540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ef1dd21d6140be0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f2554ff9640 1 -- 172.21.15.69:0/1620564835 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f25500f0550 con 0x7f2558107570 2024-05-09T09:16:39.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f254dffb640 1 -- 172.21.15.69:0/1510307003 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540012510 con 0x56526fee4420 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f254dffb640 1 -- 172.21.15.69:0/1510307003 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2540012b90 con 0x56526fee4420 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.740+0000 7f254dffb640 1 -- 172.21.15.69:0/1510307003 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540011af0 con 0x56526fee4420 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- 172.21.15.69:0/1510307003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 msgr2=0x56526fee68b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f2557fff640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 0x7f25440a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2550031a40 tx=0x7f25500e7700 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f2540005d00 tx=0x7f254000d540 comp rx=0 tx=0).stop 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- 172.21.15.69:0/1510307003 shutdown_connections 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fee68b0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fedf2b0 0x56526fee3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 --2- 172.21.15.69:0/1510307003 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fede8c0 0x56526fedece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- 172.21.15.69:0/1510307003 >> 172.21.15.69:0/1510307003 conn(0x56526fed9f30 msgr2=0x56526feda330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- 172.21.15.69:0/1510307003 shutdown_connections 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- 172.21.15.69:0/1510307003 wait complete. 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.741+0000 7f256099b740 1 -- start start 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 Processor -- start 2024-05-09T09:16:39.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fede8c0 0x56526fee8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee92a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56526fee73d0 con 0x56526fee4420 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56526fdfd8e0 con 0x56526fedf2b0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56526fee7550 con 0x56526fede8c0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f256099b740 10 client.0 _ll_register_callbacks cb 0x56526fdca9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38794/0 (socket says 172.21.15.69:38794) 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 -- 172.21.15.69:0/2847547773 learned_addr learned my addr 172.21.15.69:0/2847547773 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f25577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee92a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fede8c0 0x56526fee8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 -- 172.21.15.69:0/2847547773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fede8c0 msgr2=0x56526fee8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fede8c0 0x56526fee8d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 -- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 msgr2=0x56526fee92a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee92a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 -- 172.21.15.69:0/2847547773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56526fc206b0 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f25577fe640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2540005fa0 tx=0x7f2540001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0c91b29830bdf9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.742+0000 7f2536ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25400140f0 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2540012510 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2540012b90 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2540014290 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 10 client.14751 fetch_fsmap learned FSMap version 17 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56526fef32a0 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2540012b90 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f254001d480 con 0x56526fee4420 2024-05-09T09:16:39.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 10 client.14751 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 10 client.14751 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 20 client.14751 populate_metadata read hostname 'smithi069' 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526fef5570 0x56526fef69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 4 client.14751 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56526fe4fc00 con 0x56526fef5570 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.743+0000 7f25577fe640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526fef5570 0x56526fef69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.744+0000 7f25577fe640 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526fef5570 0x56526fef69e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5e4f34f1cb37fc9 server_cookie=ab0913ed95d507c8 in_seq=0 out_seq=0 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.744+0000 7f2536ffd640 10 client.14751 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.744+0000 7f2536ffd640 1 -- 172.21.15.69:0/2847547773 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56526fe4fc00 con 0x56526fef5570 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.744+0000 7f2536ffd640 10 client.14751 handle_command_reply: tid=0 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 client.14751 shutdown 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 msgr2=0x56526fef02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f2540005fa0 tx=0x7f2540001e70 comp rx=0 tx=0).stop 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 shutdown_connections 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526fef02a0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee92a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526fef5570 0x56526fef69e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 --2- 172.21.15.69:0/2847547773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fede8c0 0x56526fee8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f2536ffd640 0 client.14751 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 >> 172.21.15.69:0/2847547773 conn(0x56526fed9f30 msgr2=0x56526fed9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 shutdown_connections 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 1 -- 172.21.15.69:0/2847547773 wait complete. 2024-05-09T09:16:39.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.745+0000 7f256099b740 20 client.14751 trim_cache size 0 max 16384 2024-05-09T09:16:39.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 Processor -- start 2024-05-09T09:16:39.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 -- start start 2024-05-09T09:16:39.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee3ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fee4420 0x56526fed8770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56526fef4d20 con 0x56526fede8c0 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56526fef53f0 con 0x56526fedf2b0 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f256099b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56526fef5ac0 con 0x56526fee4420 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38806/0 (socket says 172.21.15.69:38806) 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 -- 172.21.15.69:0/4187125837 learned_addr learned my addr 172.21.15.69:0/4187125837 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 -- 172.21.15.69:0/4187125837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fee4420 msgr2=0x56526fed8770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fee4420 0x56526fed8770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 -- 172.21.15.69:0/4187125837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 msgr2=0x56526fee3ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee3ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 -- 172.21.15.69:0/4187125837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56526fee3960 con 0x56526fede8c0 2024-05-09T09:16:39.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f2556ffd640 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2540004700 tx=0x7f2540016f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7fe9877f7a0449c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.763+0000 7f254d7fa640 1 -- 172.21.15.69:0/4187125837 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2540005850 con 0x56526fede8c0 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f254d7fa640 1 -- 172.21.15.69:0/4187125837 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2540014290 con 0x56526fede8c0 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f254d7fa640 1 -- 172.21.15.69:0/4187125837 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25400129f0 con 0x56526fede8c0 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- 172.21.15.69:0/4187125837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 msgr2=0x56526fee9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f2540004700 tx=0x7f2540016f50 comp rx=0 tx=0).stop 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- 172.21.15.69:0/4187125837 shutdown_connections 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526fee4420 0x56526fed8770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fee3ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 --2- 172.21.15.69:0/4187125837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fede8c0 0x56526fee9200 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- 172.21.15.69:0/4187125837 >> 172.21.15.69:0/4187125837 conn(0x56526fed9f30 msgr2=0x56526fee8af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- 172.21.15.69:0/4187125837 shutdown_connections 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- 172.21.15.69:0/4187125837 wait complete. 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.764+0000 7f256099b740 1 -- start start 2024-05-09T09:16:39.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 Processor -- start 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fef02d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526ff5bab0 0x56526ff5df00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56526fef6860 con 0x56526fee4420 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56526fef6190 con 0x56526fedf2b0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56526fef6f30 con 0x56526ff5bab0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 10 client.0 _ll_register_callbacks cb 0x7f2540014470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38814/0 (socket says 172.21.15.69:38814) 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 -- 172.21.15.69:0/2346535200 learned_addr learned my addr 172.21.15.69:0/2346535200 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fef02d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 -- 172.21.15.69:0/2346535200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526ff5bab0 msgr2=0x56526ff5df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526ff5bab0 0x56526ff5df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526ff5bab0 0x56526ff5df00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 -- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 msgr2=0x56526fef02d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fef02d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 -- 172.21.15.69:0/2346535200 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56526ff5e730 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fef02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f25577fe640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x56526fef7120 tx=0x7f2548012ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f66573e423a4fac8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2536ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f254800fa90 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f254800fc30 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.765+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2540014290 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=17) ==== 24+0+0 (secure 0 0 0) 0x7f2548031050 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 10 client.14757 fetch_fsmap learned FSMap version 17 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=17,monmap=4+}) -- 0x56526ff5fb30 con 0x56526fee4420 2024-05-09T09:16:39.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2540014290 con 0x56526fee4420 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 17) ==== 8342+0+0 (secure 0 0 0) 0x7f25480049d0 con 0x56526fee4420 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 10 client.14757 fetch_fsmap finished waiting for FSMap version 17 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 10 client.14757 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 20 client.14757 populate_metadata read hostname 'smithi069' 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526ff64740 0x56526ff63450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 4 client.14757 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "status"}) -- 0x56526fe49f50 con 0x56526ff64740 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526ff64740 0x56526ff63450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2556ffd640 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526ff64740 0x56526ff63450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b00ccb346fb15c73 server_cookie=e6f6e6951b60a51e in_seq=0 out_seq=0 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.766+0000 7f2536ffd640 10 client.14757 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f2536ffd640 1 -- 172.21.15.69:0/2346535200 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x56526fe49f50 con 0x56526ff64740 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f2536ffd640 10 client.14757 handle_command_reply: tid=0 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 client.14757 shutdown 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 msgr2=0x56526ff5b520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x56526fef7120 tx=0x7f2548012ce0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 shutdown_connections 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56526ff64740 0x56526ff63450 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56526ff5bab0 0x56526ff5df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56526fee4420 0x56526ff5b520 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 --2- 172.21.15.69:0/2346535200 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56526fedf2b0 0x56526fef02d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f2536ffd640 0 client.14757 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 >> 172.21.15.69:0/2346535200 conn(0x56526fed9f30 msgr2=0x56526fee7980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 shutdown_connections 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 1 -- 172.21.15.69:0/2346535200 wait complete. 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f256099b740 20 client.14757 trim_cache size 0 max 16384 2024-05-09T09:16:39.774 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24527, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 6.9095891250000001, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 17, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 89, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 13.026997831999999 2024-05-09T09:16:39.775 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:16:39.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 msgr2=0x7f25440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 0x7f25440a5050 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f2550031a40 tx=0x7f25500e7700 comp rx=0 tx=0).stop 2024-05-09T09:16:39.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 msgr2=0x7f255819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:39.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f2550031ae0 tx=0x7f2550031ee0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 shutdown_connections 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25440a2b90 0x7f25440a5050 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2558107570 0x7f255819f120 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2558104c40 0x7f25581980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 --2- 172.21.15.69:0/1620564835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2558102310 0x7f2558197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.767+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 >> 172.21.15.69:0/1620564835 conn(0x7f25580fbd20 msgr2=0x7f2558102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.768+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 shutdown_connections 2024-05-09T09:16:39.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:39.768+0000 7f255dbb6640 1 -- 172.21.15.69:0/1620564835 wait complete. 2024-05-09T09:16:39.778 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T09:16:39.778 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:39.779 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T09:16:40.019 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:16:40.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/359748535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:39 smithi163 ceph-mon[34354]: pgmap v294: 129 pgs: 10 creating+peering, 112 active+clean, 7 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-05-09T09:16:40.206 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2197787948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.207 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3143582119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/359748535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:39 smithi069 ceph-mon[30123]: pgmap v294: 129 pgs: 10 creating+peering, 112 active+clean, 7 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-05-09T09:16:40.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2197787948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3143582119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/359748535' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:39 smithi103 ceph-mon[34421]: pgmap v294: 129 pgs: 10 creating+peering, 112 active+clean, 7 unknown; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s wr, 5 op/s 2024-05-09T09:16:40.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2197787948' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:40.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3143582119' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:40 smithi069 ceph-mon[30123]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:40 smithi069 ceph-mon[30123]: Cluster is now healthy 2024-05-09T09:16:41.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:40 smithi163 ceph-mon[34354]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:41.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:40 smithi163 ceph-mon[34354]: Cluster is now healthy 2024-05-09T09:16:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:40 smithi103 ceph-mon[34421]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-05-09T09:16:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:40 smithi103 ceph-mon[34421]: Cluster is now healthy 2024-05-09T09:16:41.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 -- 172.21.15.69:0/1144638714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4101be0 msgr2=0x7fedf410e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 --2- 172.21.15.69:0/1144638714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4101be0 0x7fedf410e0d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fedec00b0a0 tx=0x7fedec00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 -- 172.21.15.69:0/1144638714 shutdown_connections 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 --2- 172.21.15.69:0/1144638714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4101be0 0x7fedf410e0d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 --2- 172.21.15.69:0/1144638714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4107290 0x7fedf4107670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 --2- 172.21.15.69:0/1144638714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 0x7fedf41016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 -- 172.21.15.69:0/1144638714 >> 172.21.15.69:0/1144638714 conn(0x7fedf40781d0 msgr2=0x7fedf40785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 -- 172.21.15.69:0/1144638714 shutdown_connections 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.833+0000 7fedf9185700 1 -- 172.21.15.69:0/1144638714 wait complete. 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.834+0000 7fedf9185700 1 Processor -- start 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.834+0000 7fedf9185700 1 -- start start 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.834+0000 7fedf9185700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 0x7fedf4199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.834+0000 7fedf9185700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4101be0 0x7fedf4199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.834+0000 7fedf9185700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:41.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf9185700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedf4111650 con 0x7fedf4107290 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf9185700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41114d0 con 0x7fedf4101220 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf9185700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fedf41117d0 con 0x7fedf4101be0 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38826/0 (socket says 172.21.15.69:38826) 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 -- 172.21.15.69:0/2220619952 learned_addr learned my addr 172.21.15.69:0/2220619952 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf259c700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4101be0 0x7fedf4199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf2d9d700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 0x7fedf4199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 -- 172.21.15.69:0/2220619952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4101be0 msgr2=0x7fedf4199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4101be0 0x7fedf4199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 -- 172.21.15.69:0/2220619952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 msgr2=0x7fedf4199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 0x7fedf4199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:41.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 -- 172.21.15.69:0/2220619952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedf41a4fd0 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf359e700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fedec00d010 tx=0x7fedec00bde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec021070 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedf41a51a0 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedec007c00 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedec0048b0 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.835+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedf41a55f0 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.836+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fedf4109800 con 0x7fedf4107290 2024-05-09T09:16:41.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.837+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fedec004000 con 0x7fedf4107290 2024-05-09T09:16:41.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.838+0000 7feddbfff700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 0x7feddc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:41.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.838+0000 7fedf2d9d700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 0x7feddc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:41.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.838+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fedec0d0300 con 0x7fedf4107290 2024-05-09T09:16:41.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.838+0000 7fedf2d9d700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 0x7feddc0a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fede4009a10 tx=0x7fede4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:41.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:41.842+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fedec007db0 con 0x7fedf4107290 2024-05-09T09:16:42.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.085+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7fedf406a5d0 con 0x7fedf4107290 2024-05-09T09:16:42.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.086+0000 7feddbfff700 1 -- 172.21.15.69:0/2220619952 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v17) ==== 53+0+88 (secure 0 0 0) 0x7fedf406a5d0 con 0x7fedf4107290 2024-05-09T09:16:42.086 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 msgr2=0x7feddc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 0x7feddc0a50e0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fede4009a10 tx=0x7fede4009190 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 msgr2=0x7fedf41a4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fedec00d010 tx=0x7fedec00bde0 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 shutdown_connections 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedf4107290 0x7fedf41a4a90 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feddc0a2c20 0x7feddc0a50e0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedf4101220 0x7fedf4199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 --2- 172.21.15.69:0/2220619952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedf4101be0 0x7fedf4199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 >> 172.21.15.69:0/2220619952 conn(0x7fedf40781d0 msgr2=0x7fedf40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:42.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 shutdown_connections 2024-05-09T09:16:42.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42.088+0000 7fedf9185700 1 -- 172.21.15.69:0/2220619952 wait complete. 2024-05-09T09:16:42.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:41 smithi069 ceph-mon[30123]: pgmap v295: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-05-09T09:16:42.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:41 smithi163 ceph-mon[34354]: pgmap v295: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-05-09T09:16:42.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:41 smithi103 ceph-mon[34421]: pgmap v295: 129 pgs: 8 creating+peering, 121 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 3 op/s 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T09:16:42.752 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T09:16:42.753 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:42.753 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-05-09T09:16:42.770 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T09:16:42.771 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:16:42.772 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T09:16:42.772 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42212sec preferred_lft 42212sec 2024-05-09T09:16:42.772 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-05-09T09:16:42.772 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:16:42.772 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T09:16:42.772 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:16:42.772 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:16:42.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-05-09T09:16:42.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-05-09T09:16:42.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-05-09T09:16:42.773 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T09:16:42.773 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:16:42.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:16:42.966 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:42.972 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:42.972 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T09:16:42.978 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:42 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2220619952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:16:42.996 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-05-09T09:16:42.998 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:42.998 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-05-09T09:16:43.081 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-05-09T09:16:43.081 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T09:16:43.082 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T09:16:43.082 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T09:16:43.082 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T09:16:43.082 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:16:43.082 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:16:43.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T09:16:43.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T09:16:43.083 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T09:16:43.083 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:16:43.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:16:43.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:42 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2220619952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:16:43.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:42 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2220619952' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:16:43.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:43.381 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:43.381 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-05-09T09:16:43.398 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:43.398 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-05-09T09:16:43.458 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:16:43.459 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:16:43.459 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:16:43.459 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-05-09T09:16:43.459 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:16:43.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:16:43.572 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:43.578 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-05-09T09:16:43.579 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:16:43.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:16:43.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:43.749 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:16:43.749 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:16:43.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-05-09T09:16:43.749 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T09:16:43.749 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:16:43.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:16:43.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:43.827 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-05-09T09:16:43.827 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T09:16:43.827 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:43.842 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-05-09T09:16:43.842 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T09:16:43.842 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:43.898 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-05-09T09:16:43.964 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-05-09T09:16:44.021 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-05-09T09:16:44.022 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/95246700a94798b7c758d12dcade34b75241a1ef7d9896bfb7e1836770986a92/merged 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/2ba8168c32eeefe00cb48e2bc27daf254bfa9a8ee16a738a49f5fd0e291e94de/merged 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/0 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T09:16:44.023 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/f0e7162e2a3b41de44d9b3bfe8b1b74c3b94ca79d431296f32e3e52ecbc86c8b/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/9a7c37d14ed3c946936e73d2223c6c2a36e292e6ffdfd6b87cf7d74e39f7f9f1/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/c8066468fb1fab120f9c9eaa9ce4dcff5eb90ec10b25e7a8970e50afa590c08e/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/036e1a77f9c28f2f74b51d9f1e444a05cc3325070a4a1bae23921c987b8fd56e/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/f80d553c591563a29b8fd7be7c81583977bdfdb1bd0c1e2780f1fea602796d61/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/9676bee77d889ac6e34921bc20cb06cc64e49c4116b7bbd8aaaa56f57f8becae/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/dae6028816059875df24493cf459756dfab5fc489c85177aed4e32d33da90380/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/d9fabc11817efd8555067d8dee95f49a3d2db50104d79803dc73f8f8e8e1b893/merged 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:16:44.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:44.024 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-05-09T09:16:44.077 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-05-09T09:16:44.077 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-05-09T09:16:44.119 DEBUG:teuthology.orchestra.run.smithi069:> sudo modprobe fuse 2024-05-09T09:16:44.138 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:43 smithi069 ceph-mon[30123]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-05-09T09:16:44.143 DEBUG:teuthology.orchestra.run.smithi069:> cat /proc/self/mounts | awk '{print $2}' 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/proc 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/sys 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/dev 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/security 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/shm 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/pts 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/run 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/cgroup 2024-05-09T09:16:44.200 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/pstore 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/bpf 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/ 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/selinux 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/debug 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/mqueue 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/dev/hugepages 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/tracing 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/fs/fuse/connections 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/sys/kernel/config 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-sysctl.service 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/nfs/rpc_pipefs 2024-05-09T09:16:44.201 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/1000 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/ceph 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/95246700a94798b7c758d12dcade34b75241a1ef7d9896bfb7e1836770986a92/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/2ba8168c32eeefe00cb48e2bc27daf254bfa9a8ee16a738a49f5fd0e291e94de/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/run/user/0 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/proc/sys/fs/binfmt_misc 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/f0e7162e2a3b41de44d9b3bfe8b1b74c3b94ca79d431296f32e3e52ecbc86c8b/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/9a7c37d14ed3c946936e73d2223c6c2a36e292e6ffdfd6b87cf7d74e39f7f9f1/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/c8066468fb1fab120f9c9eaa9ce4dcff5eb90ec10b25e7a8970e50afa590c08e/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/036e1a77f9c28f2f74b51d9f1e444a05cc3325070a4a1bae23921c987b8fd56e/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/f80d553c591563a29b8fd7be7c81583977bdfdb1bd0c1e2780f1fea602796d61/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/9676bee77d889ac6e34921bc20cb06cc64e49c4116b7bbd8aaaa56f57f8becae/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/dae6028816059875df24493cf459756dfab5fc489c85177aed4e32d33da90380/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/var/lib/containers/storage/overlay/d9fabc11817efd8555067d8dee95f49a3d2db50104d79803dc73f8f8e8e1b893/merged 2024-05-09T09:16:44.202 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns 2024-05-09T09:16:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:16:44.203 INFO:teuthology.orchestra.run.smithi069.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:16:44.203 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:44.203 DEBUG:teuthology.orchestra.run.smithi069:> ls /sys/fs/fuse/connections 2024-05-09T09:16:44.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi069.stderr:2024-05-09T09:16:44.221+0000 7ff29b93f480 -1 init, newargv = 0x55fb66e47430 newargc=15 2024-05-09T09:16:44.221 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi069.stderr:ceph-fuse[73367]: starting ceph client 2024-05-09T09:16:44.226 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi069.stderr:ceph-fuse[73367]: starting fuse 2024-05-09T09:16:44.238 INFO:teuthology.orchestra.run.smithi069.stdout:114 2024-05-09T09:16:44.238 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [114] 2024-05-09T09:16:44.238 DEBUG:teuthology.orchestra.run.smithi069:> sudo stdin-killer -- python3 -c ' 2024-05-09T09:16:44.238 DEBUG:teuthology.orchestra.run.smithi069:> import glob 2024-05-09T09:16:44.238 DEBUG:teuthology.orchestra.run.smithi069:> import re 2024-05-09T09:16:44.238 DEBUG:teuthology.orchestra.run.smithi069:> import os 2024-05-09T09:16:44.238 DEBUG:teuthology.orchestra.run.smithi069:> import subprocess 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> def _find_admin_socket(client_name): 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> files = glob.glob(asok_path) 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> # Given a non-glob path, it better be there 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> if "*" not in asok_path: 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> assert(len(files) == 1) 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> return files[0] 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> for f in files: 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> if os.path.exists("/proc/{0}".format(pid)): 2024-05-09T09:16:44.239 DEBUG:teuthology.orchestra.run.smithi069:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> contents = proc_f.read() 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> if mountpoint in contents: 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> return f 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> print(_find_admin_socket("client.admin")) 2024-05-09T09:16:44.240 DEBUG:teuthology.orchestra.run.smithi069:> ' 2024-05-09T09:16:44.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:43 smithi163 ceph-mon[34354]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-05-09T09:16:44.357 INFO:teuthology.orchestra.run.smithi069.stdout:/var/run/ceph/ceph-client.admin.73367.asok 2024-05-09T09:16:44.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:44.366 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:43 smithi103 ceph-mon[34421]: pgmap v296: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.7 KiB/s wr, 2 op/s 2024-05-09T09:16:44.367 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.73367.asok 2024-05-09T09:16:44.368 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:44.368 DEBUG:teuthology.orchestra.run.smithi069:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.73367.asok status 2024-05-09T09:16:44.502 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "metadata": { 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_sha1": "96805dcf909541c4ca9a1e90258de8ae5b6fa1bc", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_version": "ceph version 19.0.0-2515-g96805dcf (96805dcf909541c4ca9a1e90258de8ae5b6fa1bc) squid (dev)", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "entity_id": "admin", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": "smithi069", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "pid": "73367", 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "root": "/" 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_count": 0, 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "dentry_pinned_count": 0, 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 14769, 2024-05-09T09:16:44.503 INFO:teuthology.orchestra.run.smithi069.stdout: "inst": { 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "name": { 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "client", 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "num": 14769 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 966499485 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": { 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "192.168.0.1:0", 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 966499485 2024-05-09T09:16:44.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "inst_str": "client.14769 192.168.0.1:0/966499485", 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "addr_str": "192.168.0.1:0/966499485", 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "inode_count": 1, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_epoch": 17, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch": 89, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_epoch_barrier": 0, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "blocklisted": false, 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs" 2024-05-09T09:16:44.505 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:16:44.514 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:44.514 DEBUG:teuthology.orchestra.run.smithi069:> stat --file-system '--printf=%T 2024-05-09T09:16:44.514 DEBUG:teuthology.orchestra.run.smithi069:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:44.530 INFO:teuthology.orchestra.run.smithi069.stdout:fuseblk 2024-05-09T09:16:44.530 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:44.530 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:44.530 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:44.597 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.admin && exec stdin-killer --timeout=900 -- bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-05-09T09:16:44.690 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T09:16:44.708 INFO:teuthology.orchestra.run.smithi069.stdout:# file: . 2024-05-09T09:16:44.708 INFO:teuthology.orchestra.run.smithi069.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-05-09T09:16:44.708 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:44.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:16:44.714 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:16:44.714 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:16:44.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:44.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3142767517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54107560 msgr2=0x7f9e541099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:44.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3142767517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54107560 0x7f9e541099b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f9e4c009ca0 tx=0x7f9e4c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3142767517 shutdown_connections 2024-05-09T09:16:44.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3142767517 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54107560 0x7f9e541099b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3142767517 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 0x7f9e54107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3142767517 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54102300 0x7f9e541046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3142767517 >> 172.21.15.69:0/3142767517 conn(0x7f9e540fbd30 msgr2=0x7f9e540fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3142767517 shutdown_connections 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.874+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3142767517 wait complete. 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 Processor -- start 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 -- start start 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 0x7f9e54198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54107560 0x7f9e5419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410d500 con 0x7f9e54102300 2024-05-09T09:16:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410d380 con 0x7f9e54107560 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5b1ce640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9e5410d680 con 0x7f9e54104c30 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38848/0 (socket says 172.21.15.69:38848) 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 -- 172.21.15.69:0/3673950136 learned_addr learned my addr 172.21.15.69:0/3673950136 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e5976b640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54107560 0x7f9e5419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e53fff640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 0x7f9e54198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 -- 172.21.15.69:0/3673950136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 msgr2=0x7f9e54198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 0x7f9e54198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 -- 172.21.15.69:0/3673950136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54107560 msgr2=0x7f9e5419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54107560 0x7f9e5419f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.875+0000 7f9e58f6a640 1 -- 172.21.15.69:0/3673950136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9e5419f6e0 con 0x7f9e54102300 2024-05-09T09:16:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e58f6a640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800d950 tx=0x7f9e4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e48017070 con 0x7f9e54102300 2024-05-09T09:16:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9e48009c30 con 0x7f9e54102300 2024-05-09T09:16:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9e4800f6b0 con 0x7f9e54102300 2024-05-09T09:16:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9e5419f9d0 con 0x7f9e54102300 2024-05-09T09:16:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.876+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9e5419ff70 con 0x7f9e54102300 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.877+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9e540fd820 con 0x7f9e54102300 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.878+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9e480109d0 con 0x7f9e54102300 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.879+0000 7f9e51ffb640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 0x7f9e440a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.879+0000 7f9e53fff640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 0x7f9e440a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.879+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f9e480cee70 con 0x7f9e54102300 2024-05-09T09:16:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.879+0000 7f9e53fff640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 0x7f9e440a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e40005e10 tx=0x7f9e40005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:44.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:44.883+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9e48009de0 con 0x7f9e54102300 2024-05-09T09:16:45.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.066+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f9e54062430 con 0x7f9e54102300 2024-05-09T09:16:45.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.067+0000 7f9e51ffb640 1 -- 172.21.15.69:0/3673950136 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f9e4809bb90 con 0x7f9e54102300 2024-05-09T09:16:45.068 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:45.068 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 msgr2=0x7f9e440a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 0x7f9e440a50d0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f9e40005e10 tx=0x7f9e40005da0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 msgr2=0x7f9e54197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f9e4800d950 tx=0x7f9e4800de10 comp rx=0 tx=0).stop 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 shutdown_connections 2024-05-09T09:16:45.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9e440a2c10 0x7f9e440a50d0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9e54107560 0x7f9e5419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9e54104c30 0x7f9e54198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 --2- 172.21.15.69:0/3673950136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9e54102300 0x7f9e54197ac0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.069+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 >> 172.21.15.69:0/3673950136 conn(0x7f9e540fbd30 msgr2=0x7f9e54102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.070+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 shutdown_connections 2024-05-09T09:16:45.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.070+0000 7f9e5b1ce640 1 -- 172.21.15.69:0/3673950136 wait complete. 2024-05-09T09:16:45.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:16:45.081 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:16:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 -- 172.21.15.69:0/836426129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6614101260 msgr2=0x7f6614105b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 --2- 172.21.15.69:0/836426129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6614101260 0x7f6614105b00 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f660c009900 tx=0x7f660c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 -- 172.21.15.69:0/836426129 shutdown_connections 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 --2- 172.21.15.69:0/836426129 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6614101260 0x7f6614105b00 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 --2- 172.21.15.69:0/836426129 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f66141008a0 0x7f6614100d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 --2- 172.21.15.69:0/836426129 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66140ffb40 0x7f66140fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 -- 172.21.15.69:0/836426129 >> 172.21.15.69:0/836426129 conn(0x7f66140fb470 msgr2=0x7f66140fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 -- 172.21.15.69:0/836426129 shutdown_connections 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.189+0000 7f661af4f640 1 -- 172.21.15.69:0/836426129 wait complete. 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 Processor -- start 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 -- start start 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 0x7f661419e6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6614101260 0x7f66141a5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6614113b70 con 0x7f66140ffb40 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f66141139f0 con 0x7f6614101260 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f661af4f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6614113cf0 con 0x7f66141008a0 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38864/0 (socket says 172.21.15.69:38864) 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 -- 172.21.15.69:0/2494855230 learned_addr learned my addr 172.21.15.69:0/2494855230 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f66194ec640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6614101260 0x7f66141a5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f660bfff640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 0x7f661419e6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 -- 172.21.15.69:0/2494855230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 msgr2=0x7f661419e6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 0x7f661419e6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 -- 172.21.15.69:0/2494855230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6614101260 msgr2=0x7f66141a5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6614101260 0x7f66141a5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 -- 172.21.15.69:0/2494855230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f66141a5dd0 con 0x7f66140ffb40 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f660bfff640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 0x7f661419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.190+0000 7f6618ceb640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f661000ba50 tx=0x7f661000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.191+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6610019070 con 0x7f66140ffb40 2024-05-09T09:16:45.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.191+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f661000cbe0 con 0x7f66140ffb40 2024-05-09T09:16:45.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.191+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f66141a60c0 con 0x7f66140ffb40 2024-05-09T09:16:45.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.191+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f66100076e0 con 0x7f66140ffb40 2024-05-09T09:16:45.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.191+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f66141a6640 con 0x7f66140ffb40 2024-05-09T09:16:45.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.192+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6614104910 con 0x7f66140ffb40 2024-05-09T09:16:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.193+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f661001c070 con 0x7f66140ffb40 2024-05-09T09:16:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.194+0000 7f6609ffb640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 0x7f65fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.194+0000 7f660bfff640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 0x7f65fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.194+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f66100ce8a0 con 0x7f66140ffb40 2024-05-09T09:16:45.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.194+0000 7f660bfff640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 0x7f65fc0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f661419f6c0 tx=0x7f6604005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.198+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f661009af70 con 0x7f66140ffb40 2024-05-09T09:16:45.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.374+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f6614062430 con 0x7f66140ffb40 2024-05-09T09:16:45.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.374+0000 7f6609ffb640 1 -- 172.21.15.69:0/2494855230 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7f661009f270 con 0x7f66140ffb40 2024-05-09T09:16:45.374 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:16:45.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 msgr2=0x7f65fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 0x7f65fc0a5050 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f661419f6c0 tx=0x7f6604005e70 comp rx=0 tx=0).stop 2024-05-09T09:16:45.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 msgr2=0x7f661419e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f661000ba50 tx=0x7f661000bf10 comp rx=0 tx=0).stop 2024-05-09T09:16:45.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 shutdown_connections 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65fc0a2b90 0x7f65fc0a5050 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6614101260 0x7f66141a5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f66141008a0 0x7f661419e6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 --2- 172.21.15.69:0/2494855230 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f66140ffb40 0x7f661419e1b0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 >> 172.21.15.69:0/2494855230 conn(0x7f66140fb470 msgr2=0x7f6614101ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 shutdown_connections 2024-05-09T09:16:45.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.376+0000 7f661af4f640 1 -- 172.21.15.69:0/2494855230 wait complete. 2024-05-09T09:16:45.386 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3673950136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:45.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:16:45.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 -- 172.21.15.69:0/2347695255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 msgr2=0x7fd5e4102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/2347695255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e4102f90 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8009930 tx=0x7fd5d802a420 comp rx=0 tx=0).stop 2024-05-09T09:16:45.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 -- 172.21.15.69:0/2347695255 shutdown_connections 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/2347695255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e41034d0 0x7fd5e4109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/2347695255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e4102f90 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/2347695255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e40ffb40 0x7fd5e40fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 -- 172.21.15.69:0/2347695255 >> 172.21.15.69:0/2347695255 conn(0x7fd5e40fb470 msgr2=0x7fd5e40fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 -- 172.21.15.69:0/2347695255 shutdown_connections 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.491+0000 7fd5ea31d640 1 -- 172.21.15.69:0/2347695255 wait complete. 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 Processor -- start 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 -- start start 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e40ffb40 0x7fd5e4197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e41034d0 0x7fd5e419f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410d520 con 0x7fd5e4102b10 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410d3a0 con 0x7fd5e40ffb40 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5ea31d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd5e410d6a0 con 0x7fd5e41034d0 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38882/0 (socket says 172.21.15.69:38882) 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5e37fe640 1 -- 172.21.15.69:0/4276823609 learned_addr learned my addr 172.21.15.69:0/4276823609 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5e3fff640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e40ffb40 0x7fd5e4197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.492+0000 7fd5e88ba640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e41034d0 0x7fd5e419f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 -- 172.21.15.69:0/4276823609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e41034d0 msgr2=0x7fd5e419f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e41034d0 0x7fd5e419f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 -- 172.21.15.69:0/4276823609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e40ffb40 msgr2=0x7fd5e4197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e40ffb40 0x7fd5e4197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 -- 172.21.15.69:0/4276823609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd5e419f7c0 con 0x7fd5e4102b10 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e37fe640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8031610 tx=0x7fd5d80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d803f030 con 0x7fd5e4102b10 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5d8003ce0 con 0x7fd5e4102b10 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd5e419fa50 con 0x7fd5e4102b10 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd5d8035070 con 0x7fd5e4102b10 2024-05-09T09:16:45.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.493+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd5e419ff40 con 0x7fd5e4102b10 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.494+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd5e4108c40 con 0x7fd5e4102b10 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.495+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fd5d8042030 con 0x7fd5e4102b10 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.496+0000 7fd5e17fa640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 0x7fd5cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.496+0000 7fd5e3fff640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 0x7fd5cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.496+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fd5d80ef860 con 0x7fd5e4102b10 2024-05-09T09:16:45.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.496+0000 7fd5e3fff640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 0x7fd5cc0a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4005fd0 tx=0x7fd5d400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.500+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd5d80bbf30 con 0x7fd5e4102b10 2024-05-09T09:16:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3673950136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3673950136' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:45.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.674+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fd5e4062430 con 0x7fd5e4102b10 2024-05-09T09:16:45.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.674+0000 7fd5e17fa640 1 -- 172.21.15.69:0/4276823609 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7fd5d80c0230 con 0x7fd5e4102b10 2024-05-09T09:16:45.674 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:16:45.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 msgr2=0x7fd5cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 0x7fd5cc0a5050 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd5d4005fd0 tx=0x7fd5d400b390 comp rx=0 tx=0).stop 2024-05-09T09:16:45.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 msgr2=0x7fd5e41980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fd5d8031610 tx=0x7fd5d80043b0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 shutdown_connections 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5cc0a2b90 0x7fd5cc0a5050 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd5e41034d0 0x7fd5e419f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd5e4102b10 0x7fd5e41980e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 --2- 172.21.15.69:0/4276823609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd5e40ffb40 0x7fd5e4197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 >> 172.21.15.69:0/4276823609 conn(0x7fd5e40fb470 msgr2=0x7fd5e4105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 shutdown_connections 2024-05-09T09:16:45.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.676+0000 7fd5ea31d640 1 -- 172.21.15.69:0/4276823609 wait complete. 2024-05-09T09:16:45.687 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:16:45.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 -- 172.21.15.69:0/855095941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 msgr2=0x7f8d181058a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 --2- 172.21.15.69:0/855095941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181058a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009ca0 tx=0x7f8d1002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 -- 172.21.15.69:0/855095941 shutdown_connections 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 --2- 172.21.15.69:0/855095941 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181058a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 --2- 172.21.15.69:0/855095941 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 0x7f8d18104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 --2- 172.21.15.69:0/855095941 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 0x7f8d18102530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 -- 172.21.15.69:0/855095941 >> 172.21.15.69:0/855095941 conn(0x7f8d180fbd20 msgr2=0x7f8d180fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 -- 172.21.15.69:0/855095941 shutdown_connections 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.791+0000 7f8d1f709640 1 -- 172.21.15.69:0/855095941 wait complete. 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 Processor -- start 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 -- start start 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 0x7f8d1819e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 0x7f8d1819e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.794 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18113b60 con 0x7f8d18105420 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d181139e0 con 0x7f8d18100100 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1f709640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d18113ce0 con 0x7f8d18102a70 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1dca6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1dca6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38900/0 (socket says 172.21.15.69:38900) 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.792+0000 7f8d1dca6640 1 -- 172.21.15.69:0/920187326 learned_addr learned my addr 172.21.15.69:0/920187326 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1d4a5640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 0x7f8d1819e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1cca4640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 0x7f8d1819e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 -- 172.21.15.69:0/920187326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 msgr2=0x7f8d1819e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 0x7f8d1819e6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 -- 172.21.15.69:0/920187326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 msgr2=0x7f8d1819e170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 0x7f8d1819e170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 -- 172.21.15.69:0/920187326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d181a5d90 con 0x7f8d18105420 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1dca6640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8d100098d0 tx=0x7f8d1002f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1002fc30 con 0x7f8d18105420 2024-05-09T09:16:45.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d10003e50 con 0x7f8d18105420 2024-05-09T09:16:45.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d181a6020 con 0x7f8d18105420 2024-05-09T09:16:45.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d100305c0 con 0x7f8d18105420 2024-05-09T09:16:45.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.793+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d181a65f0 con 0x7f8d18105420 2024-05-09T09:16:45.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.794+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d180fd810 con 0x7f8d18105420 2024-05-09T09:16:45.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.795+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8d10003970 con 0x7f8d18105420 2024-05-09T09:16:45.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.796+0000 7f8d0e7fc640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 0x7f8d040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:45.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.796+0000 7f8d1d4a5640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 0x7f8d040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:45.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.796+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8d100f08f0 con 0x7f8d18105420 2024-05-09T09:16:45.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.796+0000 7f8d1d4a5640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 0x7f8d040a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d140058c0 tx=0x7f8d1400a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:45.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.800+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d100bd610 con 0x7f8d18105420 2024-05-09T09:16:45.976 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.976+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f8d18062430 con 0x7f8d18105420 2024-05-09T09:16:45.977 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.977+0000 7f8d0e7fc640 1 -- 172.21.15.69:0/920187326 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f8d100bcf10 con 0x7f8d18105420 2024-05-09T09:16:45.977 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 msgr2=0x7f8d040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 0x7f8d040a5050 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8d140058c0 tx=0x7f8d1400a4c0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 msgr2=0x7f8d181a5720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f8d100098d0 tx=0x7f8d1002f840 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 shutdown_connections 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d040a2b90 0x7f8d040a5050 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d18105420 0x7f8d181a5720 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d18102a70 0x7f8d1819e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 --2- 172.21.15.69:0/920187326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d18100100 0x7f8d1819e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 >> 172.21.15.69:0/920187326 conn(0x7f8d180fbd20 msgr2=0x7f8d18100db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 shutdown_connections 2024-05-09T09:16:45.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:45.979+0000 7f8d1f709640 1 -- 172.21.15.69:0/920187326 wait complete. 2024-05-09T09:16:45.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3137914418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c101960 msgr2=0x7fb79c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3137914418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c101960 0x7fb79c101de0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fb790009ca0 tx=0x7fb79002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3137914418 shutdown_connections 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3137914418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c102320 0x7fb79c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3137914418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c101960 0x7fb79c101de0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3137914418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 0x7fb79c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3137914418 >> 172.21.15.69:0/3137914418 conn(0x7fb79c0fbd20 msgr2=0x7fb79c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3137914418 shutdown_connections 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3137914418 wait complete. 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 Processor -- start 2024-05-09T09:16:46.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.094+0000 7fb7a1f09640 1 -- start start 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 0x7fb79c197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c101960 0x7fb79c198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb79c10d510 con 0x7fb79c102320 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb79c10d390 con 0x7fb79c101960 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb79c10d690 con 0x7fb79c100770 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 0x7fb79c197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38928/0 (socket says 172.21.15.69:38928) 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 -- 172.21.15.69:0/3770978584 learned_addr learned my addr 172.21.15.69:0/3770978584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79affd640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c101960 0x7fb79c198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 -- 172.21.15.69:0/3770978584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 msgr2=0x7fb79c197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 0x7fb79c197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 -- 172.21.15.69:0/3770978584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c101960 msgr2=0x7fb79c198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c101960 0x7fb79c198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 -- 172.21.15.69:0/3770978584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb79c19f760 con 0x7fb79c102320 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb79bfff640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb79400e970 tx=0x7fb79400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb79400ccc0 con 0x7fb79c102320 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb79c19fa50 con 0x7fb79c102320 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb79400ce60 con 0x7fb79c102320 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb794010490 con 0x7fb79c102320 2024-05-09T09:16:46.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.095+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb79c19ff50 con 0x7fb79c102320 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.097+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb79c107a90 con 0x7fb79c102320 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.098+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fb794005020 con 0x7fb79c102320 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.098+0000 7fb798ff9640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 0x7fb7840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.098+0000 7fb79b7fe640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 0x7fb7840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.098+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb7940cf6d0 con 0x7fb79c102320 2024-05-09T09:16:46.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.099+0000 7fb79b7fe640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 0x7fb7840a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb79c1017c0 tx=0x7fb78c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.103+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb79409c3f0 con 0x7fb79c102320 2024-05-09T09:16:46.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:46 smithi069 ceph-mon[30123]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T09:16:46.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2494855230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4276823609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/920187326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.281+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb79c062430 con 0x7fb79c102320 2024-05-09T09:16:46.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.282+0000 7fb798ff9640 1 -- 172.21.15.69:0/3770978584 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7fb79409bcf0 con 0x7fb79c102320 2024-05-09T09:16:46.282 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 msgr2=0x7fb7840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 0x7fb7840a5050 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7fb79c1017c0 tx=0x7fb78c005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 msgr2=0x7fb79c19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fb79400e970 tx=0x7fb79400ed70 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 shutdown_connections 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb7840a2b90 0x7fb7840a5050 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb79c102320 0x7fb79c19f0f0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb79c101960 0x7fb79c198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 --2- 172.21.15.69:0/3770978584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb79c100770 0x7fb79c197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 >> 172.21.15.69:0/3770978584 conn(0x7fb79c0fbd20 msgr2=0x7fb79c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 shutdown_connections 2024-05-09T09:16:46.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.284+0000 7fb7a1f09640 1 -- 172.21.15.69:0/3770978584 wait complete. 2024-05-09T09:16:46.295 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T09:16:46.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 -- 172.21.15.69:0/39332117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434100790 msgr2=0x7f2434100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 --2- 172.21.15.69:0/39332117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434100790 0x7f2434100bb0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f2430009900 tx=0x7f243002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:46.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 -- 172.21.15.69:0/39332117 shutdown_connections 2024-05-09T09:16:46.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 --2- 172.21.15.69:0/39332117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434102340 0x7f2434108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 --2- 172.21.15.69:0/39332117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434101980 0x7f2434101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 --2- 172.21.15.69:0/39332117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434100790 0x7f2434100bb0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 -- 172.21.15.69:0/39332117 >> 172.21.15.69:0/39332117 conn(0x7f24340fbd00 msgr2=0x7f24340fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 -- 172.21.15.69:0/39332117 shutdown_connections 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.397+0000 7f243bc58640 1 -- 172.21.15.69:0/39332117 wait complete. 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 Processor -- start 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 -- start start 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434100790 0x7f2434197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434102340 0x7f243419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f243410d530 con 0x7f2434101980 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f243410d3b0 con 0x7f2434100790 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243bc58640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f243410d6b0 con 0x7f2434102340 2024-05-09T09:16:46.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38938/0 (socket says 172.21.15.69:38938) 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 -- 172.21.15.69:0/547778058 learned_addr learned my addr 172.21.15.69:0/547778058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24399f4640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434100790 0x7f2434197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f243a1f5640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434102340 0x7f243419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 -- 172.21.15.69:0/547778058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434102340 msgr2=0x7f243419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434102340 0x7f243419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 -- 172.21.15.69:0/547778058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434100790 msgr2=0x7f2434197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434100790 0x7f2434197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 -- 172.21.15.69:0/547778058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f243419f860 con 0x7f2434101980 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.398+0000 7f24391f3640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f242400d770 tx=0x7f242400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.399+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424017070 con 0x7f2434101980 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.399+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2424005500 con 0x7f2434101980 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.399+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2424004e00 con 0x7f2434101980 2024-05-09T09:16:46.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.399+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f243419fb50 con 0x7f2434101980 2024-05-09T09:16:46.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.399+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24341a0020 con 0x7f2434101980 2024-05-09T09:16:46.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.400+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2434107bd0 con 0x7f2434101980 2024-05-09T09:16:46.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.401+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2424004fa0 con 0x7f2434101980 2024-05-09T09:16:46.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.401+0000 7f242affd640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 0x7f24200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.401+0000 7f24399f4640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 0x7f24200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.402+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f24240ced90 con 0x7f2434101980 2024-05-09T09:16:46.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.402+0000 7f24399f4640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 0x7f24200a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2430005ab0 tx=0x7f2430000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.406+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f242409bab0 con 0x7f2434101980 2024-05-09T09:16:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:46 smithi163 ceph-mon[34354]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T09:16:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2494855230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4276823609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/920187326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.594+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f2434062430 con 0x7f2434101980 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.594+0000 7f242affd640 1 -- 172.21.15.69:0/547778058 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f242409b3b0 con 0x7f2434101980 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/4182744829 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1673751021 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1911852009 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:46.594 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1056628795 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/2520675955 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/3187678781 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:46.595 INFO:teuthology.orchestra.run.smithi069.stderr:listed 10 entries 2024-05-09T09:16:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 msgr2=0x7f24200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 0x7f24200a5050 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2430005ab0 tx=0x7f2430000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 msgr2=0x7f2434198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f242400d770 tx=0x7f242400dc30 comp rx=0 tx=0).stop 2024-05-09T09:16:46.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 shutdown_connections 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24200a2b90 0x7f24200a5050 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2434102340 0x7f243419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2434101980 0x7f2434198190 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 --2- 172.21.15.69:0/547778058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2434100790 0x7f2434197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 >> 172.21.15.69:0/547778058 conn(0x7f24340fbd00 msgr2=0x7f2434104da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 shutdown_connections 2024-05-09T09:16:46.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.596+0000 7f243bc58640 1 -- 172.21.15.69:0/547778058 wait complete. 2024-05-09T09:16:46.607 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:16:46.608 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:16:46.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:46 smithi103 ceph-mon[34421]: pgmap v297: 129 pgs: 129 active+clean; 579 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 0 B/s wr, 0 op/s 2024-05-09T09:16:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2494855230' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4276823609' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/920187326' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:46.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- 172.21.15.69:0/1421696511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 msgr2=0x7f63dc100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- 172.21.15.69:0/1421696511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc100c10 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009900 tx=0x7f63d002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- 172.21.15.69:0/1421696511 shutdown_connections 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- 172.21.15.69:0/1421696511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1023a0 0x7f63dc108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- 172.21.15.69:0/1421696511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1019e0 0x7f63dc101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- 172.21.15.69:0/1421696511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc100c10 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- 172.21.15.69:0/1421696511 >> 172.21.15.69:0/1421696511 conn(0x7f63dc0fbd80 msgr2=0x7f63dc0fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- 172.21.15.69:0/1421696511 shutdown_connections 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- 172.21.15.69:0/1421696511 wait complete. 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 Processor -- start 2024-05-09T09:16:46.767 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- start start 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 0x7f63dc197020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 0x7f63dc19e090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10c4a0 con 0x7f63dc1007f0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10c320 con 0x7f63dc1023a0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.765+0000 7f63e3134640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f63dc10c620 con 0x7f63dc1019e0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38960/0 (socket says 172.21.15.69:38960) 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 -- 172.21.15.69:0/1030111386 learned_addr learned my addr 172.21.15.69:0/1030111386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63dbfff640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 0x7f63dc197020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e16d1640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 0x7f63dc19e090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 -- 172.21.15.69:0/1030111386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 msgr2=0x7f63dc197020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 0x7f63dc197020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 -- 172.21.15.69:0/1030111386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 msgr2=0x7f63dc19e090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 0x7f63dc19e090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 -- 172.21.15.69:0/1030111386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f63dc19e700 con 0x7f63dc1007f0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63dbfff640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 0x7f63dc197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e16d1640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 0x7f63dc19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e0ed0640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63d0031440 tx=0x7f63d0032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d003f030 con 0x7f63dc1007f0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f63d0003ce0 con 0x7f63dc1007f0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f63dc19e990 con 0x7f63dc1007f0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f63d0035070 con 0x7f63dc1007f0 2024-05-09T09:16:46.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.766+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f63dc19ef00 con 0x7f63dc1007f0 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.767+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f63dc068aa0 con 0x7f63dc1007f0 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.769+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f63d0042030 con 0x7f63dc1007f0 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.770+0000 7f63d9ffb640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 0x7f63cc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.770+0000 7f63dbfff640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 0x7f63cc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.770+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f63d00efd40 con 0x7f63dc1007f0 2024-05-09T09:16:46.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.771+0000 7f63dbfff640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 0x7f63cc0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63c80058c0 tx=0x7f63c8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:46.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.774+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f63d00bca60 con 0x7f63dc1007f0 2024-05-09T09:16:46.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.953+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f63dc107b10 con 0x7f63dc1007f0 2024-05-09T09:16:46.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.954+0000 7f63d9ffb640 1 -- 172.21.15.69:0/1030111386 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f63d00bc360 con 0x7f63dc1007f0 2024-05-09T09:16:46.954 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:46.955 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 msgr2=0x7f63cc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 0x7f63cc0a5050 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f63c80058c0 tx=0x7f63c8005850 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 msgr2=0x7f63dc196ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f63d0031440 tx=0x7f63d0032e20 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 shutdown_connections 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f63cc0a2b90 0x7f63cc0a5050 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f63dc1023a0 0x7f63dc19e090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f63dc1019e0 0x7f63dc197020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 --2- 172.21.15.69:0/1030111386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f63dc1007f0 0x7f63dc196ae0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 >> 172.21.15.69:0/1030111386 conn(0x7f63dc0fbd80 msgr2=0x7f63dc104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 shutdown_connections 2024-05-09T09:16:46.957 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:46.956+0000 7f63e3134640 1 -- 172.21.15.69:0/1030111386 wait complete. 2024-05-09T09:16:46.968 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:16:46.969 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:16:47.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 -- 172.21.15.69:0/1134108027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc100770 msgr2=0x7fa7bc100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/1134108027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc100770 0x7fa7bc100b90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac009900 tx=0x7fa7ac02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 -- 172.21.15.69:0/1134108027 shutdown_connections 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/1134108027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 0x7fa7bc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/1134108027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc101960 0x7fa7bc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/1134108027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc100770 0x7fa7bc100b90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 -- 172.21.15.69:0/1134108027 >> 172.21.15.69:0/1134108027 conn(0x7fa7bc0fbce0 msgr2=0x7fa7bc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 -- 172.21.15.69:0/1134108027 shutdown_connections 2024-05-09T09:16:47.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.072+0000 7fa7c0e10640 1 -- 172.21.15.69:0/1134108027 wait complete. 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 Processor -- start 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 -- start start 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc100770 0x7fa7bc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 0x7fa7bc19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10c3e0 con 0x7fa7bc101960 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10c260 con 0x7fa7bc102320 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7c0e10640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7bc10c560 con 0x7fa7bc100770 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38966/0 (socket says 172.21.15.69:38966) 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 -- 172.21.15.69:0/2936215385 learned_addr learned my addr 172.21.15.69:0/2936215385 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7bad9d640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 0x7fa7bc19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 -- 172.21.15.69:0/2936215385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc100770 msgr2=0x7fa7bc197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc100770 0x7fa7bc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 -- 172.21.15.69:0/2936215385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 msgr2=0x7fa7bc19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 0x7fa7bc19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.073+0000 7fa7b3fff640 1 -- 172.21.15.69:0/2936215385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7bc19f7b0 con 0x7fa7bc101960 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7b3fff640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400d770 tx=0x7fa7a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4017070 con 0x7fa7bc101960 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7a4005500 con 0x7fa7bc101960 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa7a4004e00 con 0x7fa7bc101960 2024-05-09T09:16:47.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7bc19faa0 con 0x7fa7bc101960 2024-05-09T09:16:47.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.074+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7bc19ffa0 con 0x7fa7bc101960 2024-05-09T09:16:47.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.075+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa7bc1a01d0 con 0x7fa7bc101960 2024-05-09T09:16:47.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.076+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa7a4004fa0 con 0x7fa7bc101960 2024-05-09T09:16:47.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.076+0000 7fa7b37fe640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.076+0000 7fa7ba59c640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.077+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa7a40ced00 con 0x7fa7bc101960 2024-05-09T09:16:47.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.077+0000 7fa7ba59c640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 0x7fa7a80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0096c0 tx=0x7fa7ac031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.081+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa7a409ba20 con 0x7fa7bc101960 2024-05-09T09:16:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3770978584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/547778058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1030111386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:47.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.261+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fa7bc1a0440 con 0x7fa7bc101960 2024-05-09T09:16:47.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.261+0000 7fa7b37fe640 1 -- 172.21.15.69:0/2936215385 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) ==== 93+0+10 (secure 0 0 0) 0x7fa7a409b320 con 0x7fa7bc101960 2024-05-09T09:16:47.261 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 msgr2=0x7fa7a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 0x7fa7a80a5050 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fa7ac0096c0 tx=0x7fa7ac031400 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 msgr2=0x7fa7bc1980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa7a400d770 tx=0x7fa7a400dc30 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 shutdown_connections 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7a80a2b90 0x7fa7a80a5050 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7bc102320 0x7fa7bc19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7bc101960 0x7fa7bc1980d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 --2- 172.21.15.69:0/2936215385 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7bc100770 0x7fa7bc197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 >> 172.21.15.69:0/2936215385 conn(0x7fa7bc0fbce0 msgr2=0x7fa7bc069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 shutdown_connections 2024-05-09T09:16:47.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.263+0000 7fa7c0e10640 1 -- 172.21.15.69:0/2936215385 wait complete. 2024-05-09T09:16:47.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 -- 172.21.15.69:0/3619941874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4100770 msgr2=0x7f22f4100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 --2- 172.21.15.69:0/3619941874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4100770 0x7f22f4100b90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f22e4009ca0 tx=0x7f22e402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 -- 172.21.15.69:0/3619941874 shutdown_connections 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 --2- 172.21.15.69:0/3619941874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4102320 0x7f22f4108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 --2- 172.21.15.69:0/3619941874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 --2- 172.21.15.69:0/3619941874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4100770 0x7f22f4100b90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 -- 172.21.15.69:0/3619941874 >> 172.21.15.69:0/3619941874 conn(0x7f22f40fbce0 msgr2=0x7f22f40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 -- 172.21.15.69:0/3619941874 shutdown_connections 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.378+0000 7f22faecd640 1 -- 172.21.15.69:0/3619941874 wait complete. 2024-05-09T09:16:47.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 Processor -- start 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 -- start start 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 0x7f22f4197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410c440 con 0x7f22f4102320 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410c2c0 con 0x7f22f4101960 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22faecd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f22f410c5c0 con 0x7f22f4100770 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22f946a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22f946a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38984/0 (socket says 172.21.15.69:38984) 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.379+0000 7f22f946a640 1 -- 172.21.15.69:0/1641206665 learned_addr learned my addr 172.21.15.69:0/1641206665 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f3fff640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 -- 172.21.15.69:0/1641206665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 msgr2=0x7f22f4197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f8c69640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 0x7f22f4197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 0x7f22f4197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 -- 172.21.15.69:0/1641206665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 msgr2=0x7f22f4198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 -- 172.21.15.69:0/1641206665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22f419f800 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f3fff640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f8c69640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 0x7f22f4197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f946a640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009fd0 tx=0x7f22ec00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec011830 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f22ec016070 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22f419faf0 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f22ec00f2e0 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.380+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22f41a0090 con 0x7f22f4102320 2024-05-09T09:16:47.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.381+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f22f41a0320 con 0x7f22f4102320 2024-05-09T09:16:47.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.382+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f22ec0119d0 con 0x7f22f4102320 2024-05-09T09:16:47.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.383+0000 7f22f1ffb640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 0x7f22e00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.383+0000 7f22f8c69640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 0x7f22e00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.383+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f22ec0ce8c0 con 0x7f22f4102320 2024-05-09T09:16:47.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.384+0000 7f22f8c69640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 0x7f22e00a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22e4006010 tx=0x7f22e4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.388+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f22ec09b5e0 con 0x7f22f4102320 2024-05-09T09:16:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3770978584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/547778058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1030111386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:47.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.561+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f22f41a04f0 con 0x7f22f4102320 2024-05-09T09:16:47.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.562+0000 7f22f1ffb640 1 -- 172.21.15.69:0/1641206665 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) ==== 104+0+11 (secure 0 0 0) 0x7f22ec09aee0 con 0x7f22f4102320 2024-05-09T09:16:47.562 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 msgr2=0x7f22e00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 0x7f22e00a50d0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f22e4006010 tx=0x7f22e4000f40 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 msgr2=0x7f22f419f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f22ec009fd0 tx=0x7f22ec00b8e0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 shutdown_connections 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f22e00a2c10 0x7f22e00a50d0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f22f4102320 0x7f22f419f190 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22f4101960 0x7f22f4198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 --2- 172.21.15.69:0/1641206665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f22f4100770 0x7f22f4197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 >> 172.21.15.69:0/1641206665 conn(0x7f22f40fbce0 msgr2=0x7f22f40fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 shutdown_connections 2024-05-09T09:16:47.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.564+0000 7f22faecd640 1 -- 172.21.15.69:0/1641206665 wait complete. 2024-05-09T09:16:47.575 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:16:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3770978584' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/547778058' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1030111386' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:47.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- 172.21.15.69:0/4047593742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810101980 msgr2=0x7f9810101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 --2- 172.21.15.69:0/4047593742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810101980 0x7f9810101e00 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9804009ca0 tx=0x7f980402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- 172.21.15.69:0/4047593742 shutdown_connections 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 --2- 172.21.15.69:0/4047593742 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810102340 0x7f9810108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 --2- 172.21.15.69:0/4047593742 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810101980 0x7f9810101e00 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 --2- 172.21.15.69:0/4047593742 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810100790 0x7f9810100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- 172.21.15.69:0/4047593742 >> 172.21.15.69:0/4047593742 conn(0x7f98100fbd00 msgr2=0x7f98100fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- 172.21.15.69:0/4047593742 shutdown_connections 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- 172.21.15.69:0/4047593742 wait complete. 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 Processor -- start 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.679+0000 7f98176f5640 1 -- start start 2024-05-09T09:16:47.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810100790 0x7f9810197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810101980 0x7f98101980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f981010d530 con 0x7f9810102340 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f981010d3b0 con 0x7f9810101980 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f981010d6b0 con 0x7f9810100790 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44708/0 (socket says 172.21.15.69:44708) 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 -- 172.21.15.69:0/3255776134 learned_addr learned my addr 172.21.15.69:0/3255776134 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9814c90640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810101980 0x7f98101980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815491640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810100790 0x7f9810197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 -- 172.21.15.69:0/3255776134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810100790 msgr2=0x7f9810197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810100790 0x7f9810197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 -- 172.21.15.69:0/3255776134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810101980 msgr2=0x7f98101980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810101980 0x7f98101980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 -- 172.21.15.69:0/3255776134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f981019f7b0 con 0x7f9810102340 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f9815c92640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f980000d750 tx=0x7f980000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800017070 con 0x7f9810102340 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9800005500 con 0x7f9810102340 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f981019faa0 con 0x7f9810102340 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9800004d90 con 0x7f9810102340 2024-05-09T09:16:47.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.680+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f981019fff0 con 0x7f9810102340 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.681+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9810107ab0 con 0x7f9810102340 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.682+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f9800004f30 con 0x7f9810102340 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.683+0000 7f980e7fc640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 0x7f97f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.683+0000 7f9815491640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 0x7f97f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.683+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f98000cec50 con 0x7f9810102340 2024-05-09T09:16:47.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.684+0000 7f9815491640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 0x7f97f80a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9808005fd0 tx=0x7f9808005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.687 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.688+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98000d4020 con 0x7f9810102340 2024-05-09T09:16:47.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.866+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9810062430 con 0x7f9810102340 2024-05-09T09:16:47.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.866+0000 7f980e7fc640 1 -- 172.21.15.69:0/3255776134 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) ==== 100+0+11 (secure 0 0 0) 0x7f980009b970 con 0x7f9810102340 2024-05-09T09:16:47.866 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 msgr2=0x7f97f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 0x7f97f80a5050 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f9808005fd0 tx=0x7f9808005850 comp rx=0 tx=0).stop 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 msgr2=0x7f981019f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f980000d750 tx=0x7f980000dc10 comp rx=0 tx=0).stop 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 shutdown_connections 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f97f80a2b90 0x7f97f80a5050 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9810102340 0x7f981019f140 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9810101980 0x7f98101980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 --2- 172.21.15.69:0/3255776134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9810100790 0x7f9810197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 >> 172.21.15.69:0/3255776134 conn(0x7f98100fbd00 msgr2=0x7f9810104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 shutdown_connections 2024-05-09T09:16:47.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.868+0000 7f98176f5640 1 -- 172.21.15.69:0/3255776134 wait complete. 2024-05-09T09:16:47.879 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:16:47.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 -- 172.21.15.69:0/3591233126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 msgr2=0x7f5c0c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 --2- 172.21.15.69:0/3591233126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c100b90 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc009900 tx=0x7f5bfc02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 -- 172.21.15.69:0/3591233126 shutdown_connections 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 --2- 172.21.15.69:0/3591233126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 0x7f5c0c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 --2- 172.21.15.69:0/3591233126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 0x7f5c0c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 --2- 172.21.15.69:0/3591233126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c100b90 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 -- 172.21.15.69:0/3591233126 >> 172.21.15.69:0/3591233126 conn(0x7f5c0c0fbce0 msgr2=0x7f5c0c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 -- 172.21.15.69:0/3591233126 shutdown_connections 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.982+0000 7f5c113f9640 1 -- 172.21.15.69:0/3591233126 wait complete. 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 Processor -- start 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 -- start start 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 0x7f5c0c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 0x7f5c0c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c3c0 con 0x7f5c0c100770 2024-05-09T09:16:47.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c240 con 0x7f5c0c101960 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c113f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c0c10c540 con 0x7f5c0c102320 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c0affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c0affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44734/0 (socket says 172.21.15.69:44734) 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c0affd640 1 -- 172.21.15.69:0/485943499 learned_addr learned my addr 172.21.15.69:0/485943499 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.983+0000 7f5c0a7fc640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 0x7f5c0c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 -- 172.21.15.69:0/485943499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 msgr2=0x7f5c0c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0b7fe640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 0x7f5c0c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 0x7f5c0c19f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 -- 172.21.15.69:0/485943499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 msgr2=0x7f5c0c198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 0x7f5c0c198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 -- 172.21.15.69:0/485943499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c0c19f7b0 con 0x7f5c0c100770 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c0affd640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc031610 tx=0x7f5bfc0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc03f030 con 0x7f5c0c100770 2024-05-09T09:16:47.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bfc003ce0 con 0x7f5c0c100770 2024-05-09T09:16:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c0c19fa40 con 0x7f5c0c100770 2024-05-09T09:16:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bfc035070 con 0x7f5c0c100770 2024-05-09T09:16:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.984+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c0c19ff30 con 0x7f5c0c100770 2024-05-09T09:16:47.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.985+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5c0c1a0100 con 0x7f5c0c100770 2024-05-09T09:16:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.986+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f5bfc042030 con 0x7f5c0c100770 2024-05-09T09:16:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.987+0000 7f5bf3fff640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 0x7f5bf40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.987+0000 7f5c0a7fc640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 0x7f5bf40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.987+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f5bfc0ef820 con 0x7f5c0c100770 2024-05-09T09:16:47.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.987+0000 7f5c0a7fc640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 0x7f5bf40a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c1990a0 tx=0x7f5c0000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:47.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:47.991+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5bfc0bc540 con 0x7f5c0c100770 2024-05-09T09:16:48.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.165+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f5c0c1a0490 con 0x7f5c0c100770 2024-05-09T09:16:48.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.166+0000 7f5bf3fff640 1 -- 172.21.15.69:0/485943499 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) ==== 103+0+11 (secure 0 0 0) 0x7f5bfc0bbe40 con 0x7f5c0c100770 2024-05-09T09:16:48.166 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:16:48.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 msgr2=0x7f5bf40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 0x7f5bf40a5050 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5c0c1990a0 tx=0x7f5c0000b410 comp rx=0 tx=0).stop 2024-05-09T09:16:48.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 msgr2=0x7f5c0c197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f5bfc031610 tx=0x7f5bfc0043b0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.168 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 shutdown_connections 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bf40a2b90 0x7f5bf40a5050 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c0c102320 0x7f5c0c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c0c101960 0x7f5c0c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 --2- 172.21.15.69:0/485943499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c0c100770 0x7f5c0c197be0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 >> 172.21.15.69:0/485943499 conn(0x7f5c0c0fbce0 msgr2=0x7f5c0c0fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 shutdown_connections 2024-05-09T09:16:48.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.168+0000 7f5c113f9640 1 -- 172.21.15.69:0/485943499 wait complete. 2024-05-09T09:16:48.179 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 -- 172.21.15.69:0/779124080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd8104cc0 msgr2=0x7f4bd81070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/779124080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd8104cc0 0x7f4bd81070b0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f4bcc009900 tx=0x7f4bcc02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 -- 172.21.15.69:0/779124080 shutdown_connections 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/779124080 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd81075f0 0x7f4bd81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/779124080 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd8104cc0 0x7f4bd81070b0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/779124080 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8102390 0x7f4bd8104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 -- 172.21.15.69:0/779124080 >> 172.21.15.69:0/779124080 conn(0x7f4bd80fbd20 msgr2=0x7f4bd80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:48.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 -- 172.21.15.69:0/779124080 shutdown_connections 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 -- 172.21.15.69:0/779124080 wait complete. 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.282+0000 7f4bde0fc640 1 Processor -- start 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 -- start start 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd8102390 0x7f4bd8197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 0x7f4bd8198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d500 con 0x7f4bd81075f0 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d380 con 0x7f4bd8102390 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bde0fc640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4bd810d680 con 0x7f4bd8104cc0 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44752/0 (socket says 172.21.15.69:44752) 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 -- 172.21.15.69:0/1368727593 learned_addr learned my addr 172.21.15.69:0/1368727593 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:48.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd77fe640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd8102390 0x7f4bd8197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd6ffd640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 0x7f4bd8198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 -- 172.21.15.69:0/1368727593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 msgr2=0x7f4bd8198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 0x7f4bd8198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 -- 172.21.15.69:0/1368727593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd8102390 msgr2=0x7f4bd8197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd8102390 0x7f4bd8197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 -- 172.21.15.69:0/1368727593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4bd819f830 con 0x7f4bd81075f0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd6ffd640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 0x7f4bd8198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd7fff640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000eaa0 tx=0x7f4bd000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.283+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd000cdb0 con 0x7f4bd81075f0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.284+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4bd0005020 con 0x7f4bd81075f0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.284+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4bd819fb20 con 0x7f4bd81075f0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.284+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4bd00116b0 con 0x7f4bd81075f0 2024-05-09T09:16:48.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.284+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4bd81a0020 con 0x7f4bd81075f0 2024-05-09T09:16:48.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.285+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4bd80fd810 con 0x7f4bd81075f0 2024-05-09T09:16:48.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.286+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f4bd0005560 con 0x7f4bd81075f0 2024-05-09T09:16:48.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.286+0000 7f4bd4ff9640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 0x7f4bc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.287+0000 7f4bd77fe640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 0x7f4bc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.287+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f4bd00cf590 con 0x7f4bd81075f0 2024-05-09T09:16:48.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.287+0000 7f4bd77fe640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 0x7f4bc00a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4bd81990f0 tx=0x7f4bc800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:48.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.291+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4bd0016030 con 0x7f4bd81075f0 2024-05-09T09:16:48.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.480+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f4bd802d1c0 con 0x7f4bd81075f0 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.480+0000 7f4bd4ff9640 1 -- 172.21.15.69:0/1368727593 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) ==== 81+0+590 (secure 0 0 0) 0x7f4bd009c2b0 con 0x7f4bd81075f0 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/4182744829 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1673751021 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1911852009 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1056628795 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:48.480 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:48.481 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:48.481 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/2520675955 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:48.481 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:16:48.481 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/3187678781 2024-05-10T09:09:29.020087+0000 2024-05-09T09:16:48.481 INFO:teuthology.orchestra.run.smithi069.stderr:listed 10 entries 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 msgr2=0x7f4bc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 0x7f4bc00a5050 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f4bd81990f0 tx=0x7f4bc800b390 comp rx=0 tx=0).stop 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 msgr2=0x7f4bd819f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f4bd000eaa0 tx=0x7f4bd000ef60 comp rx=0 tx=0).stop 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 shutdown_connections 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4bc00a2b90 0x7f4bc00a5050 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4bd81075f0 0x7f4bd819f190 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4bd8104cc0 0x7f4bd8198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 --2- 172.21.15.69:0/1368727593 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4bd8102390 0x7f4bd8197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 >> 172.21.15.69:0/1368727593 conn(0x7f4bd80fbd20 msgr2=0x7f4bd8102e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:48.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 shutdown_connections 2024-05-09T09:16:48.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.482+0000 7f4bde0fc640 1 -- 172.21.15.69:0/1368727593 wait complete. 2024-05-09T09:16:48.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:48 smithi069 ceph-mon[30123]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T09:16:48.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2936215385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1641206665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3255776134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.493 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi069.front.sepia.ceph.com... 2024-05-09T09:16:48.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:48.493 DEBUG:teuthology.orchestra.run.smithi069:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-05-09T09:16:48.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi069.stderr:ceph-fuse[73367]: fuse finished with error 0 and tester_r 0 2024-05-09T09:16:48.518 INFO:teuthology.orchestra.run:waiting for 300 2024-05-09T09:16:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:48 smithi163 ceph-mon[34354]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T09:16:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2936215385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1641206665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3255776134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.605 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:16:48.605 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:16:48.606 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-05-09T09:16:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:48 smithi103 ceph-mon[34421]: pgmap v298: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 307 B/s wr, 0 op/s 2024-05-09T09:16:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2936215385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1641206665' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3255776134' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:16:48.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:48.768 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 -- 172.21.15.69:0/1754772038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c102a70 msgr2=0x7f8e4c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 --2- 172.21.15.69:0/1754772038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c102a70 0x7f8e4c102ef0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f8e40009900 tx=0x7f8e4002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 -- 172.21.15.69:0/1754772038 shutdown_connections 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 --2- 172.21.15.69:0/1754772038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4c103430 0x7f8e4c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 --2- 172.21.15.69:0/1754772038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c102a70 0x7f8e4c102ef0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 --2- 172.21.15.69:0/1754772038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c101880 0x7f8e4c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 -- 172.21.15.69:0/1754772038 >> 172.21.15.69:0/1754772038 conn(0x7f8e4c0fce30 msgr2=0x7f8e4c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 -- 172.21.15.69:0/1754772038 shutdown_connections 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 -- 172.21.15.69:0/1754772038 wait complete. 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.767+0000 7f8e51b09640 1 Processor -- start 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 -- start start 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4c102a70 0x7f8e4c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c103430 0x7f8e4c074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d500 con 0x7f8e4c101880 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d380 con 0x7f8e4c103430 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e4c10d680 con 0x7f8e4c102a70 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44784/0 (socket says 172.21.15.69:44784) 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 -- 172.21.15.69:0/4048026269 learned_addr learned my addr 172.21.15.69:0/4048026269 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 -- 172.21.15.69:0/4048026269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4c102a70 msgr2=0x7f8e4c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4c102a70 0x7f8e4c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 -- 172.21.15.69:0/4048026269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c103430 msgr2=0x7f8e4c074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c103430 0x7f8e4c074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 -- 172.21.15.69:0/4048026269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e4c0749f0 con 0x7f8e4c101880 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e4b7fe640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00c970 tx=0x7f8e3c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:48.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c007c10 con 0x7f8e4c101880 2024-05-09T09:16:48.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e3c007db0 con 0x7f8e4c101880 2024-05-09T09:16:48.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e4c1a1c30 con 0x7f8e4c101880 2024-05-09T09:16:48.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.768+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e3c00f940 con 0x7f8e4c101880 2024-05-09T09:16:48.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.769+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e4c1a2180 con 0x7f8e4c101880 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.769+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e4c108b40 con 0x7f8e4c101880 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.771+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f8e3c00fae0 con 0x7f8e4c101880 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.771+0000 7f8e48ff9640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 0x7f8e340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.771+0000 7f8e4affd640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 0x7f8e340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.771+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8e3c0cf770 con 0x7f8e4c101880 2024-05-09T09:16:48.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.772+0000 7f8e4affd640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 0x7f8e340a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e400096c0 tx=0x7f8e40031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:48.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.775+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e3c09c520 con 0x7f8e4c101880 2024-05-09T09:16:48.956 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.956+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f8e4c062570 con 0x7f8e4c101880 2024-05-09T09:16:48.958 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.958+0000 7f8e48ff9640 1 -- 172.21.15.69:0/4048026269 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f8e3c09be20 con 0x7f8e4c101880 2024-05-09T09:16:48.958 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:48.959 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 msgr2=0x7f8e340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 0x7f8e340a5050 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f8e400096c0 tx=0x7f8e40031400 comp rx=0 tx=0).stop 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 msgr2=0x7f8e4c0755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f8e3c00c970 tx=0x7f8e3c00ce30 comp rx=0 tx=0).stop 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 shutdown_connections 2024-05-09T09:16:48.961 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e340a2b90 0x7f8e340a5050 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e4c103430 0x7f8e4c074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e4c102a70 0x7f8e4c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 --2- 172.21.15.69:0/4048026269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e4c101880 0x7f8e4c0755e0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:48.962 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 >> 172.21.15.69:0/4048026269 conn(0x7f8e4c0fce30 msgr2=0x7f8e4c105d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:48.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 shutdown_connections 2024-05-09T09:16:48.963 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:48.960+0000 7f8e51b09640 1 -- 172.21.15.69:0/4048026269 wait complete. 2024-05-09T09:16:48.972 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.072+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/2548355857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd41022f0 msgr2=0x7f2cd41046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.072+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/2548355857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd41022f0 0x7f2cd41046e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4009ca0 tx=0x7f2cc402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/2548355857 shutdown_connections 2024-05-09T09:16:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/2548355857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 0x7f2cd41099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.078 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/2548355857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd4104c20 0x7f2cd4107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/2548355857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd41022f0 0x7f2cd41046e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/2548355857 >> 172.21.15.69:0/2548355857 conn(0x7f2cd40fbce0 msgr2=0x7f2cd40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/2548355857 shutdown_connections 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/2548355857 wait complete. 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 Processor -- start 2024-05-09T09:16:49.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 -- start start 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd41022f0 0x7f2cd4197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.073+0000 7f2cd8c5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 0x7f2cd419f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410d520 con 0x7f2cd4104c20 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410d3a0 con 0x7f2cd4107550 2024-05-09T09:16:49.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd410d6a0 con 0x7f2cd41022f0 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44806/0 (socket says 172.21.15.69:44806) 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 -- 172.21.15.69:0/1755805202 learned_addr learned my addr 172.21.15.69:0/1755805202 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd2d9d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 0x7f2cd419f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd259c640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd41022f0 0x7f2cd4197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 -- 172.21.15.69:0/1755805202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd41022f0 msgr2=0x7f2cd4197c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd41022f0 0x7f2cd4197c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 -- 172.21.15.69:0/1755805202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 msgr2=0x7f2cd419f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 0x7f2cd419f1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 -- 172.21.15.69:0/1755805202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd419f850 con 0x7f2cd4104c20 2024-05-09T09:16:49.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd1d9b640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c970 tx=0x7f2cbc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc007bb0 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cbc007d50 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd419fb40 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cbc0187a0 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.074+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd41a0040 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.075+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cd40fd7d0 con 0x7f2cd4104c20 2024-05-09T09:16:49.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.077+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f2cbc00f450 con 0x7f2cd4104c20 2024-05-09T09:16:49.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.077+0000 7f2ccb7fe640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 0x7f2cc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.077+0000 7f2cd259c640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 0x7f2cc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.077+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f2cbc0cf220 con 0x7f2cd4104c20 2024-05-09T09:16:49.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.078+0000 7f2cd259c640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 0x7f2cc00a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006010 tx=0x7f2cc4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:49.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.081+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2cbc014030 con 0x7f2cd4104c20 2024-05-09T09:16:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.259+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f2cd402d410 con 0x7f2cd4104c20 2024-05-09T09:16:49.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:49 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/485943499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:49.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:49 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1368727593' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:49.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:49 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4048026269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.261+0000 7f2ccb7fe640 1 -- 172.21.15.69:0/1755805202 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f2cbc09bfe0 con 0x7f2cd4104c20 2024-05-09T09:16:49.261 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:49.261 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:49.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.262+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 msgr2=0x7f2cc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.262+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 0x7f2cc00a5050 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f2cc4006010 tx=0x7f2cc4000f40 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.262+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 msgr2=0x7f2cd4198180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.262+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f2cbc00c970 tx=0x7f2cbc00ce30 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 shutdown_connections 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc00a2b90 0x7f2cc00a5050 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd4107550 0x7f2cd419f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd4104c20 0x7f2cd4198180 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 --2- 172.21.15.69:0/1755805202 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd41022f0 0x7f2cd4197c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 >> 172.21.15.69:0/1755805202 conn(0x7f2cd40fbce0 msgr2=0x7f2cd4102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 shutdown_connections 2024-05-09T09:16:49.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.263+0000 7f2cd8c5d640 1 -- 172.21.15.69:0/1755805202 wait complete. 2024-05-09T09:16:49.274 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-05-09T09:16:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:49 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/485943499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:49 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1368727593' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:49 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4048026269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 -- 172.21.15.69:0/2742773292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34102350 msgr2=0x7fef34104740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 --2- 172.21.15.69:0/2742773292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34102350 0x7fef34104740 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fef24009ca0 tx=0x7fef2402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 -- 172.21.15.69:0/2742773292 shutdown_connections 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 --2- 172.21.15.69:0/2742773292 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef341075b0 0x7fef341099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 --2- 172.21.15.69:0/2742773292 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef34104c80 0x7fef34107070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 --2- 172.21.15.69:0/2742773292 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34102350 0x7fef34104740 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 -- 172.21.15.69:0/2742773292 >> 172.21.15.69:0/2742773292 conn(0x7fef340fbce0 msgr2=0x7fef340fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 -- 172.21.15.69:0/2742773292 shutdown_connections 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.378+0000 7fef39822640 1 -- 172.21.15.69:0/2742773292 wait complete. 2024-05-09T09:16:49.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 Processor -- start 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 -- start start 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef34102350 0x7fef34197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341075b0 0x7fef3419f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410d4e0 con 0x7fef34104c80 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410d360 con 0x7fef341075b0 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef39822640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef3410d660 con 0x7fef34102350 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.379+0000 7fef327fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44826/0 (socket says 172.21.15.69:44826) 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 -- 172.21.15.69:0/3038767805 learned_addr learned my addr 172.21.15.69:0/3038767805 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:49.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef32ffd640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef34102350 0x7fef34197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef337fe640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341075b0 0x7fef3419f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 -- 172.21.15.69:0/3038767805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef34102350 msgr2=0x7fef34197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef34102350 0x7fef34197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 -- 172.21.15.69:0/3038767805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341075b0 msgr2=0x7fef3419f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341075b0 0x7fef3419f110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:49.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 -- 172.21.15.69:0/3038767805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef3419f780 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef327fc640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c970 tx=0x7fef1c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c007bb0 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef3419fa70 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef1c007d50 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef1c00f450 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.380+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef3419ffc0 con 0x7fef34104c80 2024-05-09T09:16:49.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.381+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef340fd7d0 con 0x7fef34104c80 2024-05-09T09:16:49.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.383+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fef1c004f70 con 0x7fef34104c80 2024-05-09T09:16:49.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.384+0000 7fef38820640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 0x7fef200a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:49.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.384+0000 7fef32ffd640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 0x7fef200a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:49.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.384+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fef1c0cf4e0 con 0x7fef34104c80 2024-05-09T09:16:49.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.384+0000 7fef32ffd640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 0x7fef200a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef24004af0 tx=0x7fef24000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:49.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.387+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef1c014030 con 0x7fef34104c80 2024-05-09T09:16:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:49 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/485943499' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:16:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:49 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1368727593' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:16:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:49 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4048026269' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:49.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:49.601+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) -- 0x7fef3402d1c0 con 0x7fef34104c80 2024-05-09T09:16:50.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.174+0000 7fef38820640 1 -- 172.21.15.69:0/3038767805 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v18) ==== 118+0+0 (secure 0 0 0) 0x7fef1c09c1a0 con 0x7fef34104c80 2024-05-09T09:16:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 msgr2=0x7fef200a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 0x7fef200a5050 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fef24004af0 tx=0x7fef24000f40 comp rx=0 tx=0).stop 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 msgr2=0x7fef341980a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fef1c00c970 tx=0x7fef1c00ce30 comp rx=0 tx=0).stop 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 shutdown_connections 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef200a2b90 0x7fef200a5050 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef341075b0 0x7fef3419f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef34104c80 0x7fef341980a0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 --2- 172.21.15.69:0/3038767805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef34102350 0x7fef34197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 >> 172.21.15.69:0/3038767805 conn(0x7fef340fbce0 msgr2=0x7fef34102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 shutdown_connections 2024-05-09T09:16:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.177+0000 7fef39822640 1 -- 172.21.15.69:0/3038767805 wait complete. 2024-05-09T09:16:50.191 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 3, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-05-09T09:16:50.192 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 3 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- 172.21.15.69:0/1862236287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c101960 msgr2=0x7f644c101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 --2- 172.21.15.69:0/1862236287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c101960 0x7f644c101de0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f6434009900 tx=0x7f643402a410 comp rx=0 tx=0).stop 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- 172.21.15.69:0/1862236287 shutdown_connections 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 --2- 172.21.15.69:0/1862236287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c102320 0x7f644c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 --2- 172.21.15.69:0/1862236287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c101960 0x7f644c101de0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 --2- 172.21.15.69:0/1862236287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c100770 0x7f644c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- 172.21.15.69:0/1862236287 >> 172.21.15.69:0/1862236287 conn(0x7f644c0fbce0 msgr2=0x7f644c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:50.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- 172.21.15.69:0/1862236287 shutdown_connections 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- 172.21.15.69:0/1862236287 wait complete. 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 Processor -- start 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 -- start start 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.302+0000 7f6451b09640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c101960 0x7f644c1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c102320 0x7f644c19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10c420 con 0x7f644c100770 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10c2a0 con 0x7f644c102320 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f644c10c5a0 con 0x7f644c101960 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44844/0 (socket says 172.21.15.69:44844) 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 -- 172.21.15.69:0/136242497 learned_addr learned my addr 172.21.15.69:0/136242497 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6443fff640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c101960 0x7f644c1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644bfff640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c102320 0x7f644c19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:50.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 -- 172.21.15.69:0/136242497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c101960 msgr2=0x7f644c1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c101960 0x7f644c1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 -- 172.21.15.69:0/136242497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c102320 msgr2=0x7f644c19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c102320 0x7f644c19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 -- 172.21.15.69:0/136242497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f644c19f7b0 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f644b7fe640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f643c00e9c0 tx=0x7f643c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c00cd80 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f643c0035f0 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f644c19faa0 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f643c010690 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.303+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f644c1a0010 con 0x7f644c100770 2024-05-09T09:16:50.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.304+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f644c1a01e0 con 0x7f644c100770 2024-05-09T09:16:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.305+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f643c0037a0 con 0x7f644c100770 2024-05-09T09:16:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.306+0000 7f64497fa640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 0x7f64380a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.306+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f643c0cf770 con 0x7f644c100770 2024-05-09T09:16:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.306+0000 7f6443fff640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 0x7f64380a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:50.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.307+0000 7f6443fff640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 0x7f64380a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6434006010 tx=0x7f6434000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:50.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.310+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f643c09c490 con 0x7f644c100770 2024-05-09T09:16:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:50 smithi069 ceph-mon[30123]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s wr, 0 op/s 2024-05-09T09:16:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1755805202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T09:16:50.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:50.526+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"} v 0) -- 0x7f644c1a03b0 con 0x7f644c100770 2024-05-09T09:16:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:50 smithi163 ceph-mon[34354]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s wr, 0 op/s 2024-05-09T09:16:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1755805202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T09:16:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:50 smithi103 ceph-mon[34421]: pgmap v299: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 274 B/s wr, 0 op/s 2024-05-09T09:16:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1755805202' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-05-09T09:16:51.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.179+0000 7f64497fa640 1 -- 172.21.15.69:0/136242497 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]=0 v19) ==== 105+0+0 (secure 0 0 0) 0x7f643c09bd90 con 0x7f644c100770 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 msgr2=0x7f64380a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 0x7f64380a50d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f6434006010 tx=0x7f6434000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 msgr2=0x7f644c197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f643c00e9c0 tx=0x7f643c00ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 shutdown_connections 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64380a2c10 0x7f64380a50d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f644c102320 0x7f644c19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f644c101960 0x7f644c1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 --2- 172.21.15.69:0/136242497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f644c100770 0x7f644c197b90 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 >> 172.21.15.69:0/136242497 conn(0x7f644c0fbce0 msgr2=0x7f644c0fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 shutdown_connections 2024-05-09T09:16:51.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.182+0000 7f6451b09640 1 -- 172.21.15.69:0/136242497 wait complete. 2024-05-09T09:16:51.195 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-05-09T09:16:51.304 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 -- 172.21.15.69:0/3730476020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918101960 msgr2=0x7f0918101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 --2- 172.21.15.69:0/3730476020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918101960 0x7f0918101de0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f090c009900 tx=0x7f090c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 -- 172.21.15.69:0/3730476020 shutdown_connections 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 --2- 172.21.15.69:0/3730476020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918102320 0x7f0918108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 --2- 172.21.15.69:0/3730476020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918101960 0x7f0918101de0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 --2- 172.21.15.69:0/3730476020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918100770 0x7f0918100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 -- 172.21.15.69:0/3730476020 >> 172.21.15.69:0/3730476020 conn(0x7f09180fbce0 msgr2=0x7f09180fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 -- 172.21.15.69:0/3730476020 shutdown_connections 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.302+0000 7f091fbb6640 1 -- 172.21.15.69:0/3730476020 wait complete. 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 Processor -- start 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 -- start start 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 0x7f091819c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918102320 0x7f091819ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:51.305 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c3f0 con 0x7f0918100770 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c270 con 0x7f0918101960 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091fbb6640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f091810c570 con 0x7f0918102320 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44858/0 (socket says 172.21.15.69:44858) 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 -- 172.21.15.69:0/4221268563 learned_addr learned my addr 172.21.15.69:0/4221268563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d151640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 0x7f091819c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091e153640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918102320 0x7f091819ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 -- 172.21.15.69:0/4221268563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918102320 msgr2=0x7f091819ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918102320 0x7f091819ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 -- 172.21.15.69:0/4221268563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 msgr2=0x7f091819c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 0x7f091819c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d952640 1 -- 172.21.15.69:0/4221268563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f091819f810 con 0x7f0918100770 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.303+0000 7f091d151640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 0x7f091819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:51.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f091d952640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f091400e9c0 tx=0x7f091400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f091400cd80 con 0x7f0918100770 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09140035f0 con 0x7f0918100770 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f091819fb00 con 0x7f0918100770 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0914010690 con 0x7f0918100770 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.304+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f091819ffd0 con 0x7f0918100770 2024-05-09T09:16:51.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.305+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0918107ae0 con 0x7f0918100770 2024-05-09T09:16:51.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.306+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f09140037a0 con 0x7f0918100770 2024-05-09T09:16:51.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.306+0000 7f090affd640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 0x7f09040a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:51.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.307+0000 7f091d151640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 0x7f09040a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:51.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.307+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f09140cf770 con 0x7f0918100770 2024-05-09T09:16:51.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.307+0000 7f091d151640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 0x7f09040a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f090c000c00 tx=0x7f090c0095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:51.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.311+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f091409c490 con 0x7f0918100770 2024-05-09T09:16:51.490 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T09:16:51.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:51 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:51.491 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T09:16:51.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T09:16:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:51 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T09:16:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:51.527+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) -- 0x7f091802d1c0 con 0x7f0918100770 2024-05-09T09:16:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3038767805' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-05-09T09:16:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:51 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]: dispatch 2024-05-09T09:16:52.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.188+0000 7f090affd640 1 -- 172.21.15.69:0/4221268563 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v21) ==== 121+0+0 (secure 0 0 0) 0x7f091409bd90 con 0x7f0918100770 2024-05-09T09:16:52.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.190+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 msgr2=0x7f09040a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.190+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 0x7f09040a5050 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f090c000c00 tx=0x7f090c0095f0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.190+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 msgr2=0x7f0918069590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.190+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f091400e9c0 tx=0x7f091400ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 shutdown_connections 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09040a2b90 0x7f09040a5050 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0918102320 0x7f091819ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0918101960 0x7f091819c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 --2- 172.21.15.69:0/4221268563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0918100770 0x7f0918069590 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 >> 172.21.15.69:0/4221268563 conn(0x7f09180fbce0 msgr2=0x7f0918104cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 shutdown_connections 2024-05-09T09:16:52.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.191+0000 7f091fbb6640 1 -- 172.21.15.69:0/4221268563 wait complete. 2024-05-09T09:16:52.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-05-09T09:16:52.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 -- 172.21.15.69:0/2892779167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 msgr2=0x7f70d0104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 --2- 172.21.15.69:0/2892779167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0104700 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009900 tx=0x7f70c002a410 comp rx=0 tx=0).stop 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 -- 172.21.15.69:0/2892779167 shutdown_connections 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 --2- 172.21.15.69:0/2892779167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0107570 0x7f70d01099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 --2- 172.21.15.69:0/2892779167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0104c40 0x7f70d0107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 --2- 172.21.15.69:0/2892779167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0104700 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 -- 172.21.15.69:0/2892779167 >> 172.21.15.69:0/2892779167 conn(0x7f70d00fbd00 msgr2=0x7f70d00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 -- 172.21.15.69:0/2892779167 shutdown_connections 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 -- 172.21.15.69:0/2892779167 wait complete. 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.312+0000 7f70d6790640 1 Processor -- start 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 -- start start 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0104c40 0x7f70d0198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:52.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0107570 0x7f70d019f180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70d010d530 con 0x7f70d0102310 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70d010d3b0 con 0x7f70d0107570 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d6790640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70d010d6b0 con 0x7f70d0104c40 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44876/0 (socket says 172.21.15.69:44876) 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 -- 172.21.15.69:0/1068788752 learned_addr learned my addr 172.21.15.69:0/1068788752 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70d4d2d640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0107570 0x7f70d019f180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cf7fe640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0104c40 0x7f70d0198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 -- 172.21.15.69:0/1068788752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0104c40 msgr2=0x7f70d0198110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0104c40 0x7f70d0198110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 -- 172.21.15.69:0/1068788752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0107570 msgr2=0x7f70d019f180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0107570 0x7f70d019f180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 -- 172.21.15.69:0/1068788752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70d019f7f0 con 0x7f70d0102310 2024-05-09T09:16:52.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.313+0000 7f70cffff640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009230 tx=0x7f70c0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.314+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c003f030 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.314+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70c0003ad0 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.314+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70d019fa80 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.314+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70c0035070 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.314+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70d019ffc0 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.315+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f70d00fd7f0 con 0x7f70d0102310 2024-05-09T09:16:52.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.316+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f70c0042030 con 0x7f70d0102310 2024-05-09T09:16:52.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.316+0000 7f70cd7fa640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 0x7f70b80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:52.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.317+0000 7f70cf7fe640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 0x7f70b80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:52.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.317+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f70c00ef790 con 0x7f70d0102310 2024-05-09T09:16:52.319 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.317+0000 7f70cf7fe640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 0x7f70b80a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f70d01990e0 tx=0x7f70c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:52.320 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.321+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f70c00bc4b0 con 0x7f70d0102310 2024-05-09T09:16:52.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T09:16:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:52 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T09:16:52.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:52.537+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) -- 0x7f70d0062430 con 0x7f70d0102310 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T09:16:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:52 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: pgmap v300: 129 pgs: 129 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/136242497' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "3"}]': finished 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: daemon mds.i assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-05-09T09:16:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: fsmap cephfs:2 {0=b=up:active,1=i=up:creating} 10 up:standby 2024-05-09T09:16:52.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: daemon mds.i is now active in filesystem cephfs as rank 1 2024-05-09T09:16:52.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:52 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-05-09T09:16:53.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.201+0000 7f70cd7fa640 1 -- 172.21.15.69:0/1068788752 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v23) ==== 115+0+0 (secure 0 0 0) 0x7f70c00bbdb0 con 0x7f70d0102310 2024-05-09T09:16:53.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.204+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 msgr2=0x7f70b80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.204+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 0x7f70b80a50d0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f70d01990e0 tx=0x7f70c4005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:53.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.204+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 msgr2=0x7f70d0197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.204+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f70c0009230 tx=0x7f70c0031b60 comp rx=0 tx=0).stop 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 shutdown_connections 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70b80a2c10 0x7f70b80a50d0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70d0107570 0x7f70d019f180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70d0104c40 0x7f70d0198110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 --2- 172.21.15.69:0/1068788752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70d0102310 0x7f70d0197bd0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 >> 172.21.15.69:0/1068788752 conn(0x7f70d00fbd00 msgr2=0x7f70d0102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 shutdown_connections 2024-05-09T09:16:53.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.205+0000 7f70d6790640 1 -- 172.21.15.69:0/1068788752 wait complete. 2024-05-09T09:16:53.218 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-05-09T09:16:53.218 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-05-09T09:16:53.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: Cluster is now healthy 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-05-09T09:16:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 -- 172.21.15.69:0/789144589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4101970 msgr2=0x7effb4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 --2- 172.21.15.69:0/789144589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4101970 0x7effb4101df0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7eff9c009900 tx=0x7eff9c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 -- 172.21.15.69:0/789144589 shutdown_connections 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 --2- 172.21.15.69:0/789144589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4102330 0x7effb4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 --2- 172.21.15.69:0/789144589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4101970 0x7effb4101df0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 --2- 172.21.15.69:0/789144589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 -- 172.21.15.69:0/789144589 >> 172.21.15.69:0/789144589 conn(0x7effb40fbcf0 msgr2=0x7effb40fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 -- 172.21.15.69:0/789144589 shutdown_connections 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.324+0000 7effb9688640 1 -- 172.21.15.69:0/789144589 wait complete. 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 Processor -- start 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 -- start start 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4101970 0x7effb4198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d520 con 0x7effb4102330 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d3a0 con 0x7effb4100780 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb9688640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7effb410d6a0 con 0x7effb4101970 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44900/0 (socket says 172.21.15.69:44900) 2024-05-09T09:16:53.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb37fe640 1 -- 172.21.15.69:0/4023866505 learned_addr learned my addr 172.21.15.69:0/4023866505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effabfff640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4101970 0x7effb4198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 -- 172.21.15.69:0/4023866505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4101970 msgr2=0x7effb4198190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4101970 0x7effb4198190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 -- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 msgr2=0x7effb419f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb2ffd640 1 -- 172.21.15.69:0/4023866505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effb419f860 con 0x7effb4100780 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.325+0000 7effb37fe640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb2ffd640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4197c50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effa400e9c0 tx=0x7effa400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa400cd80 con 0x7effb4100780 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effb419fb50 con 0x7effb4100780 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effa4010910 con 0x7effb4100780 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effa4018980 con 0x7effb4100780 2024-05-09T09:16:53.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.326+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effb41a00a0 con 0x7effb4100780 2024-05-09T09:16:53.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.327+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7effb4107b10 con 0x7effb4100780 2024-05-09T09:16:53.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.328+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7effa401f020 con 0x7effb4100780 2024-05-09T09:16:53.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.328+0000 7effb0ff9640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.329+0000 7effabfff640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.329+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7effa40cf410 con 0x7effb4100780 2024-05-09T09:16:53.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.329+0000 7effabfff640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 0x7effa00a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000c00 tx=0x7eff9c0315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:53.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.332+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7effa409c0d0 con 0x7effb4100780 2024-05-09T09:16:53.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.532+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7effb4061c70 con 0x7effa00a2b90 2024-05-09T09:16:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.554+0000 7effb0ff9640 1 -- 172.21.15.69:0/4023866505 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7effb4061c70 con 0x7effa00a2b90 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: Cluster is now healthy 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-05-09T09:16:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 msgr2=0x7effa00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 0x7effa00a5050 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7eff9c000c00 tx=0x7eff9c0315b0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 msgr2=0x7effb4197c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4197c50 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7effa400e9c0 tx=0x7effa400ee80 comp rx=0 tx=0).stop 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 shutdown_connections 2024-05-09T09:16:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effa00a2b90 0x7effa00a5050 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effb4102330 0x7effb419f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effb4101970 0x7effb4198190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 --2- 172.21.15.69:0/4023866505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effb4100780 0x7effb4197c50 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 >> 172.21.15.69:0/4023866505 conn(0x7effb40fbcf0 msgr2=0x7effb4104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 shutdown_connections 2024-05-09T09:16:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.556+0000 7effb9688640 1 -- 172.21.15.69:0/4023866505 wait complete. 2024-05-09T09:16:53.568 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4221268563' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: daemon mds.h assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: Cluster is now healthy 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: fsmap cephfs:2 {0=b=up:active,1=i=up:active} 10 up:standby 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:creating} 9 up:standby 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: daemon mds.h is now active in filesystem cephfs as rank 2 2024-05-09T09:16:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 -- 172.21.15.69:0/2071526045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8100780 msgr2=0x7facb8100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 --2- 172.21.15.69:0/2071526045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8100780 0x7facb8100ba0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7faca8009900 tx=0x7faca802a410 comp rx=0 tx=0).stop 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 -- 172.21.15.69:0/2071526045 shutdown_connections 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 --2- 172.21.15.69:0/2071526045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb8108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 --2- 172.21.15.69:0/2071526045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8101970 0x7facb8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 --2- 172.21.15.69:0/2071526045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8100780 0x7facb8100ba0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 -- 172.21.15.69:0/2071526045 >> 172.21.15.69:0/2071526045 conn(0x7facb80fbcf0 msgr2=0x7facb80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:53.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 -- 172.21.15.69:0/2071526045 shutdown_connections 2024-05-09T09:16:53.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 -- 172.21.15.69:0/2071526045 wait complete. 2024-05-09T09:16:53.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.716+0000 7facbd568640 1 Processor -- start 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 -- start start 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8100780 0x7facb8197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7facb810c3d0 con 0x7facb8101970 2024-05-09T09:16:53.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7facb810c250 con 0x7facb8102330 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7facb810c550 con 0x7facb8100780 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44924/0 (socket says 172.21.15.69:44924) 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 -- 172.21.15.69:0/2911922021 learned_addr learned my addr 172.21.15.69:0/2911922021 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb77fe640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 -- 172.21.15.69:0/2911922021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8100780 msgr2=0x7facb8197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb6ffd640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8100780 0x7facb8197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8100780 0x7facb8197c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 -- 172.21.15.69:0/2911922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 msgr2=0x7facb819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 -- 172.21.15.69:0/2911922021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7facb819f860 con 0x7facb8101970 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb77fe640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facb67fc640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7facac00d950 tx=0x7facac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac017030 con 0x7facb8101970 2024-05-09T09:16:53.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7facac004020 con 0x7facb8101970 2024-05-09T09:16:53.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.717+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7facb819fb50 con 0x7facb8101970 2024-05-09T09:16:53.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.718+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7facac0047a0 con 0x7facb8101970 2024-05-09T09:16:53.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.718+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7facb81a00a0 con 0x7facb8101970 2024-05-09T09:16:53.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.718+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7facb81a02d0 con 0x7facb8101970 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.720+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7facac0041d0 con 0x7facb8101970 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.720+0000 7fac9ffff640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 0x7faca00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.720+0000 7facb6ffd640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 0x7faca00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.720+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7facac0cf490 con 0x7facb8101970 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.721+0000 7facb6ffd640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 0x7faca00a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faca8031610 tx=0x7faca8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:53.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.724+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7facac09c1b0 con 0x7facb8101970 2024-05-09T09:16:53.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.925+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7facb806a240 con 0x7faca00a2b90 2024-05-09T09:16:53.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.948+0000 7fac9ffff640 1 -- 172.21.15.69:0/2911922021 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7facb806a240 con 0x7faca00a2b90 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 msgr2=0x7faca00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 0x7faca00a5050 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7faca8031610 tx=0x7faca8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 msgr2=0x7facb8198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7facac00d950 tx=0x7facac00de10 comp rx=0 tx=0).stop 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 shutdown_connections 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faca00a2b90 0x7faca00a5050 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7facb8102330 0x7facb819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7facb8101970 0x7facb8198190 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 --2- 172.21.15.69:0/2911922021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7facb8100780 0x7facb8197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:53.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 >> 172.21.15.69:0/2911922021 conn(0x7facb80fbcf0 msgr2=0x7facb8069b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:53.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 shutdown_connections 2024-05-09T09:16:53.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:53.950+0000 7facbd568640 1 -- 172.21.15.69:0/2911922021 wait complete. 2024-05-09T09:16:53.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:16:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:16:53.537+0000 7fb1b4fa5700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T09:16:53.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:16:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:16:53.537+0000 7fb1b4fa5700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T09:16:53.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:16:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:16:53.537+0000 7fb1b4fa5700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T09:16:53.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:16:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:16:53.537+0000 7fb1b4fa5700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T09:16:53.960 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:16:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:16:53.537+0000 7fb1b4fa5700 -1 client.0 error registering admin socket command: (17) File exists 2024-05-09T09:16:53.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-05-09T09:16:54.069 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.067+0000 7f301a864640 1 -- 172.21.15.69:0/41124799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014100790 msgr2=0x7f3014100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.067+0000 7f301a864640 1 --2- 172.21.15.69:0/41124799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014100790 0x7f3014100bb0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f3004009900 tx=0x7f300402a410 comp rx=0 tx=0).stop 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- 172.21.15.69:0/41124799 shutdown_connections 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- 172.21.15.69:0/41124799 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014102340 0x7f3014108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- 172.21.15.69:0/41124799 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101980 0x7f3014101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- 172.21.15.69:0/41124799 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014100790 0x7f3014100bb0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- 172.21.15.69:0/41124799 >> 172.21.15.69:0/41124799 conn(0x7f30140fbd00 msgr2=0x7f30140fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- 172.21.15.69:0/41124799 shutdown_connections 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- 172.21.15.69:0/41124799 wait complete. 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 Processor -- start 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- start start 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014100790 0x7f301406a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101980 0x7f301419c970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f301410c400 con 0x7f3014102340 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f301410c280 con 0x7f3014101980 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f301a864640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f301410c580 con 0x7f3014100790 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f3018e01640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f3018e01640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44956/0 (socket says 172.21.15.69:44956) 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.068+0000 7f3018e01640 1 -- 172.21.15.69:0/3194146425 learned_addr learned my addr 172.21.15.69:0/3194146425 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f300bfff640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101980 0x7f301419c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 -- 172.21.15.69:0/3194146425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014100790 msgr2=0x7f301406a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3013fff640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014100790 0x7f301406a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014100790 0x7f301406a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 -- 172.21.15.69:0/3194146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101980 msgr2=0x7f301419c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101980 0x7f301419c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 -- 172.21.15.69:0/3194146425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f301419f8a0 con 0x7f3014102340 2024-05-09T09:16:54.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3018e01640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f300c009fd0 tx=0x7f300c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c017070 con 0x7f3014102340 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f300c003e50 con 0x7f3014102340 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f301419fb30 con 0x7f3014102340 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f300c0106c0 con 0x7f3014102340 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.069+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30141a0080 con 0x7f3014102340 2024-05-09T09:16:54.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.070+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3014107b80 con 0x7f3014102340 2024-05-09T09:16:54.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.071+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f300c003970 con 0x7f3014102340 2024-05-09T09:16:54.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.072+0000 7f3011ffb640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 0x7f30000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.072+0000 7f3013fff640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 0x7f30000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.072+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f300c0cef90 con 0x7f3014102340 2024-05-09T09:16:54.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.073+0000 7f3013fff640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 0x7f30000a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3004031610 tx=0x7f3004000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.077+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f300c09bcb0 con 0x7f3014102340 2024-05-09T09:16:54.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:54 smithi069 ceph-mon[30123]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-09T09:16:54.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T09:16:54.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:54 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:16:54.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:54 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-05-09T09:16:54.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:54 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:16:54.285 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.285+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) -- 0x7f3014061d00 con 0x7f30000a2c10 2024-05-09T09:16:54.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.290+0000 7f3011ffb640 1 -- 172.21.15.69:0/3194146425 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f3014061d00 con 0x7f30000a2c10 2024-05-09T09:16:54.290 INFO:teuthology.orchestra.run.smithi069.stdout:/volumes/qa/sv_0/ccb8f9fc-6bdd-41de-b91d-864418e75755 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 msgr2=0x7f30000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 0x7f30000a50d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f3004031610 tx=0x7f3004000bc0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 msgr2=0x7f301419ef90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f300c009fd0 tx=0x7f300c00b260 comp rx=0 tx=0).stop 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 shutdown_connections 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f30000a2c10 0x7f30000a50d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014102340 0x7f301419ef90 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101980 0x7f301419c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 --2- 172.21.15.69:0/3194146425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014100790 0x7f301406a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 >> 172.21.15.69:0/3194146425 conn(0x7f30140fbd00 msgr2=0x7f3014104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 shutdown_connections 2024-05-09T09:16:54.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.292+0000 7f301a864640 1 -- 172.21.15.69:0/3194146425 wait complete. 2024-05-09T09:16:54.304 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-05-09T09:16:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 -- 172.21.15.69:0/102817565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 msgr2=0x7f10941070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 --2- 172.21.15.69:0/102817565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941070b0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f107c009900 tx=0x7f107c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:54.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 -- 172.21.15.69:0/102817565 shutdown_connections 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 --2- 172.21.15.69:0/102817565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 0x7f10941099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 --2- 172.21.15.69:0/102817565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941070b0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 --2- 172.21.15.69:0/102817565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 0x7f1094104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 -- 172.21.15.69:0/102817565 >> 172.21.15.69:0/102817565 conn(0x7f10940fbce0 msgr2=0x7f10940fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 -- 172.21.15.69:0/102817565 shutdown_connections 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.410+0000 7f1099c9e640 1 -- 172.21.15.69:0/102817565 wait complete. 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 Processor -- start 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 -- start start 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 0x7f1094197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 0x7f109419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f109410d4c0 con 0x7f1094104cc0 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f109410d340 con 0x7f10941075f0 2024-05-09T09:16:54.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1099c9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f109410d640 con 0x7f1094102390 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1092ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1092ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44966/0 (socket says 172.21.15.69:44966) 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1092ffd640 1 -- 172.21.15.69:0/1135344270 learned_addr learned my addr 172.21.15.69:0/1135344270 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.411+0000 7f1093fff640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 0x7f109419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f10937fe640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 0x7f1094197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 -- 172.21.15.69:0/1135344270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 msgr2=0x7f1094197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 0x7f1094197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 -- 172.21.15.69:0/1135344270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 msgr2=0x7f109419f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 0x7f109419f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 -- 172.21.15.69:0/1135344270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f109419f790 con 0x7f1094104cc0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1092ffd640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f107c031610 tx=0x7f107c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c03f030 con 0x7f1094104cc0 2024-05-09T09:16:54.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f109419fa20 con 0x7f1094104cc0 2024-05-09T09:16:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f107c003ce0 con 0x7f1094104cc0 2024-05-09T09:16:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f107c035070 con 0x7f1094104cc0 2024-05-09T09:16:54.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.412+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f109419ff60 con 0x7f1094104cc0 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.413+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f10940fd7d0 con 0x7f1094104cc0 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.415+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f107c042030 con 0x7f1094104cc0 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.416+0000 7f1090ff9640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 0x7f10800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.416+0000 7f10937fe640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 0x7f10800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.416+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f107c0ef9b0 con 0x7f1094104cc0 2024-05-09T09:16:54.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.416+0000 7f10937fe640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 0x7f10800a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f108c005fd0 tx=0x7f108c009760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.419+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f107c0bc080 con 0x7f1094104cc0 2024-05-09T09:16:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:54 smithi163 ceph-mon[34354]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-09T09:16:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T09:16:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:54 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:16:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:54 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-05-09T09:16:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:54 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:16:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:54 smithi103 ceph-mon[34421]: pgmap v301: 129 pgs: 129 active+clean; 582 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 2 op/s 2024-05-09T09:16:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1068788752' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-05-09T09:16:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:54 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:16:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:54 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 9 up:standby 2024-05-09T09:16:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:54 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:16:54.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.623+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) -- 0x7f1094061d00 con 0x7f10800a2b90 2024-05-09T09:16:54.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.647+0000 7f1090ff9640 1 -- 172.21.15.69:0/1135344270 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+0 (secure 0 0 0) 0x7f1094061d00 con 0x7f10800a2b90 2024-05-09T09:16:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 msgr2=0x7f10800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 0x7f10800a5050 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f108c005fd0 tx=0x7f108c009760 comp rx=0 tx=0).stop 2024-05-09T09:16:54.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 msgr2=0x7f10941980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f107c031610 tx=0x7f107c0043b0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 shutdown_connections 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f10800a2b90 0x7f10800a5050 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f10941075f0 0x7f109419f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1094104cc0 0x7f10941980b0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 --2- 172.21.15.69:0/1135344270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1094102390 0x7f1094197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.649+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 >> 172.21.15.69:0/1135344270 conn(0x7f10940fbce0 msgr2=0x7f1094102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.650+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 shutdown_connections 2024-05-09T09:16:54.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.650+0000 7f1099c9e640 1 -- 172.21.15.69:0/1135344270 wait complete. 2024-05-09T09:16:54.661 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-05-09T09:16:54.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 -- 172.21.15.69:0/3344693127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598101980 msgr2=0x7f6598101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 --2- 172.21.15.69:0/3344693127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598101980 0x7f6598101e00 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f658c009900 tx=0x7f658c02a410 comp rx=0 tx=0).stop 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 -- 172.21.15.69:0/3344693127 shutdown_connections 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 --2- 172.21.15.69:0/3344693127 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598102340 0x7f6598108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 --2- 172.21.15.69:0/3344693127 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598101980 0x7f6598101e00 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 --2- 172.21.15.69:0/3344693127 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 -- 172.21.15.69:0/3344693127 >> 172.21.15.69:0/3344693127 conn(0x7f65980fbd20 msgr2=0x7f65980fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 -- 172.21.15.69:0/3344693127 shutdown_connections 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.767+0000 7f659e215640 1 -- 172.21.15.69:0/3344693127 wait complete. 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 Processor -- start 2024-05-09T09:16:54.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 -- start start 2024-05-09T09:16:54.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598199e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598101980 0x7f659819a3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f659810c410 con 0x7f6598102340 2024-05-09T09:16:54.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f659810c290 con 0x7f6598100790 2024-05-09T09:16:54.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f659e215640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f659810c590 con 0x7f6598101980 2024-05-09T09:16:54.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44996/0 (socket says 172.21.15.69:44996) 2024-05-09T09:16:54.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f65977fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598199e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 -- 172.21.15.69:0/2918725664 learned_addr learned my addr 172.21.15.69:0/2918725664 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6596ffd640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598101980 0x7f659819a3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 -- 172.21.15.69:0/2918725664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598101980 msgr2=0x7f659819a3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598101980 0x7f659819a3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 -- 172.21.15.69:0/2918725664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 msgr2=0x7f6598199e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598199e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f6597fff640 1 -- 172.21.15.69:0/2918725664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65981a1a80 con 0x7f6598102340 2024-05-09T09:16:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.768+0000 7f65977fe640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:16:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f6597fff640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f659000eaa0 tx=0x7f659000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f659000cdb0 con 0x7f6598102340 2024-05-09T09:16:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6590005020 con 0x7f6598102340 2024-05-09T09:16:54.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65981a1d70 con 0x7f6598102340 2024-05-09T09:16:54.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65900116b0 con 0x7f6598102340 2024-05-09T09:16:54.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.769+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65981a2230 con 0x7f6598102340 2024-05-09T09:16:54.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.770+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65981a24c0 con 0x7f6598102340 2024-05-09T09:16:54.776 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.771+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7f6590005560 con 0x7f6598102340 2024-05-09T09:16:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.771+0000 7f6594ff9640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.772+0000 7f65977fe640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.772+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f65900cf4d0 con 0x7f6598102340 2024-05-09T09:16:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.772+0000 7f65977fe640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 0x7f65800a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65981017e0 tx=0x7f6588005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:54.777 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.776+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6590016030 con 0x7f6598102340 2024-05-09T09:16:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.980+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) -- 0x7f65980fdef0 con 0x7f65800a2b90 2024-05-09T09:16:54.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.985+0000 7f6594ff9640 1 -- 172.21.15.69:0/2918725664 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+54 (secure 0 0 0) 0x7f65980fdef0 con 0x7f65800a2b90 2024-05-09T09:16:54.984 INFO:teuthology.orchestra.run.smithi069.stdout:/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 msgr2=0x7f65800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 0x7f65800a5050 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f65981017e0 tx=0x7f6588005f90 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 msgr2=0x7f65981a1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f659000eaa0 tx=0x7f659000ef60 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 shutdown_connections 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65800a2b90 0x7f65800a5050 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6598102340 0x7f65981a1410 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6598101980 0x7f659819a3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 --2- 172.21.15.69:0/2918725664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6598100790 0x7f6598199e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 >> 172.21.15.69:0/2918725664 conn(0x7f65980fbd20 msgr2=0x7f65980fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 shutdown_connections 2024-05-09T09:16:54.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:54.987+0000 7f659e215640 1 -- 172.21.15.69:0/2918725664 wait complete. 2024-05-09T09:16:54.997 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- 172.21.15.69:0/2230203154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8381018a0 msgr2=0x7fa838101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 --2- 172.21.15.69:0/2230203154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8381018a0 0x7fa838101cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa830009ca0 tx=0x7fa83002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- 172.21.15.69:0/2230203154 shutdown_connections 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 --2- 172.21.15.69:0/2230203154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838103450 0x7fa838109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 --2- 172.21.15.69:0/2230203154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa838102a90 0x7fa838102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 --2- 172.21.15.69:0/2230203154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa8381018a0 0x7fa838101cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- 172.21.15.69:0/2230203154 >> 172.21.15.69:0/2230203154 conn(0x7fa8380fce30 msgr2=0x7fa8380ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- 172.21.15.69:0/2230203154 shutdown_connections 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- 172.21.15.69:0/2230203154 wait complete. 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 Processor -- start 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.100+0000 7fa83cfee640 1 -- start start 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8381018a0 0x7fa838197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 0x7fa838198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa83810d540 con 0x7fa838103450 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa83810d3c0 con 0x7fa838102a90 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa83810d6c0 con 0x7fa8381018a0 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45012/0 (socket says 172.21.15.69:45012) 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 -- 172.21.15.69:0/3160523291 learned_addr learned my addr 172.21.15.69:0/3160523291 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83659c640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8381018a0 0x7fa838197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa835d9b640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 0x7fa838198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 -- 172.21.15.69:0/3160523291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8381018a0 msgr2=0x7fa838197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8381018a0 0x7fa838197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 -- 172.21.15.69:0/3160523291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 msgr2=0x7fa838198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 0x7fa838198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 -- 172.21.15.69:0/3160523291 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa83819f770 con 0x7fa838103450 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa835d9b640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 0x7fa838198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa836d9d640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c970 tx=0x7fa82800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828007bb0 con 0x7fa838103450 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa83819fa60 con 0x7fa838103450 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa828007d50 con 0x7fa838103450 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa828018800 con 0x7fa838103450 2024-05-09T09:16:55.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.101+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa83819ff60 con 0x7fa838103450 2024-05-09T09:16:55.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.103+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa838108b60 con 0x7fa838103450 2024-05-09T09:16:55.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.104+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 16) ==== 137833+0+0 (secure 0 0 0) 0x7fa82800fa50 con 0x7fa838103450 2024-05-09T09:16:55.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.104+0000 7fa82f7fe640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 0x7fa8240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.104+0000 7fa83659c640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 0x7fa8240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.104+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa8280cf370 con 0x7fa838103450 2024-05-09T09:16:55.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.105+0000 7fa83659c640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 0x7fa8240a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa830006010 tx=0x7fa830031bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.109+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa828014030 con 0x7fa838103450 2024-05-09T09:16:55.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.221+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa828098000 con 0x7fa838103450 2024-05-09T09:16:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:55 smithi069 ceph-mon[30123]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:55 smithi069 ceph-mon[30123]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.329+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fa83802d1c0 con 0x7fa838103450 2024-05-09T09:16:55.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.330+0000 7fa82f7fe640 1 -- 172.21.15.69:0/3160523291 <== mon.0 v2:172.21.15.69:3300/0 8 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7fa82809c090 con 0x7fa838103450 2024-05-09T09:16:55.329 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:55.330 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:16:55.330 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 msgr2=0x7fa8240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 0x7fa8240a5050 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa830006010 tx=0x7fa830031bd0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 msgr2=0x7fa83819f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa82800c970 tx=0x7fa82800ce30 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 shutdown_connections 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa8240a2b90 0x7fa8240a5050 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa838103450 0x7fa83819f100 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa838102a90 0x7fa838198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 --2- 172.21.15.69:0/3160523291 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa8381018a0 0x7fa838197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 >> 172.21.15.69:0/3160523291 conn(0x7fa8380fce30 msgr2=0x7fa838105d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 shutdown_connections 2024-05-09T09:16:55.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.332+0000 7fa83cfee640 1 -- 172.21.15.69:0/3160523291 wait complete. 2024-05-09T09:16:55.343 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:16:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 -- 172.21.15.69:0/4232124982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4101970 msgr2=0x7fe1e4101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 --2- 172.21.15.69:0/4232124982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4101970 0x7fe1e4101df0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fe1d8009900 tx=0x7fe1d802a410 comp rx=0 tx=0).stop 2024-05-09T09:16:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 -- 172.21.15.69:0/4232124982 shutdown_connections 2024-05-09T09:16:55.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 --2- 172.21.15.69:0/4232124982 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4102330 0x7fe1e4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 --2- 172.21.15.69:0/4232124982 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4101970 0x7fe1e4101df0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 --2- 172.21.15.69:0/4232124982 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4100780 0x7fe1e4100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 -- 172.21.15.69:0/4232124982 >> 172.21.15.69:0/4232124982 conn(0x7fe1e40fbd30 msgr2=0x7fe1e40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 -- 172.21.15.69:0/4232124982 shutdown_connections 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.444+0000 7fe1e928c640 1 -- 172.21.15.69:0/4232124982 wait complete. 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 Processor -- start 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 -- start start 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4100780 0x7fe1e4196af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4101970 0x7fe1e4197030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e410c470 con 0x7fe1e4102330 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e410c2f0 con 0x7fe1e4100780 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e928c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe1e410c5f0 con 0x7fe1e4101970 2024-05-09T09:16:55.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45038/0 (socket says 172.21.15.69:45038) 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 -- 172.21.15.69:0/3326273126 learned_addr learned my addr 172.21.15.69:0/3326273126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e2ffd640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4100780 0x7fe1e4196af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e27fc640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4101970 0x7fe1e4197030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 -- 172.21.15.69:0/3326273126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4101970 msgr2=0x7fe1e4197030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4101970 0x7fe1e4197030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 -- 172.21.15.69:0/3326273126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4100780 msgr2=0x7fe1e4196af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4100780 0x7fe1e4196af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.445+0000 7fe1e37fe640 1 -- 172.21.15.69:0/3326273126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe1e419e710 con 0x7fe1e4102330 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1e37fe640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00ebb0 tx=0x7fe1dc009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc00e2e0 con 0x7fe1e4102330 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe1dc007950 con 0x7fe1e4102330 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe1e4069ed0 con 0x7fe1e4102330 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe1dc005790 con 0x7fe1e4102330 2024-05-09T09:16:55.448 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.446+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe1e406a450 con 0x7fe1e4102330 2024-05-09T09:16:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.447+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe1e4068aa0 con 0x7fe1e4102330 2024-05-09T09:16:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.448+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fe1dc007b00 con 0x7fe1e4102330 2024-05-09T09:16:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.448+0000 7fe1cbfff640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 0x7fe1cc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.448+0000 7fe1e2ffd640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 0x7fe1cc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.448+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fe1dc0d0a10 con 0x7fe1e4102330 2024-05-09T09:16:55.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.449+0000 7fe1e2ffd640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 0x7fe1cc0a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1e41017d0 tx=0x7fe1d4009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.453+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe1dc09d5d0 con 0x7fe1e4102330 2024-05-09T09:16:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:55 smithi163 ceph-mon[34354]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:55 smithi163 ceph-mon[34354]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:55 smithi103 ceph-mon[34421]: from='client.24707 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:55 smithi103 ceph-mon[34421]: from='client.14895 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:55.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.629+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fe1e4107680 con 0x7fe1e4102330 2024-05-09T09:16:55.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.630+0000 7fe1cbfff640 1 -- 172.21.15.69:0/3326273126 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7fe1dc09ced0 con 0x7fe1e4102330 2024-05-09T09:16:55.631 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:55.631 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 msgr2=0x7fe1cc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 0x7fe1cc0a50e0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fe1e41017d0 tx=0x7fe1d4009110 comp rx=0 tx=0).stop 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 msgr2=0x7fe1e419e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fe1dc00ebb0 tx=0x7fe1dc009f60 comp rx=0 tx=0).stop 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 shutdown_connections 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe1cc0a2c20 0x7fe1cc0a50e0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe1e4102330 0x7fe1e419e0a0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe1e4101970 0x7fe1e4197030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.632+0000 7fe1e928c640 1 --2- 172.21.15.69:0/3326273126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe1e4100780 0x7fe1e4196af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.633+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 >> 172.21.15.69:0/3326273126 conn(0x7fe1e40fbd30 msgr2=0x7fe1e4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.633+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 shutdown_connections 2024-05-09T09:16:55.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.633+0000 7fe1e928c640 1 -- 172.21.15.69:0/3326273126 wait complete. 2024-05-09T09:16:55.644 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-05-09T09:16:55.645 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T09:16:55.749 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/611008302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74105480 msgr2=0x7f1b74105900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/611008302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74105480 0x7f1b74105900 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f1b64009ca0 tx=0x7f1b6402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/611008302 shutdown_connections 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/611008302 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74105480 0x7f1b74105900 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/611008302 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74102ad0 0x7f1b74104f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/611008302 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74100160 0x7f1b74102590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/611008302 >> 172.21.15.69:0/611008302 conn(0x7f1b740fbd80 msgr2=0x7f1b740fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/611008302 shutdown_connections 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/611008302 wait complete. 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 Processor -- start 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.746+0000 7f1b7ba9a640 1 -- start start 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74100160 0x7f1b7419e180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 0x7f1b741a56c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113af0 con 0x7f1b74102ad0 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113970 con 0x7f1b74100160 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1b74113c70 con 0x7f1b74105480 2024-05-09T09:16:55.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45052/0 (socket says 172.21.15.69:45052) 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 -- 172.21.15.69:0/2504967326 learned_addr learned my addr 172.21.15.69:0/2504967326 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79836640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74100160 0x7f1b7419e180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7a037640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 0x7f1b741a56c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 -- 172.21.15.69:0/2504967326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 msgr2=0x7f1b741a56c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 0x7f1b741a56c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 -- 172.21.15.69:0/2504967326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74100160 msgr2=0x7f1b7419e180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74100160 0x7f1b7419e180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 -- 172.21.15.69:0/2504967326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1b741a5d30 con 0x7f1b74102ad0 2024-05-09T09:16:55.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7a037640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 0x7f1b741a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b79035640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800d950 tx=0x7f1b6800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b68017030 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1b68004020 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1b741a5fc0 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1b680047a0 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.747+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1b741a64e0 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.748+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1b740fd870 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.749+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f1b680041d0 con 0x7f1b74102ad0 2024-05-09T09:16:55.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.750+0000 7f1b72ffd640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 0x7f1b5c0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:55.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.750+0000 7f1b79836640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 0x7f1b5c0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:55.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.750+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f1b680cee30 con 0x7f1b74102ad0 2024-05-09T09:16:55.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.750+0000 7f1b79836640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 0x7f1b5c0a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c005fd0 tx=0x7f1b6c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:55.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.754+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1b6809b9f0 con 0x7f1b74102ad0 2024-05-09T09:16:55.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.928+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1b74061c70 con 0x7f1b5c0a2c20 2024-05-09T09:16:55.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.938+0000 7f1b72ffd640 1 -- 172.21.15.69:0/2504967326 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340564 (secure 0 0 0) 0x7f1b74061c70 con 0x7f1b5c0a2c20 2024-05-09T09:16:55.943 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:55.945 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.941+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 msgr2=0x7f1b5c0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.941+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 0x7f1b5c0a50e0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f1b6c005fd0 tx=0x7f1b6c00b390 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.941+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 msgr2=0x7f1b7419e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.941+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f1b6800d950 tx=0x7f1b6800de10 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 shutdown_connections 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1b5c0a2c20 0x7f1b5c0a50e0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1b74105480 0x7f1b741a56c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1b74102ad0 0x7f1b7419e6c0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 --2- 172.21.15.69:0/2504967326 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1b74100160 0x7f1b7419e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:55.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 >> 172.21.15.69:0/2504967326 conn(0x7f1b740fbd80 msgr2=0x7f1b74100b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:55.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 shutdown_connections 2024-05-09T09:16:55.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:55.942+0000 7f1b7ba9a640 1 -- 172.21.15.69:0/2504967326 wait complete. 2024-05-09T09:16:55.953 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-05-09T09:16:55.106282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596526,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":282,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352428,"kb_used_data":11584,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589396,"statfs":{"total":1151940427776,"available":1151579541504,"internally_reserved":0,"allocated":11862016,"data_stored":7515630,"data_compressed":283197,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3748,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":56,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003075"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101841+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:16:38.101841+0000","last_peered":"2024-05-09T09:16:38.101841+0000","last_clean":"2024-05-09T09:16:38.101841+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:16:38.101841+0000","last_undegraded":"2024-05-09T09:16:38.101841+0000","last_fullsized":"2024-05-09T09:16:38.101841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102447+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:16:38.102447+0000","last_peered":"2024-05-09T09:16:38.102447+0000","last_clean":"2024-05-09T09:16:38.102447+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:16:38.102447+0000","last_undegraded":"2024-05-09T09:16:38.102447+0000","last_fullsized":"2024-05-09T09:16:38.102447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101929+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:16:38.101929+0000","last_peered":"2024-05-09T09:16:38.101929+0000","last_clean":"2024-05-09T09:16:38.101929+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:16:38.101929+0000","last_undegraded":"2024-05-09T09:16:38.101929+0000","last_fullsized":"2024-05-09T09:16:38.101929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102930+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:16:38.102930+0000","last_peered":"2024-05-09T09:16:38.102930+0000","last_clean":"2024-05-09T09:16:38.102930+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:16:38.102930+0000","last_undegraded":"2024-05-09T09:16:38.102930+0000","last_fullsized":"2024-05-09T09:16:38.102930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101564+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:16:38.101564+0000","last_peered":"2024-05-09T09:16:38.101564+0000","last_clean":"2024-05-09T09:16:38.101564+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:16:38.101564+0000","last_undegraded":"2024-05-09T09:16:38.101564+0000","last_fullsized":"2024-05-09T09:16:38.101564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906118+0000","last_change":"2024-05-09T09:16:32.882462+0000","last_active":"2024-05-09T09:16:37.906118+0000","last_peered":"2024-05-09T09:16:37.906118+0000","last_clean":"2024-05-09T09:16:37.906118+0000","last_became_active":"2024-05-09T09:16:32.881548+0000","last_became_peered":"2024-05-09T09:16:32.881548+0000","last_unstale":"2024-05-09T09:16:37.906118+0000","last_undegraded":"2024-05-09T09:16:37.906118+0000","last_fullsized":"2024-05-09T09:16:37.906118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:53.804496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101436+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:16:38.101436+0000","last_peered":"2024-05-09T09:16:38.101436+0000","last_clean":"2024-05-09T09:16:38.101436+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:16:38.101436+0000","last_undegraded":"2024-05-09T09:16:38.101436+0000","last_fullsized":"2024-05-09T09:16:38.101436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100212+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:16:38.100212+0000","last_peered":"2024-05-09T09:16:38.100212+0000","last_clean":"2024-05-09T09:16:38.100212+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:16:38.100212+0000","last_undegraded":"2024-05-09T09:16:38.100212+0000","last_fullsized":"2024-05-09T09:16:38.100212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102322+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:16:38.102322+0000","last_peered":"2024-05-09T09:16:38.102322+0000","last_clean":"2024-05-09T09:16:38.102322+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:16:38.102322+0000","last_undegraded":"2024-05-09T09:16:38.102322+0000","last_fullsized":"2024-05-09T09:16:38.102322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906477+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:16:37.906477+0000","last_peered":"2024-05-09T09:16:37.906477+0000","last_clean":"2024-05-09T09:16:37.906477+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:16:37.906477+0000","last_undegraded":"2024-05-09T09:16:37.906477+0000","last_fullsized":"2024-05-09T09:16:37.906477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102172+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:16:38.102172+0000","last_peered":"2024-05-09T09:16:38.102172+0000","last_clean":"2024-05-09T09:16:38.102172+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:16:38.102172+0000","last_undegraded":"2024-05-09T09:16:38.102172+0000","last_fullsized":"2024-05-09T09:16:38.102172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102878+0000","last_change":"2024-05-09T09:16:32.890715+0000","last_active":"2024-05-09T09:16:38.102878+0000","last_peered":"2024-05-09T09:16:38.102878+0000","last_clean":"2024-05-09T09:16:38.102878+0000","last_became_active":"2024-05-09T09:16:32.890371+0000","last_became_peered":"2024-05-09T09:16:32.890371+0000","last_unstale":"2024-05-09T09:16:38.102878+0000","last_undegraded":"2024-05-09T09:16:38.102878+0000","last_fullsized":"2024-05-09T09:16:38.102878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:07.491933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101056+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:16:38.101056+0000","last_peered":"2024-05-09T09:16:38.101056+0000","last_clean":"2024-05-09T09:16:38.101056+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:16:38.101056+0000","last_undegraded":"2024-05-09T09:16:38.101056+0000","last_fullsized":"2024-05-09T09:16:38.101056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101545+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:16:38.101545+0000","last_peered":"2024-05-09T09:16:38.101545+0000","last_clean":"2024-05-09T09:16:38.101545+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:16:38.101545+0000","last_undegraded":"2024-05-09T09:16:38.101545+0000","last_fullsized":"2024-05-09T09:16:38.101545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101426+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:16:38.101426+0000","last_peered":"2024-05-09T09:16:38.101426+0000","last_clean":"2024-05-09T09:16:38.101426+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:16:38.101426+0000","last_undegraded":"2024-05-09T09:16:38.101426+0000","last_fullsized":"2024-05-09T09:16:38.101426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101173+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:16:38.101173+0000","last_peered":"2024-05-09T09:16:38.101173+0000","last_clean":"2024-05-09T09:16:38.101173+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:16:38.101173+0000","last_undegraded":"2024-05-09T09:16:38.101173+0000","last_fullsized":"2024-05-09T09:16:38.101173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101493+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:16:38.101493+0000","last_peered":"2024-05-09T09:16:38.101493+0000","last_clean":"2024-05-09T09:16:38.101493+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:16:38.101493+0000","last_undegraded":"2024-05-09T09:16:38.101493+0000","last_fullsized":"2024-05-09T09:16:38.101493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905846+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:16:37.905846+0000","last_peered":"2024-05-09T09:16:37.905846+0000","last_clean":"2024-05-09T09:16:37.905846+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:16:37.905846+0000","last_undegraded":"2024-05-09T09:16:37.905846+0000","last_fullsized":"2024-05-09T09:16:37.905846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906686+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:16:37.906686+0000","last_peered":"2024-05-09T09:16:37.906686+0000","last_clean":"2024-05-09T09:16:37.906686+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:16:37.906686+0000","last_undegraded":"2024-05-09T09:16:37.906686+0000","last_fullsized":"2024-05-09T09:16:37.906686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906213+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:16:37.906213+0000","last_peered":"2024-05-09T09:16:37.906213+0000","last_clean":"2024-05-09T09:16:37.906213+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:16:37.906213+0000","last_undegraded":"2024-05-09T09:16:37.906213+0000","last_fullsized":"2024-05-09T09:16:37.906213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905879+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:16:37.905879+0000","last_peered":"2024-05-09T09:16:37.905879+0000","last_clean":"2024-05-09T09:16:37.905879+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:16:37.905879+0000","last_undegraded":"2024-05-09T09:16:37.905879+0000","last_fullsized":"2024-05-09T09:16:37.905879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101481+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:16:38.101481+0000","last_peered":"2024-05-09T09:16:38.101481+0000","last_clean":"2024-05-09T09:16:38.101481+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:16:38.101481+0000","last_undegraded":"2024-05-09T09:16:38.101481+0000","last_fullsized":"2024-05-09T09:16:38.101481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101176+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:16:38.101176+0000","last_peered":"2024-05-09T09:16:38.101176+0000","last_clean":"2024-05-09T09:16:38.101176+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:16:38.101176+0000","last_undegraded":"2024-05-09T09:16:38.101176+0000","last_fullsized":"2024-05-09T09:16:38.101176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101771+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:16:38.101771+0000","last_peered":"2024-05-09T09:16:38.101771+0000","last_clean":"2024-05-09T09:16:38.101771+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:16:38.101771+0000","last_undegraded":"2024-05-09T09:16:38.101771+0000","last_fullsized":"2024-05-09T09:16:38.101771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101649+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:16:38.101649+0000","last_peered":"2024-05-09T09:16:38.101649+0000","last_clean":"2024-05-09T09:16:38.101649+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:16:38.101649+0000","last_undegraded":"2024-05-09T09:16:38.101649+0000","last_fullsized":"2024-05-09T09:16:38.101649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102629+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:16:38.102629+0000","last_peered":"2024-05-09T09:16:38.102629+0000","last_clean":"2024-05-09T09:16:38.102629+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:16:38.102629+0000","last_undegraded":"2024-05-09T09:16:38.102629+0000","last_fullsized":"2024-05-09T09:16:38.102629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.204185+0000","last_change":"2024-05-09T09:16:31.875945+0000","last_active":"2024-05-09T09:16:52.204185+0000","last_peered":"2024-05-09T09:16:52.204185+0000","last_clean":"2024-05-09T09:16:52.204185+0000","last_became_active":"2024-05-09T09:16:31.875632+0000","last_became_peered":"2024-05-09T09:16:31.875632+0000","last_unstale":"2024-05-09T09:16:52.204185+0000","last_undegraded":"2024-05-09T09:16:52.204185+0000","last_fullsized":"2024-05-09T09:16:52.204185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:55:47.714661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101740+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:16:38.101740+0000","last_peered":"2024-05-09T09:16:38.101740+0000","last_clean":"2024-05-09T09:16:38.101740+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:16:38.101740+0000","last_undegraded":"2024-05-09T09:16:38.101740+0000","last_fullsized":"2024-05-09T09:16:38.101740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102442+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:16:38.102442+0000","last_peered":"2024-05-09T09:16:38.102442+0000","last_clean":"2024-05-09T09:16:38.102442+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:16:38.102442+0000","last_undegraded":"2024-05-09T09:16:38.102442+0000","last_fullsized":"2024-05-09T09:16:38.102442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102207+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:16:38.102207+0000","last_peered":"2024-05-09T09:16:38.102207+0000","last_clean":"2024-05-09T09:16:38.102207+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:16:38.102207+0000","last_undegraded":"2024-05-09T09:16:38.102207+0000","last_fullsized":"2024-05-09T09:16:38.102207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102274+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:16:38.102274+0000","last_peered":"2024-05-09T09:16:38.102274+0000","last_clean":"2024-05-09T09:16:38.102274+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:16:38.102274+0000","last_undegraded":"2024-05-09T09:16:38.102274+0000","last_fullsized":"2024-05-09T09:16:38.102274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906464+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:16:37.906464+0000","last_peered":"2024-05-09T09:16:37.906464+0000","last_clean":"2024-05-09T09:16:37.906464+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:16:37.906464+0000","last_undegraded":"2024-05-09T09:16:37.906464+0000","last_fullsized":"2024-05-09T09:16:37.906464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200701+0000","last_change":"2024-05-09T09:16:31.872156+0000","last_active":"2024-05-09T09:16:51.200701+0000","last_peered":"2024-05-09T09:16:51.200701+0000","last_clean":"2024-05-09T09:16:51.200701+0000","last_became_active":"2024-05-09T09:16:31.871847+0000","last_became_peered":"2024-05-09T09:16:31.871847+0000","last_unstale":"2024-05-09T09:16:51.200701+0000","last_undegraded":"2024-05-09T09:16:51.200701+0000","last_fullsized":"2024-05-09T09:16:51.200701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:47.201774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102668+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:16:38.102668+0000","last_peered":"2024-05-09T09:16:38.102668+0000","last_clean":"2024-05-09T09:16:38.102668+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:16:38.102668+0000","last_undegraded":"2024-05-09T09:16:38.102668+0000","last_fullsized":"2024-05-09T09:16:38.102668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906279+0000","last_change":"2024-05-09T09:16:36.913312+0000","last_active":"2024-05-09T09:16:37.906279+0000","last_peered":"2024-05-09T09:16:37.906279+0000","last_clean":"2024-05-09T09:16:37.906279+0000","last_became_active":"2024-05-09T09:16:36.912908+0000","last_became_peered":"2024-05-09T09:16:36.912908+0000","last_unstale":"2024-05-09T09:16:37.906279+0000","last_undegraded":"2024-05-09T09:16:37.906279+0000","last_fullsized":"2024-05-09T09:16:37.906279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:59.214206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905984+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:16:37.905984+0000","last_peered":"2024-05-09T09:16:37.905984+0000","last_clean":"2024-05-09T09:16:37.905984+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:16:37.905984+0000","last_undegraded":"2024-05-09T09:16:37.905984+0000","last_fullsized":"2024-05-09T09:16:37.905984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102584+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:16:38.102584+0000","last_peered":"2024-05-09T09:16:38.102584+0000","last_clean":"2024-05-09T09:16:38.102584+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:16:38.102584+0000","last_undegraded":"2024-05-09T09:16:38.102584+0000","last_fullsized":"2024-05-09T09:16:38.102584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101426+0000","last_change":"2024-05-09T09:16:36.908433+0000","last_active":"2024-05-09T09:16:38.101426+0000","last_peered":"2024-05-09T09:16:38.101426+0000","last_clean":"2024-05-09T09:16:38.101426+0000","last_became_active":"2024-05-09T09:16:36.907956+0000","last_became_peered":"2024-05-09T09:16:36.907956+0000","last_unstale":"2024-05-09T09:16:38.101426+0000","last_undegraded":"2024-05-09T09:16:38.101426+0000","last_fullsized":"2024-05-09T09:16:38.101426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:00.963741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.203922+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:16:52.203922+0000","last_peered":"2024-05-09T09:16:52.203922+0000","last_clean":"2024-05-09T09:16:52.203922+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:16:52.203922+0000","last_undegraded":"2024-05-09T09:16:52.203922+0000","last_fullsized":"2024-05-09T09:16:52.203922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100547+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:16:38.100547+0000","last_peered":"2024-05-09T09:16:38.100547+0000","last_clean":"2024-05-09T09:16:38.100547+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:16:38.100547+0000","last_undegraded":"2024-05-09T09:16:38.100547+0000","last_fullsized":"2024-05-09T09:16:38.100547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102391+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:16:38.102391+0000","last_peered":"2024-05-09T09:16:38.102391+0000","last_clean":"2024-05-09T09:16:38.102391+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:16:38.102391+0000","last_undegraded":"2024-05-09T09:16:38.102391+0000","last_fullsized":"2024-05-09T09:16:38.102391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101772+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:16:38.101772+0000","last_peered":"2024-05-09T09:16:38.101772+0000","last_clean":"2024-05-09T09:16:38.101772+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:16:38.101772+0000","last_undegraded":"2024-05-09T09:16:38.101772+0000","last_fullsized":"2024-05-09T09:16:38.101772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905150+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:16:37.905150+0000","last_peered":"2024-05-09T09:16:37.905150+0000","last_clean":"2024-05-09T09:16:37.905150+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:16:37.905150+0000","last_undegraded":"2024-05-09T09:16:37.905150+0000","last_fullsized":"2024-05-09T09:16:37.905150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100885+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:16:38.100885+0000","last_peered":"2024-05-09T09:16:38.100885+0000","last_clean":"2024-05-09T09:16:38.100885+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:16:38.100885+0000","last_undegraded":"2024-05-09T09:16:38.100885+0000","last_fullsized":"2024-05-09T09:16:38.100885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.198669+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:16:51.198669+0000","last_peered":"2024-05-09T09:16:51.198669+0000","last_clean":"2024-05-09T09:16:51.198669+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:16:51.198669+0000","last_undegraded":"2024-05-09T09:16:51.198669+0000","last_fullsized":"2024-05-09T09:16:51.198669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102282+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:16:38.102282+0000","last_peered":"2024-05-09T09:16:38.102282+0000","last_clean":"2024-05-09T09:16:38.102282+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:16:38.102282+0000","last_undegraded":"2024-05-09T09:16:38.102282+0000","last_fullsized":"2024-05-09T09:16:38.102282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905121+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:16:37.905121+0000","last_peered":"2024-05-09T09:16:37.905121+0000","last_clean":"2024-05-09T09:16:37.905121+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:16:37.905121+0000","last_undegraded":"2024-05-09T09:16:37.905121+0000","last_fullsized":"2024-05-09T09:16:37.905121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199251+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:16:51.199251+0000","last_peered":"2024-05-09T09:16:51.199251+0000","last_clean":"2024-05-09T09:16:51.199251+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:16:51.199251+0000","last_undegraded":"2024-05-09T09:16:51.199251+0000","last_fullsized":"2024-05-09T09:16:51.199251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101651+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:16:38.101651+0000","last_peered":"2024-05-09T09:16:38.101651+0000","last_clean":"2024-05-09T09:16:38.101651+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:16:38.101651+0000","last_undegraded":"2024-05-09T09:16:38.101651+0000","last_fullsized":"2024-05-09T09:16:38.101651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102765+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:16:38.102765+0000","last_peered":"2024-05-09T09:16:38.102765+0000","last_clean":"2024-05-09T09:16:38.102765+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:16:38.102765+0000","last_undegraded":"2024-05-09T09:16:38.102765+0000","last_fullsized":"2024-05-09T09:16:38.102765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102207+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:16:38.102207+0000","last_peered":"2024-05-09T09:16:38.102207+0000","last_clean":"2024-05-09T09:16:38.102207+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:16:38.102207+0000","last_undegraded":"2024-05-09T09:16:38.102207+0000","last_fullsized":"2024-05-09T09:16:38.102207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101853+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:16:38.101853+0000","last_peered":"2024-05-09T09:16:38.101853+0000","last_clean":"2024-05-09T09:16:38.101853+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:16:38.101853+0000","last_undegraded":"2024-05-09T09:16:38.101853+0000","last_fullsized":"2024-05-09T09:16:38.101853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905255+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:16:37.905255+0000","last_peered":"2024-05-09T09:16:37.905255+0000","last_clean":"2024-05-09T09:16:37.905255+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:16:37.905255+0000","last_undegraded":"2024-05-09T09:16:37.905255+0000","last_fullsized":"2024-05-09T09:16:37.905255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199929+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:16:51.199929+0000","last_peered":"2024-05-09T09:16:51.199929+0000","last_clean":"2024-05-09T09:16:51.199929+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:16:51.199929+0000","last_undegraded":"2024-05-09T09:16:51.199929+0000","last_fullsized":"2024-05-09T09:16:51.199929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101739+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:16:38.101739+0000","last_peered":"2024-05-09T09:16:38.101739+0000","last_clean":"2024-05-09T09:16:38.101739+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:16:38.101739+0000","last_undegraded":"2024-05-09T09:16:38.101739+0000","last_fullsized":"2024-05-09T09:16:38.101739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102632+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:16:38.102632+0000","last_peered":"2024-05-09T09:16:38.102632+0000","last_clean":"2024-05-09T09:16:38.102632+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:16:38.102632+0000","last_undegraded":"2024-05-09T09:16:38.102632+0000","last_fullsized":"2024-05-09T09:16:38.102632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200100+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:16:51.200100+0000","last_peered":"2024-05-09T09:16:51.200100+0000","last_clean":"2024-05-09T09:16:51.200100+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:16:51.200100+0000","last_undegraded":"2024-05-09T09:16:51.200100+0000","last_fullsized":"2024-05-09T09:16:51.200100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100794+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:16:38.100794+0000","last_peered":"2024-05-09T09:16:38.100794+0000","last_clean":"2024-05-09T09:16:38.100794+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:16:38.100794+0000","last_undegraded":"2024-05-09T09:16:38.100794+0000","last_fullsized":"2024-05-09T09:16:38.100794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101249+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:16:38.101249+0000","last_peered":"2024-05-09T09:16:38.101249+0000","last_clean":"2024-05-09T09:16:38.101249+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:16:38.101249+0000","last_undegraded":"2024-05-09T09:16:38.101249+0000","last_fullsized":"2024-05-09T09:16:38.101249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199838+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:16:51.199838+0000","last_peered":"2024-05-09T09:16:51.199838+0000","last_clean":"2024-05-09T09:16:51.199838+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:16:51.199838+0000","last_undegraded":"2024-05-09T09:16:51.199838+0000","last_fullsized":"2024-05-09T09:16:51.199838+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101575+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:16:38.101575+0000","last_peered":"2024-05-09T09:16:38.101575+0000","last_clean":"2024-05-09T09:16:38.101575+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:16:38.101575+0000","last_undegraded":"2024-05-09T09:16:38.101575+0000","last_fullsized":"2024-05-09T09:16:38.101575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905680+0000","last_change":"2024-05-09T09:16:36.916907+0000","last_active":"2024-05-09T09:16:37.905680+0000","last_peered":"2024-05-09T09:16:37.905680+0000","last_clean":"2024-05-09T09:16:37.905680+0000","last_became_active":"2024-05-09T09:16:36.912548+0000","last_became_peered":"2024-05-09T09:16:36.912548+0000","last_unstale":"2024-05-09T09:16:37.905680+0000","last_undegraded":"2024-05-09T09:16:37.905680+0000","last_fullsized":"2024-05-09T09:16:37.905680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:02:01.880012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102186+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:16:38.102186+0000","last_peered":"2024-05-09T09:16:38.102186+0000","last_clean":"2024-05-09T09:16:38.102186+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:16:38.102186+0000","last_undegraded":"2024-05-09T09:16:38.102186+0000","last_fullsized":"2024-05-09T09:16:38.102186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101221+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:16:38.101221+0000","last_peered":"2024-05-09T09:16:38.101221+0000","last_clean":"2024-05-09T09:16:38.101221+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:16:38.101221+0000","last_undegraded":"2024-05-09T09:16:38.101221+0000","last_fullsized":"2024-05-09T09:16:38.101221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101001+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:16:38.101001+0000","last_peered":"2024-05-09T09:16:38.101001+0000","last_clean":"2024-05-09T09:16:38.101001+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:16:38.101001+0000","last_undegraded":"2024-05-09T09:16:38.101001+0000","last_fullsized":"2024-05-09T09:16:38.101001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.206068+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:16:52.206068+0000","last_peered":"2024-05-09T09:16:52.206068+0000","last_clean":"2024-05-09T09:16:52.206068+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:16:52.206068+0000","last_undegraded":"2024-05-09T09:16:52.206068+0000","last_fullsized":"2024-05-09T09:16:52.206068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102047+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:16:38.102047+0000","last_peered":"2024-05-09T09:16:38.102047+0000","last_clean":"2024-05-09T09:16:38.102047+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:16:38.102047+0000","last_undegraded":"2024-05-09T09:16:38.102047+0000","last_fullsized":"2024-05-09T09:16:38.102047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101928+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:16:38.101928+0000","last_peered":"2024-05-09T09:16:38.101928+0000","last_clean":"2024-05-09T09:16:38.101928+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:16:38.101928+0000","last_undegraded":"2024-05-09T09:16:38.101928+0000","last_fullsized":"2024-05-09T09:16:38.101928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102122+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:16:38.102122+0000","last_peered":"2024-05-09T09:16:38.102122+0000","last_clean":"2024-05-09T09:16:38.102122+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:16:38.102122+0000","last_undegraded":"2024-05-09T09:16:38.102122+0000","last_fullsized":"2024-05-09T09:16:38.102122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101311+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:16:38.101311+0000","last_peered":"2024-05-09T09:16:38.101311+0000","last_clean":"2024-05-09T09:16:38.101311+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:16:38.101311+0000","last_undegraded":"2024-05-09T09:16:38.101311+0000","last_fullsized":"2024-05-09T09:16:38.101311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102682+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:16:38.102682+0000","last_peered":"2024-05-09T09:16:38.102682+0000","last_clean":"2024-05-09T09:16:38.102682+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:16:38.102682+0000","last_undegraded":"2024-05-09T09:16:38.102682+0000","last_fullsized":"2024-05-09T09:16:38.102682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905516+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:16:37.905516+0000","last_peered":"2024-05-09T09:16:37.905516+0000","last_clean":"2024-05-09T09:16:37.905516+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:16:37.905516+0000","last_undegraded":"2024-05-09T09:16:37.905516+0000","last_fullsized":"2024-05-09T09:16:37.905516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906337+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:16:37.906337+0000","last_peered":"2024-05-09T09:16:37.906337+0000","last_clean":"2024-05-09T09:16:37.906337+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:16:37.906337+0000","last_undegraded":"2024-05-09T09:16:37.906337+0000","last_fullsized":"2024-05-09T09:16:37.906337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100930+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:16:38.100930+0000","last_peered":"2024-05-09T09:16:38.100930+0000","last_clean":"2024-05-09T09:16:38.100930+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:16:38.100930+0000","last_undegraded":"2024-05-09T09:16:38.100930+0000","last_fullsized":"2024-05-09T09:16:38.100930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101429+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:16:38.101429+0000","last_peered":"2024-05-09T09:16:38.101429+0000","last_clean":"2024-05-09T09:16:38.101429+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:16:38.101429+0000","last_undegraded":"2024-05-09T09:16:38.101429+0000","last_fullsized":"2024-05-09T09:16:38.101429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905974+0000","last_change":"2024-05-09T09:16:32.884105+0000","last_active":"2024-05-09T09:16:37.905974+0000","last_peered":"2024-05-09T09:16:37.905974+0000","last_clean":"2024-05-09T09:16:37.905974+0000","last_became_active":"2024-05-09T09:16:32.883414+0000","last_became_peered":"2024-05-09T09:16:32.883414+0000","last_unstale":"2024-05-09T09:16:37.905974+0000","last_undegraded":"2024-05-09T09:16:37.905974+0000","last_fullsized":"2024-05-09T09:16:37.905974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:34:48.508149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101136+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:16:38.101136+0000","last_peered":"2024-05-09T09:16:38.101136+0000","last_clean":"2024-05-09T09:16:38.101136+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:16:38.101136+0000","last_undegraded":"2024-05-09T09:16:38.101136+0000","last_fullsized":"2024-05-09T09:16:38.101136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101917+0000","last_change":"2024-05-09T09:16:32.876410+0000","last_active":"2024-05-09T09:16:38.101917+0000","last_peered":"2024-05-09T09:16:38.101917+0000","last_clean":"2024-05-09T09:16:38.101917+0000","last_became_active":"2024-05-09T09:16:32.876092+0000","last_became_peered":"2024-05-09T09:16:32.876092+0000","last_unstale":"2024-05-09T09:16:38.101917+0000","last_undegraded":"2024-05-09T09:16:38.101917+0000","last_fullsized":"2024-05-09T09:16:38.101917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:36:04.050890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.204820+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:16:52.204820+0000","last_peered":"2024-05-09T09:16:52.204820+0000","last_clean":"2024-05-09T09:16:52.204820+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:16:52.204820+0000","last_undegraded":"2024-05-09T09:16:52.204820+0000","last_fullsized":"2024-05-09T09:16:52.204820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102053+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:16:38.102053+0000","last_peered":"2024-05-09T09:16:38.102053+0000","last_clean":"2024-05-09T09:16:38.102053+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:16:38.102053+0000","last_undegraded":"2024-05-09T09:16:38.102053+0000","last_fullsized":"2024-05-09T09:16:38.102053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101101+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:16:38.101101+0000","last_peered":"2024-05-09T09:16:38.101101+0000","last_clean":"2024-05-09T09:16:38.101101+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:16:38.101101+0000","last_undegraded":"2024-05-09T09:16:38.101101+0000","last_fullsized":"2024-05-09T09:16:38.101101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200002+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:16:51.200002+0000","last_peered":"2024-05-09T09:16:51.200002+0000","last_clean":"2024-05-09T09:16:51.200002+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:16:51.200002+0000","last_undegraded":"2024-05-09T09:16:51.200002+0000","last_fullsized":"2024-05-09T09:16:51.200002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100997+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:16:38.100997+0000","last_peered":"2024-05-09T09:16:38.100997+0000","last_clean":"2024-05-09T09:16:38.100997+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:16:38.100997+0000","last_undegraded":"2024-05-09T09:16:38.100997+0000","last_fullsized":"2024-05-09T09:16:38.100997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101634+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:16:38.101634+0000","last_peered":"2024-05-09T09:16:38.101634+0000","last_clean":"2024-05-09T09:16:38.101634+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:16:38.101634+0000","last_undegraded":"2024-05-09T09:16:38.101634+0000","last_fullsized":"2024-05-09T09:16:38.101634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100221+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:16:38.100221+0000","last_peered":"2024-05-09T09:16:38.100221+0000","last_clean":"2024-05-09T09:16:38.100221+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:16:38.100221+0000","last_undegraded":"2024-05-09T09:16:38.100221+0000","last_fullsized":"2024-05-09T09:16:38.100221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.203545+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:16:52.203545+0000","last_peered":"2024-05-09T09:16:52.203545+0000","last_clean":"2024-05-09T09:16:52.203545+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:16:52.203545+0000","last_undegraded":"2024-05-09T09:16:52.203545+0000","last_fullsized":"2024-05-09T09:16:52.203545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101209+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:16:38.101209+0000","last_peered":"2024-05-09T09:16:38.101209+0000","last_clean":"2024-05-09T09:16:38.101209+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:16:38.101209+0000","last_undegraded":"2024-05-09T09:16:38.101209+0000","last_fullsized":"2024-05-09T09:16:38.101209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199285+0000","last_change":"2024-05-09T09:16:31.878349+0000","last_active":"2024-05-09T09:16:51.199285+0000","last_peered":"2024-05-09T09:16:51.199285+0000","last_clean":"2024-05-09T09:16:51.199285+0000","last_became_active":"2024-05-09T09:16:31.878064+0000","last_became_peered":"2024-05-09T09:16:31.878064+0000","last_unstale":"2024-05-09T09:16:51.199285+0000","last_undegraded":"2024-05-09T09:16:51.199285+0000","last_fullsized":"2024-05-09T09:16:51.199285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:24:17.713948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100763+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:16:38.100763+0000","last_peered":"2024-05-09T09:16:38.100763+0000","last_clean":"2024-05-09T09:16:38.100763+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:16:38.100763+0000","last_undegraded":"2024-05-09T09:16:38.100763+0000","last_fullsized":"2024-05-09T09:16:38.100763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100427+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:16:38.100427+0000","last_peered":"2024-05-09T09:16:38.100427+0000","last_clean":"2024-05-09T09:16:38.100427+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:16:38.100427+0000","last_undegraded":"2024-05-09T09:16:38.100427+0000","last_fullsized":"2024-05-09T09:16:38.100427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905836+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:16:37.905836+0000","last_peered":"2024-05-09T09:16:37.905836+0000","last_clean":"2024-05-09T09:16:37.905836+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:16:37.905836+0000","last_undegraded":"2024-05-09T09:16:37.905836+0000","last_fullsized":"2024-05-09T09:16:37.905836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102199+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:16:38.102199+0000","last_peered":"2024-05-09T09:16:38.102199+0000","last_clean":"2024-05-09T09:16:38.102199+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:16:38.102199+0000","last_undegraded":"2024-05-09T09:16:38.102199+0000","last_fullsized":"2024-05-09T09:16:38.102199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905934+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:16:37.905934+0000","last_peered":"2024-05-09T09:16:37.905934+0000","last_clean":"2024-05-09T09:16:37.905934+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:16:37.905934+0000","last_undegraded":"2024-05-09T09:16:37.905934+0000","last_fullsized":"2024-05-09T09:16:37.905934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199086+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:16:51.199086+0000","last_peered":"2024-05-09T09:16:51.199086+0000","last_clean":"2024-05-09T09:16:51.199086+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:16:51.199086+0000","last_undegraded":"2024-05-09T09:16:51.199086+0000","last_fullsized":"2024-05-09T09:16:51.199086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102544+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:16:38.102544+0000","last_peered":"2024-05-09T09:16:38.102544+0000","last_clean":"2024-05-09T09:16:38.102544+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:16:38.102544+0000","last_undegraded":"2024-05-09T09:16:38.102544+0000","last_fullsized":"2024-05-09T09:16:38.102544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102946+0000","last_change":"2024-05-09T09:16:36.967460+0000","last_active":"2024-05-09T09:16:38.102946+0000","last_peered":"2024-05-09T09:16:38.102946+0000","last_clean":"2024-05-09T09:16:38.102946+0000","last_became_active":"2024-05-09T09:16:36.966966+0000","last_became_peered":"2024-05-09T09:16:36.966966+0000","last_unstale":"2024-05-09T09:16:38.102946+0000","last_undegraded":"2024-05-09T09:16:38.102946+0000","last_fullsized":"2024-05-09T09:16:38.102946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:14.954608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.198263+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:16:51.198263+0000","last_peered":"2024-05-09T09:16:51.198263+0000","last_clean":"2024-05-09T09:16:51.198263+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:16:51.198263+0000","last_undegraded":"2024-05-09T09:16:51.198263+0000","last_fullsized":"2024-05-09T09:16:51.198263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101425+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:16:38.101425+0000","last_peered":"2024-05-09T09:16:38.101425+0000","last_clean":"2024-05-09T09:16:38.101425+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:16:38.101425+0000","last_undegraded":"2024-05-09T09:16:38.101425+0000","last_fullsized":"2024-05-09T09:16:38.101425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905989+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:16:37.905989+0000","last_peered":"2024-05-09T09:16:37.905989+0000","last_clean":"2024-05-09T09:16:37.905989+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:16:37.905989+0000","last_undegraded":"2024-05-09T09:16:37.905989+0000","last_fullsized":"2024-05-09T09:16:37.905989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100980+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:16:38.100980+0000","last_peered":"2024-05-09T09:16:38.100980+0000","last_clean":"2024-05-09T09:16:38.100980+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:16:38.100980+0000","last_undegraded":"2024-05-09T09:16:38.100980+0000","last_fullsized":"2024-05-09T09:16:38.100980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100818+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:16:38.100818+0000","last_peered":"2024-05-09T09:16:38.100818+0000","last_clean":"2024-05-09T09:16:38.100818+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:16:38.100818+0000","last_undegraded":"2024-05-09T09:16:38.100818+0000","last_fullsized":"2024-05-09T09:16:38.100818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102793+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:16:38.102793+0000","last_peered":"2024-05-09T09:16:38.102793+0000","last_clean":"2024-05-09T09:16:38.102793+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:16:38.102793+0000","last_undegraded":"2024-05-09T09:16:38.102793+0000","last_fullsized":"2024-05-09T09:16:38.102793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906078+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:16:37.906078+0000","last_peered":"2024-05-09T09:16:37.906078+0000","last_clean":"2024-05-09T09:16:37.906078+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:16:37.906078+0000","last_undegraded":"2024-05-09T09:16:37.906078+0000","last_fullsized":"2024-05-09T09:16:37.906078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101270+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:16:38.101270+0000","last_peered":"2024-05-09T09:16:38.101270+0000","last_clean":"2024-05-09T09:16:38.101270+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:16:38.101270+0000","last_undegraded":"2024-05-09T09:16:38.101270+0000","last_fullsized":"2024-05-09T09:16:38.101270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102906+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:16:38.102906+0000","last_peered":"2024-05-09T09:16:38.102906+0000","last_clean":"2024-05-09T09:16:38.102906+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:16:38.102906+0000","last_undegraded":"2024-05-09T09:16:38.102906+0000","last_fullsized":"2024-05-09T09:16:38.102906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200268+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:16:51.200268+0000","last_peered":"2024-05-09T09:16:51.200268+0000","last_clean":"2024-05-09T09:16:51.200268+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:16:51.200268+0000","last_undegraded":"2024-05-09T09:16:51.200268+0000","last_fullsized":"2024-05-09T09:16:51.200268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101452+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:16:38.101452+0000","last_peered":"2024-05-09T09:16:38.101452+0000","last_clean":"2024-05-09T09:16:38.101452+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:16:38.101452+0000","last_undegraded":"2024-05-09T09:16:38.101452+0000","last_fullsized":"2024-05-09T09:16:38.101452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101316+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:16:38.101316+0000","last_peered":"2024-05-09T09:16:38.101316+0000","last_clean":"2024-05-09T09:16:38.101316+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:16:38.101316+0000","last_undegraded":"2024-05-09T09:16:38.101316+0000","last_fullsized":"2024-05-09T09:16:38.101316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906457+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:16:37.906457+0000","last_peered":"2024-05-09T09:16:37.906457+0000","last_clean":"2024-05-09T09:16:37.906457+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:16:37.906457+0000","last_undegraded":"2024-05-09T09:16:37.906457+0000","last_fullsized":"2024-05-09T09:16:37.906457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102344+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:16:38.102344+0000","last_peered":"2024-05-09T09:16:38.102344+0000","last_clean":"2024-05-09T09:16:38.102344+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:16:38.102344+0000","last_undegraded":"2024-05-09T09:16:38.102344+0000","last_fullsized":"2024-05-09T09:16:38.102344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102539+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:16:38.102539+0000","last_peered":"2024-05-09T09:16:38.102539+0000","last_clean":"2024-05-09T09:16:38.102539+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:16:38.102539+0000","last_undegraded":"2024-05-09T09:16:38.102539+0000","last_fullsized":"2024-05-09T09:16:38.102539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102388+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:16:38.102388+0000","last_peered":"2024-05-09T09:16:38.102388+0000","last_clean":"2024-05-09T09:16:38.102388+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:16:38.102388+0000","last_undegraded":"2024-05-09T09:16:38.102388+0000","last_fullsized":"2024-05-09T09:16:38.102388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101466+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:16:38.101466+0000","last_peered":"2024-05-09T09:16:38.101466+0000","last_clean":"2024-05-09T09:16:38.101466+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:16:38.101466+0000","last_undegraded":"2024-05-09T09:16:38.101466+0000","last_fullsized":"2024-05-09T09:16:38.101466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100906+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:16:38.100906+0000","last_peered":"2024-05-09T09:16:38.100906+0000","last_clean":"2024-05-09T09:16:38.100906+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:16:38.100906+0000","last_undegraded":"2024-05-09T09:16:38.100906+0000","last_fullsized":"2024-05-09T09:16:38.100906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100794+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:16:38.100794+0000","last_peered":"2024-05-09T09:16:38.100794+0000","last_clean":"2024-05-09T09:16:38.100794+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:16:38.100794+0000","last_undegraded":"2024-05-09T09:16:38.100794+0000","last_fullsized":"2024-05-09T09:16:38.100794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101999+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:16:38.101999+0000","last_peered":"2024-05-09T09:16:38.101999+0000","last_clean":"2024-05-09T09:16:38.101999+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:16:38.101999+0000","last_undegraded":"2024-05-09T09:16:38.101999+0000","last_fullsized":"2024-05-09T09:16:38.101999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100913+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:16:38.100913+0000","last_peered":"2024-05-09T09:16:38.100913+0000","last_clean":"2024-05-09T09:16:38.100913+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:16:38.100913+0000","last_undegraded":"2024-05-09T09:16:38.100913+0000","last_fullsized":"2024-05-09T09:16:38.100913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906452+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:16:37.906452+0000","last_peered":"2024-05-09T09:16:37.906452+0000","last_clean":"2024-05-09T09:16:37.906452+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:16:37.906452+0000","last_undegraded":"2024-05-09T09:16:37.906452+0000","last_fullsized":"2024-05-09T09:16:37.906452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905960+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:16:37.905960+0000","last_peered":"2024-05-09T09:16:37.905960+0000","last_clean":"2024-05-09T09:16:37.905960+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:16:37.905960+0000","last_undegraded":"2024-05-09T09:16:37.905960+0000","last_fullsized":"2024-05-09T09:16:37.905960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102653+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:16:38.102653+0000","last_peered":"2024-05-09T09:16:38.102653+0000","last_clean":"2024-05-09T09:16:38.102653+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:16:38.102653+0000","last_undegraded":"2024-05-09T09:16:38.102653+0000","last_fullsized":"2024-05-09T09:16:38.102653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100675+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:16:38.100675+0000","last_peered":"2024-05-09T09:16:38.100675+0000","last_clean":"2024-05-09T09:16:38.100675+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:16:38.100675+0000","last_undegraded":"2024-05-09T09:16:38.100675+0000","last_fullsized":"2024-05-09T09:16:38.100675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101284+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:16:38.101284+0000","last_peered":"2024-05-09T09:16:38.101284+0000","last_clean":"2024-05-09T09:16:38.101284+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:16:38.101284+0000","last_undegraded":"2024-05-09T09:16:38.101284+0000","last_fullsized":"2024-05-09T09:16:38.101284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906621+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:16:37.906621+0000","last_peered":"2024-05-09T09:16:37.906621+0000","last_clean":"2024-05-09T09:16:37.906621+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:16:37.906621+0000","last_undegraded":"2024-05-09T09:16:37.906621+0000","last_fullsized":"2024-05-09T09:16:37.906621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102396+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:16:38.102396+0000","last_peered":"2024-05-09T09:16:38.102396+0000","last_clean":"2024-05-09T09:16:38.102396+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:16:38.102396+0000","last_undegraded":"2024-05-09T09:16:38.102396+0000","last_fullsized":"2024-05-09T09:16:38.102396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101827+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:16:38.101827+0000","last_peered":"2024-05-09T09:16:38.101827+0000","last_clean":"2024-05-09T09:16:38.101827+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:16:38.101827+0000","last_undegraded":"2024-05-09T09:16:38.101827+0000","last_fullsized":"2024-05-09T09:16:38.101827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100989+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:16:38.100989+0000","last_peered":"2024-05-09T09:16:38.100989+0000","last_clean":"2024-05-09T09:16:38.100989+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:16:38.100989+0000","last_undegraded":"2024-05-09T09:16:38.100989+0000","last_fullsized":"2024-05-09T09:16:38.100989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102469+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:16:38.102469+0000","last_peered":"2024-05-09T09:16:38.102469+0000","last_clean":"2024-05-09T09:16:38.102469+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:16:38.102469+0000","last_undegraded":"2024-05-09T09:16:38.102469+0000","last_fullsized":"2024-05-09T09:16:38.102469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100896+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:16:38.100896+0000","last_peered":"2024-05-09T09:16:38.100896+0000","last_clean":"2024-05-09T09:16:38.100896+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:16:38.100896+0000","last_undegraded":"2024-05-09T09:16:38.100896+0000","last_fullsized":"2024-05-09T09:16:38.100896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101495+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:16:38.101495+0000","last_peered":"2024-05-09T09:16:38.101495+0000","last_clean":"2024-05-09T09:16:38.101495+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:16:38.101495+0000","last_undegraded":"2024-05-09T09:16:38.101495+0000","last_fullsized":"2024-05-09T09:16:38.101495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6158,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":101,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":20393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:16:55.961 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- 172.21.15.69:0/1541353255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 msgr2=0x7f4180100bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 --2- 172.21.15.69:0/1541353255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f4180100bd0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f4178009ca0 tx=0x7f417802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- 172.21.15.69:0/1541353255 shutdown_connections 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 --2- 172.21.15.69:0/1541353255 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4180102360 0x7f4180108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 --2- 172.21.15.69:0/1541353255 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 0x7f4180101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 --2- 172.21.15.69:0/1541353255 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f4180100bd0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- 172.21.15.69:0/1541353255 >> 172.21.15.69:0/1541353255 conn(0x7f41800fbd20 msgr2=0x7f41800fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- 172.21.15.69:0/1541353255 shutdown_connections 2024-05-09T09:16:56.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- 172.21.15.69:0/1541353255 wait complete. 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 Processor -- start 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.096+0000 7f418571b640 1 -- start start 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 0x7f418019edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4180102360 0x7f41801a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f418010c400 con 0x7f41801007b0 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f418010c280 con 0x7f4180102360 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f418010c580 con 0x7f41801019a0 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45074/0 (socket says 172.21.15.69:45074) 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 -- 172.21.15.69:0/2897873633 learned_addr learned my addr 172.21.15.69:0/2897873633 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417e7fc640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 0x7f418019edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:56.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 -- 172.21.15.69:0/2897873633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 msgr2=0x7f418019edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 0x7f418019edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 -- 172.21.15.69:0/2897873633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4180102360 msgr2=0x7f41801a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4180102360 0x7f41801a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 -- 172.21.15.69:0/2897873633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41801a1b10 con 0x7f41801007b0 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f417effd640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4178000f80 tx=0x7f417802f840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417802fdc0 con 0x7f41801007b0 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41801a1d40 con 0x7f41801007b0 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4178030aa0 con 0x7f41801007b0 2024-05-09T09:16:56.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f417803aaa0 con 0x7f41801007b0 2024-05-09T09:16:56.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.097+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41801a2220 con 0x7f41801007b0 2024-05-09T09:16:56.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.098+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4180107bc0 con 0x7f41801007b0 2024-05-09T09:16:56.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.099+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4178030c50 con 0x7f41801007b0 2024-05-09T09:16:56.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.100+0000 7f4177fff640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 0x7f416c0a5110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:56.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.100+0000 7f417e7fc640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 0x7f416c0a5110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:56.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.100+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f41780f0bc0 con 0x7f41801007b0 2024-05-09T09:16:56.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.100+0000 7f417e7fc640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 0x7f416c0a5110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f418006a440 tx=0x7f416800a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:56.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.104+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41780bd890 con 0x7f41801007b0 2024-05-09T09:16:56.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.280+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f4180061c70 con 0x7f416c0a2c50 2024-05-09T09:16:56.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.289+0000 7f4177fff640 1 -- 172.21.15.69:0/2897873633 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340564 (secure 0 0 0) 0x7f4180061c70 con 0x7f416c0a2c50 2024-05-09T09:16:56.293 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:16:56.295 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 msgr2=0x7f416c0a5110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 0x7f416c0a5110 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f418006a440 tx=0x7f416800a4c0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 msgr2=0x7f418006a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f4178000f80 tx=0x7f417802f840 comp rx=0 tx=0).stop 2024-05-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 shutdown_connections 2024-05-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f416c0a2c50 0x7f416c0a5110 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4180102360 0x7f41801a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41801019a0 0x7f418019edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 --2- 172.21.15.69:0/2897873633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41801007b0 0x7f418006a6d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 >> 172.21.15.69:0/2897873633 conn(0x7f41800fbd20 msgr2=0x7f4180104d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 shutdown_connections 2024-05-09T09:16:56.299 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:56.293+0000 7f418571b640 1 -- 172.21.15.69:0/2897873633 wait complete. 2024-05-09T09:16:56.304 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":302,"stamp":"2024-05-09T09:16:55.106282+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":596526,"num_objects":48,"num_object_clones":0,"num_object_copies":144,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":48,"num_whiteouts":0,"num_read":131,"num_read_kb":114,"num_write":282,"num_write_kb":2977,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":226,"ondisk_log_size":226,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352428,"kb_used_data":11584,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589396,"statfs":{"total":1151940427776,"available":1151579541504,"internally_reserved":0,"allocated":11862016,"data_stored":7515630,"data_compressed":283197,"data_compressed_allocated":380928,"data_compressed_original":2260992,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":3748,"num_objects":24,"num_object_clones":0,"num_object_copies":72,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":24,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":56,"num_write_kb":25,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":16,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003075"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101841+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:16:38.101841+0000","last_peered":"2024-05-09T09:16:38.101841+0000","last_clean":"2024-05-09T09:16:38.101841+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:16:38.101841+0000","last_undegraded":"2024-05-09T09:16:38.101841+0000","last_fullsized":"2024-05-09T09:16:38.101841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102447+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:16:38.102447+0000","last_peered":"2024-05-09T09:16:38.102447+0000","last_clean":"2024-05-09T09:16:38.102447+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:16:38.102447+0000","last_undegraded":"2024-05-09T09:16:38.102447+0000","last_fullsized":"2024-05-09T09:16:38.102447+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101929+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:16:38.101929+0000","last_peered":"2024-05-09T09:16:38.101929+0000","last_clean":"2024-05-09T09:16:38.101929+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:16:38.101929+0000","last_undegraded":"2024-05-09T09:16:38.101929+0000","last_fullsized":"2024-05-09T09:16:38.101929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102930+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:16:38.102930+0000","last_peered":"2024-05-09T09:16:38.102930+0000","last_clean":"2024-05-09T09:16:38.102930+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:16:38.102930+0000","last_undegraded":"2024-05-09T09:16:38.102930+0000","last_fullsized":"2024-05-09T09:16:38.102930+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101564+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:16:38.101564+0000","last_peered":"2024-05-09T09:16:38.101564+0000","last_clean":"2024-05-09T09:16:38.101564+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:16:38.101564+0000","last_undegraded":"2024-05-09T09:16:38.101564+0000","last_fullsized":"2024-05-09T09:16:38.101564+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906118+0000","last_change":"2024-05-09T09:16:32.882462+0000","last_active":"2024-05-09T09:16:37.906118+0000","last_peered":"2024-05-09T09:16:37.906118+0000","last_clean":"2024-05-09T09:16:37.906118+0000","last_became_active":"2024-05-09T09:16:32.881548+0000","last_became_peered":"2024-05-09T09:16:32.881548+0000","last_unstale":"2024-05-09T09:16:37.906118+0000","last_undegraded":"2024-05-09T09:16:37.906118+0000","last_fullsized":"2024-05-09T09:16:37.906118+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:45:53.804496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101436+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:16:38.101436+0000","last_peered":"2024-05-09T09:16:38.101436+0000","last_clean":"2024-05-09T09:16:38.101436+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:16:38.101436+0000","last_undegraded":"2024-05-09T09:16:38.101436+0000","last_fullsized":"2024-05-09T09:16:38.101436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100212+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:16:38.100212+0000","last_peered":"2024-05-09T09:16:38.100212+0000","last_clean":"2024-05-09T09:16:38.100212+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:16:38.100212+0000","last_undegraded":"2024-05-09T09:16:38.100212+0000","last_fullsized":"2024-05-09T09:16:38.100212+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102322+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:16:38.102322+0000","last_peered":"2024-05-09T09:16:38.102322+0000","last_clean":"2024-05-09T09:16:38.102322+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:16:38.102322+0000","last_undegraded":"2024-05-09T09:16:38.102322+0000","last_fullsized":"2024-05-09T09:16:38.102322+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906477+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:16:37.906477+0000","last_peered":"2024-05-09T09:16:37.906477+0000","last_clean":"2024-05-09T09:16:37.906477+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:16:37.906477+0000","last_undegraded":"2024-05-09T09:16:37.906477+0000","last_fullsized":"2024-05-09T09:16:37.906477+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102172+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:16:38.102172+0000","last_peered":"2024-05-09T09:16:38.102172+0000","last_clean":"2024-05-09T09:16:38.102172+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:16:38.102172+0000","last_undegraded":"2024-05-09T09:16:38.102172+0000","last_fullsized":"2024-05-09T09:16:38.102172+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102878+0000","last_change":"2024-05-09T09:16:32.890715+0000","last_active":"2024-05-09T09:16:38.102878+0000","last_peered":"2024-05-09T09:16:38.102878+0000","last_clean":"2024-05-09T09:16:38.102878+0000","last_became_active":"2024-05-09T09:16:32.890371+0000","last_became_peered":"2024-05-09T09:16:32.890371+0000","last_unstale":"2024-05-09T09:16:38.102878+0000","last_undegraded":"2024-05-09T09:16:38.102878+0000","last_fullsized":"2024-05-09T09:16:38.102878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:08:07.491933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101056+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:16:38.101056+0000","last_peered":"2024-05-09T09:16:38.101056+0000","last_clean":"2024-05-09T09:16:38.101056+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:16:38.101056+0000","last_undegraded":"2024-05-09T09:16:38.101056+0000","last_fullsized":"2024-05-09T09:16:38.101056+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101545+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:16:38.101545+0000","last_peered":"2024-05-09T09:16:38.101545+0000","last_clean":"2024-05-09T09:16:38.101545+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:16:38.101545+0000","last_undegraded":"2024-05-09T09:16:38.101545+0000","last_fullsized":"2024-05-09T09:16:38.101545+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101426+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:16:38.101426+0000","last_peered":"2024-05-09T09:16:38.101426+0000","last_clean":"2024-05-09T09:16:38.101426+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:16:38.101426+0000","last_undegraded":"2024-05-09T09:16:38.101426+0000","last_fullsized":"2024-05-09T09:16:38.101426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101173+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:16:38.101173+0000","last_peered":"2024-05-09T09:16:38.101173+0000","last_clean":"2024-05-09T09:16:38.101173+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:16:38.101173+0000","last_undegraded":"2024-05-09T09:16:38.101173+0000","last_fullsized":"2024-05-09T09:16:38.101173+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101493+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:16:38.101493+0000","last_peered":"2024-05-09T09:16:38.101493+0000","last_clean":"2024-05-09T09:16:38.101493+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:16:38.101493+0000","last_undegraded":"2024-05-09T09:16:38.101493+0000","last_fullsized":"2024-05-09T09:16:38.101493+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905846+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:16:37.905846+0000","last_peered":"2024-05-09T09:16:37.905846+0000","last_clean":"2024-05-09T09:16:37.905846+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:16:37.905846+0000","last_undegraded":"2024-05-09T09:16:37.905846+0000","last_fullsized":"2024-05-09T09:16:37.905846+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906686+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:16:37.906686+0000","last_peered":"2024-05-09T09:16:37.906686+0000","last_clean":"2024-05-09T09:16:37.906686+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:16:37.906686+0000","last_undegraded":"2024-05-09T09:16:37.906686+0000","last_fullsized":"2024-05-09T09:16:37.906686+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906213+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:16:37.906213+0000","last_peered":"2024-05-09T09:16:37.906213+0000","last_clean":"2024-05-09T09:16:37.906213+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:16:37.906213+0000","last_undegraded":"2024-05-09T09:16:37.906213+0000","last_fullsized":"2024-05-09T09:16:37.906213+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905879+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:16:37.905879+0000","last_peered":"2024-05-09T09:16:37.905879+0000","last_clean":"2024-05-09T09:16:37.905879+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:16:37.905879+0000","last_undegraded":"2024-05-09T09:16:37.905879+0000","last_fullsized":"2024-05-09T09:16:37.905879+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101481+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:16:38.101481+0000","last_peered":"2024-05-09T09:16:38.101481+0000","last_clean":"2024-05-09T09:16:38.101481+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:16:38.101481+0000","last_undegraded":"2024-05-09T09:16:38.101481+0000","last_fullsized":"2024-05-09T09:16:38.101481+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101176+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:16:38.101176+0000","last_peered":"2024-05-09T09:16:38.101176+0000","last_clean":"2024-05-09T09:16:38.101176+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:16:38.101176+0000","last_undegraded":"2024-05-09T09:16:38.101176+0000","last_fullsized":"2024-05-09T09:16:38.101176+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101771+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:16:38.101771+0000","last_peered":"2024-05-09T09:16:38.101771+0000","last_clean":"2024-05-09T09:16:38.101771+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:16:38.101771+0000","last_undegraded":"2024-05-09T09:16:38.101771+0000","last_fullsized":"2024-05-09T09:16:38.101771+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101649+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:16:38.101649+0000","last_peered":"2024-05-09T09:16:38.101649+0000","last_clean":"2024-05-09T09:16:38.101649+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:16:38.101649+0000","last_undegraded":"2024-05-09T09:16:38.101649+0000","last_fullsized":"2024-05-09T09:16:38.101649+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102629+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:16:38.102629+0000","last_peered":"2024-05-09T09:16:38.102629+0000","last_clean":"2024-05-09T09:16:38.102629+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:16:38.102629+0000","last_undegraded":"2024-05-09T09:16:38.102629+0000","last_fullsized":"2024-05-09T09:16:38.102629+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.204185+0000","last_change":"2024-05-09T09:16:31.875945+0000","last_active":"2024-05-09T09:16:52.204185+0000","last_peered":"2024-05-09T09:16:52.204185+0000","last_clean":"2024-05-09T09:16:52.204185+0000","last_became_active":"2024-05-09T09:16:31.875632+0000","last_became_peered":"2024-05-09T09:16:31.875632+0000","last_unstale":"2024-05-09T09:16:52.204185+0000","last_undegraded":"2024-05-09T09:16:52.204185+0000","last_fullsized":"2024-05-09T09:16:52.204185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:55:47.714661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101740+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:16:38.101740+0000","last_peered":"2024-05-09T09:16:38.101740+0000","last_clean":"2024-05-09T09:16:38.101740+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:16:38.101740+0000","last_undegraded":"2024-05-09T09:16:38.101740+0000","last_fullsized":"2024-05-09T09:16:38.101740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102442+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:16:38.102442+0000","last_peered":"2024-05-09T09:16:38.102442+0000","last_clean":"2024-05-09T09:16:38.102442+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:16:38.102442+0000","last_undegraded":"2024-05-09T09:16:38.102442+0000","last_fullsized":"2024-05-09T09:16:38.102442+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102207+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:16:38.102207+0000","last_peered":"2024-05-09T09:16:38.102207+0000","last_clean":"2024-05-09T09:16:38.102207+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:16:38.102207+0000","last_undegraded":"2024-05-09T09:16:38.102207+0000","last_fullsized":"2024-05-09T09:16:38.102207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102274+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:16:38.102274+0000","last_peered":"2024-05-09T09:16:38.102274+0000","last_clean":"2024-05-09T09:16:38.102274+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:16:38.102274+0000","last_undegraded":"2024-05-09T09:16:38.102274+0000","last_fullsized":"2024-05-09T09:16:38.102274+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906464+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:16:37.906464+0000","last_peered":"2024-05-09T09:16:37.906464+0000","last_clean":"2024-05-09T09:16:37.906464+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:16:37.906464+0000","last_undegraded":"2024-05-09T09:16:37.906464+0000","last_fullsized":"2024-05-09T09:16:37.906464+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"89'5","reported_seq":28,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200701+0000","last_change":"2024-05-09T09:16:31.872156+0000","last_active":"2024-05-09T09:16:51.200701+0000","last_peered":"2024-05-09T09:16:51.200701+0000","last_clean":"2024-05-09T09:16:51.200701+0000","last_became_active":"2024-05-09T09:16:31.871847+0000","last_became_peered":"2024-05-09T09:16:31.871847+0000","last_unstale":"2024-05-09T09:16:51.200701+0000","last_undegraded":"2024-05-09T09:16:51.200701+0000","last_fullsized":"2024-05-09T09:16:51.200701+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:37:47.201774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102668+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:16:38.102668+0000","last_peered":"2024-05-09T09:16:38.102668+0000","last_clean":"2024-05-09T09:16:38.102668+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:16:38.102668+0000","last_undegraded":"2024-05-09T09:16:38.102668+0000","last_fullsized":"2024-05-09T09:16:38.102668+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906279+0000","last_change":"2024-05-09T09:16:36.913312+0000","last_active":"2024-05-09T09:16:37.906279+0000","last_peered":"2024-05-09T09:16:37.906279+0000","last_clean":"2024-05-09T09:16:37.906279+0000","last_became_active":"2024-05-09T09:16:36.912908+0000","last_became_peered":"2024-05-09T09:16:36.912908+0000","last_unstale":"2024-05-09T09:16:37.906279+0000","last_undegraded":"2024-05-09T09:16:37.906279+0000","last_fullsized":"2024-05-09T09:16:37.906279+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:59.214206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905984+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:16:37.905984+0000","last_peered":"2024-05-09T09:16:37.905984+0000","last_clean":"2024-05-09T09:16:37.905984+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:16:37.905984+0000","last_undegraded":"2024-05-09T09:16:37.905984+0000","last_fullsized":"2024-05-09T09:16:37.905984+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102584+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:16:38.102584+0000","last_peered":"2024-05-09T09:16:38.102584+0000","last_clean":"2024-05-09T09:16:38.102584+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:16:38.102584+0000","last_undegraded":"2024-05-09T09:16:38.102584+0000","last_fullsized":"2024-05-09T09:16:38.102584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101426+0000","last_change":"2024-05-09T09:16:36.908433+0000","last_active":"2024-05-09T09:16:38.101426+0000","last_peered":"2024-05-09T09:16:38.101426+0000","last_clean":"2024-05-09T09:16:38.101426+0000","last_became_active":"2024-05-09T09:16:36.907956+0000","last_became_peered":"2024-05-09T09:16:36.907956+0000","last_unstale":"2024-05-09T09:16:38.101426+0000","last_undegraded":"2024-05-09T09:16:38.101426+0000","last_fullsized":"2024-05-09T09:16:38.101426+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:53:00.963741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.203922+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:16:52.203922+0000","last_peered":"2024-05-09T09:16:52.203922+0000","last_clean":"2024-05-09T09:16:52.203922+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:16:52.203922+0000","last_undegraded":"2024-05-09T09:16:52.203922+0000","last_fullsized":"2024-05-09T09:16:52.203922+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100547+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:16:38.100547+0000","last_peered":"2024-05-09T09:16:38.100547+0000","last_clean":"2024-05-09T09:16:38.100547+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:16:38.100547+0000","last_undegraded":"2024-05-09T09:16:38.100547+0000","last_fullsized":"2024-05-09T09:16:38.100547+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102391+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:16:38.102391+0000","last_peered":"2024-05-09T09:16:38.102391+0000","last_clean":"2024-05-09T09:16:38.102391+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:16:38.102391+0000","last_undegraded":"2024-05-09T09:16:38.102391+0000","last_fullsized":"2024-05-09T09:16:38.102391+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101772+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:16:38.101772+0000","last_peered":"2024-05-09T09:16:38.101772+0000","last_clean":"2024-05-09T09:16:38.101772+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:16:38.101772+0000","last_undegraded":"2024-05-09T09:16:38.101772+0000","last_fullsized":"2024-05-09T09:16:38.101772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905150+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:16:37.905150+0000","last_peered":"2024-05-09T09:16:37.905150+0000","last_clean":"2024-05-09T09:16:37.905150+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:16:37.905150+0000","last_undegraded":"2024-05-09T09:16:37.905150+0000","last_fullsized":"2024-05-09T09:16:37.905150+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100885+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:16:38.100885+0000","last_peered":"2024-05-09T09:16:38.100885+0000","last_clean":"2024-05-09T09:16:38.100885+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:16:38.100885+0000","last_undegraded":"2024-05-09T09:16:38.100885+0000","last_fullsized":"2024-05-09T09:16:38.100885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.198669+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:16:51.198669+0000","last_peered":"2024-05-09T09:16:51.198669+0000","last_clean":"2024-05-09T09:16:51.198669+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:16:51.198669+0000","last_undegraded":"2024-05-09T09:16:51.198669+0000","last_fullsized":"2024-05-09T09:16:51.198669+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102282+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:16:38.102282+0000","last_peered":"2024-05-09T09:16:38.102282+0000","last_clean":"2024-05-09T09:16:38.102282+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:16:38.102282+0000","last_undegraded":"2024-05-09T09:16:38.102282+0000","last_fullsized":"2024-05-09T09:16:38.102282+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905121+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:16:37.905121+0000","last_peered":"2024-05-09T09:16:37.905121+0000","last_clean":"2024-05-09T09:16:37.905121+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:16:37.905121+0000","last_undegraded":"2024-05-09T09:16:37.905121+0000","last_fullsized":"2024-05-09T09:16:37.905121+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199251+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:16:51.199251+0000","last_peered":"2024-05-09T09:16:51.199251+0000","last_clean":"2024-05-09T09:16:51.199251+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:16:51.199251+0000","last_undegraded":"2024-05-09T09:16:51.199251+0000","last_fullsized":"2024-05-09T09:16:51.199251+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101651+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:16:38.101651+0000","last_peered":"2024-05-09T09:16:38.101651+0000","last_clean":"2024-05-09T09:16:38.101651+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:16:38.101651+0000","last_undegraded":"2024-05-09T09:16:38.101651+0000","last_fullsized":"2024-05-09T09:16:38.101651+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102765+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:16:38.102765+0000","last_peered":"2024-05-09T09:16:38.102765+0000","last_clean":"2024-05-09T09:16:38.102765+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:16:38.102765+0000","last_undegraded":"2024-05-09T09:16:38.102765+0000","last_fullsized":"2024-05-09T09:16:38.102765+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102207+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:16:38.102207+0000","last_peered":"2024-05-09T09:16:38.102207+0000","last_clean":"2024-05-09T09:16:38.102207+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:16:38.102207+0000","last_undegraded":"2024-05-09T09:16:38.102207+0000","last_fullsized":"2024-05-09T09:16:38.102207+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101853+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:16:38.101853+0000","last_peered":"2024-05-09T09:16:38.101853+0000","last_clean":"2024-05-09T09:16:38.101853+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:16:38.101853+0000","last_undegraded":"2024-05-09T09:16:38.101853+0000","last_fullsized":"2024-05-09T09:16:38.101853+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905255+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:16:37.905255+0000","last_peered":"2024-05-09T09:16:37.905255+0000","last_clean":"2024-05-09T09:16:37.905255+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:16:37.905255+0000","last_undegraded":"2024-05-09T09:16:37.905255+0000","last_fullsized":"2024-05-09T09:16:37.905255+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'12","reported_seq":35,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199929+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:16:51.199929+0000","last_peered":"2024-05-09T09:16:51.199929+0000","last_clean":"2024-05-09T09:16:51.199929+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:16:51.199929+0000","last_undegraded":"2024-05-09T09:16:51.199929+0000","last_fullsized":"2024-05-09T09:16:51.199929+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101739+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:16:38.101739+0000","last_peered":"2024-05-09T09:16:38.101739+0000","last_clean":"2024-05-09T09:16:38.101739+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:16:38.101739+0000","last_undegraded":"2024-05-09T09:16:38.101739+0000","last_fullsized":"2024-05-09T09:16:38.101739+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102632+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:16:38.102632+0000","last_peered":"2024-05-09T09:16:38.102632+0000","last_clean":"2024-05-09T09:16:38.102632+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:16:38.102632+0000","last_undegraded":"2024-05-09T09:16:38.102632+0000","last_fullsized":"2024-05-09T09:16:38.102632+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200100+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:16:51.200100+0000","last_peered":"2024-05-09T09:16:51.200100+0000","last_clean":"2024-05-09T09:16:51.200100+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:16:51.200100+0000","last_undegraded":"2024-05-09T09:16:51.200100+0000","last_fullsized":"2024-05-09T09:16:51.200100+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100794+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:16:38.100794+0000","last_peered":"2024-05-09T09:16:38.100794+0000","last_clean":"2024-05-09T09:16:38.100794+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:16:38.100794+0000","last_undegraded":"2024-05-09T09:16:38.100794+0000","last_fullsized":"2024-05-09T09:16:38.100794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101249+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:16:38.101249+0000","last_peered":"2024-05-09T09:16:38.101249+0000","last_clean":"2024-05-09T09:16:38.101249+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:16:38.101249+0000","last_undegraded":"2024-05-09T09:16:38.101249+0000","last_fullsized":"2024-05-09T09:16:38.101249+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199838+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:16:51.199838+0000","last_peered":"2024-05-09T09:16:51.199838+0000","last_clean":"2024-05-09T09:16:51.199838+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:16:51.199838+0000","last_undegraded":"2024-05-09T09:16:51.199838+0000","last_fullsized":"2024-05-09T09:16:51.199838+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101575+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:16:38.101575+0000","last_peered":"2024-05-09T09:16:38.101575+0000","last_clean":"2024-05-09T09:16:38.101575+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:16:38.101575+0000","last_undegraded":"2024-05-09T09:16:38.101575+0000","last_fullsized":"2024-05-09T09:16:38.101575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905680+0000","last_change":"2024-05-09T09:16:36.916907+0000","last_active":"2024-05-09T09:16:37.905680+0000","last_peered":"2024-05-09T09:16:37.905680+0000","last_clean":"2024-05-09T09:16:37.905680+0000","last_became_active":"2024-05-09T09:16:36.912548+0000","last_became_peered":"2024-05-09T09:16:36.912548+0000","last_unstale":"2024-05-09T09:16:37.905680+0000","last_undegraded":"2024-05-09T09:16:37.905680+0000","last_fullsized":"2024-05-09T09:16:37.905680+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:02:01.880012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102186+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:16:38.102186+0000","last_peered":"2024-05-09T09:16:38.102186+0000","last_clean":"2024-05-09T09:16:38.102186+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:16:38.102186+0000","last_undegraded":"2024-05-09T09:16:38.102186+0000","last_fullsized":"2024-05-09T09:16:38.102186+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101221+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:16:38.101221+0000","last_peered":"2024-05-09T09:16:38.101221+0000","last_clean":"2024-05-09T09:16:38.101221+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:16:38.101221+0000","last_undegraded":"2024-05-09T09:16:38.101221+0000","last_fullsized":"2024-05-09T09:16:38.101221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101001+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:16:38.101001+0000","last_peered":"2024-05-09T09:16:38.101001+0000","last_clean":"2024-05-09T09:16:38.101001+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:16:38.101001+0000","last_undegraded":"2024-05-09T09:16:38.101001+0000","last_fullsized":"2024-05-09T09:16:38.101001+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":38,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.206068+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:16:52.206068+0000","last_peered":"2024-05-09T09:16:52.206068+0000","last_clean":"2024-05-09T09:16:52.206068+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:16:52.206068+0000","last_undegraded":"2024-05-09T09:16:52.206068+0000","last_fullsized":"2024-05-09T09:16:52.206068+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102047+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:16:38.102047+0000","last_peered":"2024-05-09T09:16:38.102047+0000","last_clean":"2024-05-09T09:16:38.102047+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:16:38.102047+0000","last_undegraded":"2024-05-09T09:16:38.102047+0000","last_fullsized":"2024-05-09T09:16:38.102047+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101928+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:16:38.101928+0000","last_peered":"2024-05-09T09:16:38.101928+0000","last_clean":"2024-05-09T09:16:38.101928+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:16:38.101928+0000","last_undegraded":"2024-05-09T09:16:38.101928+0000","last_fullsized":"2024-05-09T09:16:38.101928+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102122+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:16:38.102122+0000","last_peered":"2024-05-09T09:16:38.102122+0000","last_clean":"2024-05-09T09:16:38.102122+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:16:38.102122+0000","last_undegraded":"2024-05-09T09:16:38.102122+0000","last_fullsized":"2024-05-09T09:16:38.102122+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101311+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:16:38.101311+0000","last_peered":"2024-05-09T09:16:38.101311+0000","last_clean":"2024-05-09T09:16:38.101311+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:16:38.101311+0000","last_undegraded":"2024-05-09T09:16:38.101311+0000","last_fullsized":"2024-05-09T09:16:38.101311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102682+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:16:38.102682+0000","last_peered":"2024-05-09T09:16:38.102682+0000","last_clean":"2024-05-09T09:16:38.102682+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:16:38.102682+0000","last_undegraded":"2024-05-09T09:16:38.102682+0000","last_fullsized":"2024-05-09T09:16:38.102682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905516+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:16:37.905516+0000","last_peered":"2024-05-09T09:16:37.905516+0000","last_clean":"2024-05-09T09:16:37.905516+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:16:37.905516+0000","last_undegraded":"2024-05-09T09:16:37.905516+0000","last_fullsized":"2024-05-09T09:16:37.905516+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906337+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:16:37.906337+0000","last_peered":"2024-05-09T09:16:37.906337+0000","last_clean":"2024-05-09T09:16:37.906337+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:16:37.906337+0000","last_undegraded":"2024-05-09T09:16:37.906337+0000","last_fullsized":"2024-05-09T09:16:37.906337+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100930+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:16:38.100930+0000","last_peered":"2024-05-09T09:16:38.100930+0000","last_clean":"2024-05-09T09:16:38.100930+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:16:38.100930+0000","last_undegraded":"2024-05-09T09:16:38.100930+0000","last_fullsized":"2024-05-09T09:16:38.100930+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101429+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:16:38.101429+0000","last_peered":"2024-05-09T09:16:38.101429+0000","last_clean":"2024-05-09T09:16:38.101429+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:16:38.101429+0000","last_undegraded":"2024-05-09T09:16:38.101429+0000","last_fullsized":"2024-05-09T09:16:38.101429+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905974+0000","last_change":"2024-05-09T09:16:32.884105+0000","last_active":"2024-05-09T09:16:37.905974+0000","last_peered":"2024-05-09T09:16:37.905974+0000","last_clean":"2024-05-09T09:16:37.905974+0000","last_became_active":"2024-05-09T09:16:32.883414+0000","last_became_peered":"2024-05-09T09:16:32.883414+0000","last_unstale":"2024-05-09T09:16:37.905974+0000","last_undegraded":"2024-05-09T09:16:37.905974+0000","last_fullsized":"2024-05-09T09:16:37.905974+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:34:48.508149+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101136+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:16:38.101136+0000","last_peered":"2024-05-09T09:16:38.101136+0000","last_clean":"2024-05-09T09:16:38.101136+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:16:38.101136+0000","last_undegraded":"2024-05-09T09:16:38.101136+0000","last_fullsized":"2024-05-09T09:16:38.101136+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101917+0000","last_change":"2024-05-09T09:16:32.876410+0000","last_active":"2024-05-09T09:16:38.101917+0000","last_peered":"2024-05-09T09:16:38.101917+0000","last_clean":"2024-05-09T09:16:38.101917+0000","last_became_active":"2024-05-09T09:16:32.876092+0000","last_became_peered":"2024-05-09T09:16:32.876092+0000","last_unstale":"2024-05-09T09:16:38.101917+0000","last_undegraded":"2024-05-09T09:16:38.101917+0000","last_fullsized":"2024-05-09T09:16:38.101917+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:36:04.050890+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.204820+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:16:52.204820+0000","last_peered":"2024-05-09T09:16:52.204820+0000","last_clean":"2024-05-09T09:16:52.204820+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:16:52.204820+0000","last_undegraded":"2024-05-09T09:16:52.204820+0000","last_fullsized":"2024-05-09T09:16:52.204820+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102053+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:16:38.102053+0000","last_peered":"2024-05-09T09:16:38.102053+0000","last_clean":"2024-05-09T09:16:38.102053+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:16:38.102053+0000","last_undegraded":"2024-05-09T09:16:38.102053+0000","last_fullsized":"2024-05-09T09:16:38.102053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101101+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:16:38.101101+0000","last_peered":"2024-05-09T09:16:38.101101+0000","last_clean":"2024-05-09T09:16:38.101101+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:16:38.101101+0000","last_undegraded":"2024-05-09T09:16:38.101101+0000","last_fullsized":"2024-05-09T09:16:38.101101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200002+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:16:51.200002+0000","last_peered":"2024-05-09T09:16:51.200002+0000","last_clean":"2024-05-09T09:16:51.200002+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:16:51.200002+0000","last_undegraded":"2024-05-09T09:16:51.200002+0000","last_fullsized":"2024-05-09T09:16:51.200002+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":347,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100997+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:16:38.100997+0000","last_peered":"2024-05-09T09:16:38.100997+0000","last_clean":"2024-05-09T09:16:38.100997+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:16:38.100997+0000","last_undegraded":"2024-05-09T09:16:38.100997+0000","last_fullsized":"2024-05-09T09:16:38.100997+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101634+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:16:38.101634+0000","last_peered":"2024-05-09T09:16:38.101634+0000","last_clean":"2024-05-09T09:16:38.101634+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:16:38.101634+0000","last_undegraded":"2024-05-09T09:16:38.101634+0000","last_fullsized":"2024-05-09T09:16:38.101634+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100221+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:16:38.100221+0000","last_peered":"2024-05-09T09:16:38.100221+0000","last_clean":"2024-05-09T09:16:38.100221+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:16:38.100221+0000","last_undegraded":"2024-05-09T09:16:38.100221+0000","last_fullsized":"2024-05-09T09:16:38.100221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:52.203545+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:16:52.203545+0000","last_peered":"2024-05-09T09:16:52.203545+0000","last_clean":"2024-05-09T09:16:52.203545+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:16:52.203545+0000","last_undegraded":"2024-05-09T09:16:52.203545+0000","last_fullsized":"2024-05-09T09:16:52.203545+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101209+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:16:38.101209+0000","last_peered":"2024-05-09T09:16:38.101209+0000","last_clean":"2024-05-09T09:16:38.101209+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:16:38.101209+0000","last_undegraded":"2024-05-09T09:16:38.101209+0000","last_fullsized":"2024-05-09T09:16:38.101209+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199285+0000","last_change":"2024-05-09T09:16:31.878349+0000","last_active":"2024-05-09T09:16:51.199285+0000","last_peered":"2024-05-09T09:16:51.199285+0000","last_clean":"2024-05-09T09:16:51.199285+0000","last_became_active":"2024-05-09T09:16:31.878064+0000","last_became_peered":"2024-05-09T09:16:31.878064+0000","last_unstale":"2024-05-09T09:16:51.199285+0000","last_undegraded":"2024-05-09T09:16:51.199285+0000","last_fullsized":"2024-05-09T09:16:51.199285+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:24:17.713948+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100763+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:16:38.100763+0000","last_peered":"2024-05-09T09:16:38.100763+0000","last_clean":"2024-05-09T09:16:38.100763+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:16:38.100763+0000","last_undegraded":"2024-05-09T09:16:38.100763+0000","last_fullsized":"2024-05-09T09:16:38.100763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100427+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:16:38.100427+0000","last_peered":"2024-05-09T09:16:38.100427+0000","last_clean":"2024-05-09T09:16:38.100427+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:16:38.100427+0000","last_undegraded":"2024-05-09T09:16:38.100427+0000","last_fullsized":"2024-05-09T09:16:38.100427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905836+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:16:37.905836+0000","last_peered":"2024-05-09T09:16:37.905836+0000","last_clean":"2024-05-09T09:16:37.905836+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:16:37.905836+0000","last_undegraded":"2024-05-09T09:16:37.905836+0000","last_fullsized":"2024-05-09T09:16:37.905836+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102199+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:16:38.102199+0000","last_peered":"2024-05-09T09:16:38.102199+0000","last_clean":"2024-05-09T09:16:38.102199+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:16:38.102199+0000","last_undegraded":"2024-05-09T09:16:38.102199+0000","last_fullsized":"2024-05-09T09:16:38.102199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905934+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:16:37.905934+0000","last_peered":"2024-05-09T09:16:37.905934+0000","last_clean":"2024-05-09T09:16:37.905934+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:16:37.905934+0000","last_undegraded":"2024-05-09T09:16:37.905934+0000","last_fullsized":"2024-05-09T09:16:37.905934+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.199086+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:16:51.199086+0000","last_peered":"2024-05-09T09:16:51.199086+0000","last_clean":"2024-05-09T09:16:51.199086+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:16:51.199086+0000","last_undegraded":"2024-05-09T09:16:51.199086+0000","last_fullsized":"2024-05-09T09:16:51.199086+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102544+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:16:38.102544+0000","last_peered":"2024-05-09T09:16:38.102544+0000","last_clean":"2024-05-09T09:16:38.102544+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:16:38.102544+0000","last_undegraded":"2024-05-09T09:16:38.102544+0000","last_fullsized":"2024-05-09T09:16:38.102544+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102946+0000","last_change":"2024-05-09T09:16:36.967460+0000","last_active":"2024-05-09T09:16:38.102946+0000","last_peered":"2024-05-09T09:16:38.102946+0000","last_clean":"2024-05-09T09:16:38.102946+0000","last_became_active":"2024-05-09T09:16:36.966966+0000","last_became_peered":"2024-05-09T09:16:36.966966+0000","last_unstale":"2024-05-09T09:16:38.102946+0000","last_undegraded":"2024-05-09T09:16:38.102946+0000","last_fullsized":"2024-05-09T09:16:38.102946+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:14.954608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.198263+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:16:51.198263+0000","last_peered":"2024-05-09T09:16:51.198263+0000","last_clean":"2024-05-09T09:16:51.198263+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:16:51.198263+0000","last_undegraded":"2024-05-09T09:16:51.198263+0000","last_fullsized":"2024-05-09T09:16:51.198263+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101425+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:16:38.101425+0000","last_peered":"2024-05-09T09:16:38.101425+0000","last_clean":"2024-05-09T09:16:38.101425+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:16:38.101425+0000","last_undegraded":"2024-05-09T09:16:38.101425+0000","last_fullsized":"2024-05-09T09:16:38.101425+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905989+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:16:37.905989+0000","last_peered":"2024-05-09T09:16:37.905989+0000","last_clean":"2024-05-09T09:16:37.905989+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:16:37.905989+0000","last_undegraded":"2024-05-09T09:16:37.905989+0000","last_fullsized":"2024-05-09T09:16:37.905989+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100980+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:16:38.100980+0000","last_peered":"2024-05-09T09:16:38.100980+0000","last_clean":"2024-05-09T09:16:38.100980+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:16:38.100980+0000","last_undegraded":"2024-05-09T09:16:38.100980+0000","last_fullsized":"2024-05-09T09:16:38.100980+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100818+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:16:38.100818+0000","last_peered":"2024-05-09T09:16:38.100818+0000","last_clean":"2024-05-09T09:16:38.100818+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:16:38.100818+0000","last_undegraded":"2024-05-09T09:16:38.100818+0000","last_fullsized":"2024-05-09T09:16:38.100818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102793+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:16:38.102793+0000","last_peered":"2024-05-09T09:16:38.102793+0000","last_clean":"2024-05-09T09:16:38.102793+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:16:38.102793+0000","last_undegraded":"2024-05-09T09:16:38.102793+0000","last_fullsized":"2024-05-09T09:16:38.102793+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906078+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:16:37.906078+0000","last_peered":"2024-05-09T09:16:37.906078+0000","last_clean":"2024-05-09T09:16:37.906078+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:16:37.906078+0000","last_undegraded":"2024-05-09T09:16:37.906078+0000","last_fullsized":"2024-05-09T09:16:37.906078+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101270+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:16:38.101270+0000","last_peered":"2024-05-09T09:16:38.101270+0000","last_clean":"2024-05-09T09:16:38.101270+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:16:38.101270+0000","last_undegraded":"2024-05-09T09:16:38.101270+0000","last_fullsized":"2024-05-09T09:16:38.101270+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102906+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:16:38.102906+0000","last_peered":"2024-05-09T09:16:38.102906+0000","last_clean":"2024-05-09T09:16:38.102906+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:16:38.102906+0000","last_undegraded":"2024-05-09T09:16:38.102906+0000","last_fullsized":"2024-05-09T09:16:38.102906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:51.200268+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:16:51.200268+0000","last_peered":"2024-05-09T09:16:51.200268+0000","last_clean":"2024-05-09T09:16:51.200268+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:16:51.200268+0000","last_undegraded":"2024-05-09T09:16:51.200268+0000","last_fullsized":"2024-05-09T09:16:51.200268+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101452+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:16:38.101452+0000","last_peered":"2024-05-09T09:16:38.101452+0000","last_clean":"2024-05-09T09:16:38.101452+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:16:38.101452+0000","last_undegraded":"2024-05-09T09:16:38.101452+0000","last_fullsized":"2024-05-09T09:16:38.101452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101316+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:16:38.101316+0000","last_peered":"2024-05-09T09:16:38.101316+0000","last_clean":"2024-05-09T09:16:38.101316+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:16:38.101316+0000","last_undegraded":"2024-05-09T09:16:38.101316+0000","last_fullsized":"2024-05-09T09:16:38.101316+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906457+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:16:37.906457+0000","last_peered":"2024-05-09T09:16:37.906457+0000","last_clean":"2024-05-09T09:16:37.906457+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:16:37.906457+0000","last_undegraded":"2024-05-09T09:16:37.906457+0000","last_fullsized":"2024-05-09T09:16:37.906457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102344+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:16:38.102344+0000","last_peered":"2024-05-09T09:16:38.102344+0000","last_clean":"2024-05-09T09:16:38.102344+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:16:38.102344+0000","last_undegraded":"2024-05-09T09:16:38.102344+0000","last_fullsized":"2024-05-09T09:16:38.102344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102539+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:16:38.102539+0000","last_peered":"2024-05-09T09:16:38.102539+0000","last_clean":"2024-05-09T09:16:38.102539+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:16:38.102539+0000","last_undegraded":"2024-05-09T09:16:38.102539+0000","last_fullsized":"2024-05-09T09:16:38.102539+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102388+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:16:38.102388+0000","last_peered":"2024-05-09T09:16:38.102388+0000","last_clean":"2024-05-09T09:16:38.102388+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:16:38.102388+0000","last_undegraded":"2024-05-09T09:16:38.102388+0000","last_fullsized":"2024-05-09T09:16:38.102388+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101466+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:16:38.101466+0000","last_peered":"2024-05-09T09:16:38.101466+0000","last_clean":"2024-05-09T09:16:38.101466+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:16:38.101466+0000","last_undegraded":"2024-05-09T09:16:38.101466+0000","last_fullsized":"2024-05-09T09:16:38.101466+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100906+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:16:38.100906+0000","last_peered":"2024-05-09T09:16:38.100906+0000","last_clean":"2024-05-09T09:16:38.100906+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:16:38.100906+0000","last_undegraded":"2024-05-09T09:16:38.100906+0000","last_fullsized":"2024-05-09T09:16:38.100906+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100794+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:16:38.100794+0000","last_peered":"2024-05-09T09:16:38.100794+0000","last_clean":"2024-05-09T09:16:38.100794+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:16:38.100794+0000","last_undegraded":"2024-05-09T09:16:38.100794+0000","last_fullsized":"2024-05-09T09:16:38.100794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101999+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:16:38.101999+0000","last_peered":"2024-05-09T09:16:38.101999+0000","last_clean":"2024-05-09T09:16:38.101999+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:16:38.101999+0000","last_undegraded":"2024-05-09T09:16:38.101999+0000","last_fullsized":"2024-05-09T09:16:38.101999+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100913+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:16:38.100913+0000","last_peered":"2024-05-09T09:16:38.100913+0000","last_clean":"2024-05-09T09:16:38.100913+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:16:38.100913+0000","last_undegraded":"2024-05-09T09:16:38.100913+0000","last_fullsized":"2024-05-09T09:16:38.100913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906452+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:16:37.906452+0000","last_peered":"2024-05-09T09:16:37.906452+0000","last_clean":"2024-05-09T09:16:37.906452+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:16:37.906452+0000","last_undegraded":"2024-05-09T09:16:37.906452+0000","last_fullsized":"2024-05-09T09:16:37.906452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.905960+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:16:37.905960+0000","last_peered":"2024-05-09T09:16:37.905960+0000","last_clean":"2024-05-09T09:16:37.905960+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:16:37.905960+0000","last_undegraded":"2024-05-09T09:16:37.905960+0000","last_fullsized":"2024-05-09T09:16:37.905960+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102653+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:16:38.102653+0000","last_peered":"2024-05-09T09:16:38.102653+0000","last_clean":"2024-05-09T09:16:38.102653+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:16:38.102653+0000","last_undegraded":"2024-05-09T09:16:38.102653+0000","last_fullsized":"2024-05-09T09:16:38.102653+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100675+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:16:38.100675+0000","last_peered":"2024-05-09T09:16:38.100675+0000","last_clean":"2024-05-09T09:16:38.100675+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:16:38.100675+0000","last_undegraded":"2024-05-09T09:16:38.100675+0000","last_fullsized":"2024-05-09T09:16:38.100675+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101284+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:16:38.101284+0000","last_peered":"2024-05-09T09:16:38.101284+0000","last_clean":"2024-05-09T09:16:38.101284+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:16:38.101284+0000","last_undegraded":"2024-05-09T09:16:38.101284+0000","last_fullsized":"2024-05-09T09:16:38.101284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:37.906621+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:16:37.906621+0000","last_peered":"2024-05-09T09:16:37.906621+0000","last_clean":"2024-05-09T09:16:37.906621+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:16:37.906621+0000","last_undegraded":"2024-05-09T09:16:37.906621+0000","last_fullsized":"2024-05-09T09:16:37.906621+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102396+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:16:38.102396+0000","last_peered":"2024-05-09T09:16:38.102396+0000","last_clean":"2024-05-09T09:16:38.102396+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:16:38.102396+0000","last_undegraded":"2024-05-09T09:16:38.102396+0000","last_fullsized":"2024-05-09T09:16:38.102396+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101827+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:16:38.101827+0000","last_peered":"2024-05-09T09:16:38.101827+0000","last_clean":"2024-05-09T09:16:38.101827+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:16:38.101827+0000","last_undegraded":"2024-05-09T09:16:38.101827+0000","last_fullsized":"2024-05-09T09:16:38.101827+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100989+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:16:38.100989+0000","last_peered":"2024-05-09T09:16:38.100989+0000","last_clean":"2024-05-09T09:16:38.100989+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:16:38.100989+0000","last_undegraded":"2024-05-09T09:16:38.100989+0000","last_fullsized":"2024-05-09T09:16:38.100989+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.102469+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:16:38.102469+0000","last_peered":"2024-05-09T09:16:38.102469+0000","last_clean":"2024-05-09T09:16:38.102469+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:16:38.102469+0000","last_undegraded":"2024-05-09T09:16:38.102469+0000","last_fullsized":"2024-05-09T09:16:38.102469+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.100896+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:16:38.100896+0000","last_peered":"2024-05-09T09:16:38.100896+0000","last_clean":"2024-05-09T09:16:38.100896+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:16:38.100896+0000","last_undegraded":"2024-05-09T09:16:38.100896+0000","last_fullsized":"2024-05-09T09:16:38.100896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-05-09T09:16:38.101495+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:16:38.101495+0000","last_peered":"2024-05-09T09:16:38.101495+0000","last_clean":"2024-05-09T09:16:38.101495+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:16:38.101495+0000","last_undegraded":"2024-05-09T09:16:38.101495+0000","last_fullsized":"2024-05-09T09:16:38.101495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":6158,"num_objects":46,"num_object_clones":0,"num_object_copies":138,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":46,"num_whiteouts":0,"num_read":5,"num_read_kb":5,"num_write":101,"num_write_kb":39,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":30,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":192512,"data_stored":20393,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":89,"ondisk_log_size":89,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383697,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645334,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":1071619,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808859,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27964,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717188,"statfs":{"total":95995035648,"available":95966400512,"internally_reserved":0,"allocated":966656,"data_stored":477797,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070497,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":477953,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332134,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1028096,"data_stored":1069375,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495659,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32120,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713032,"statfs":{"total":95995035648,"available":95962144768,"internally_reserved":0,"allocated":966656,"data_stored":477305,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574915,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":952,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":974848,"data_stored":481057,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738441,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":991232,"data_stored":478693,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411391,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":481713,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182457,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27968,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717184,"statfs":{"total":95995035648,"available":95966396416,"internally_reserved":0,"allocated":970752,"data_stored":477319,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920821,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":477237,"data_compressed":14671,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757295,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":1067765,"data_compressed":50386,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4498,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":4798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:16:56.309 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-05-09T09:16:56.309 INFO:tasks.cephadm:Setting up client nodes... 2024-05-09T09:16:56.309 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-05-09T09:16:56.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: pgmap v302: 129 pgs: 129 active+clean; 583 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 2.1 KiB/s wr, 5 op/s 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3160523291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:56 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3326273126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: pgmap v302: 129 pgs: 129 active+clean; 583 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 2.1 KiB/s wr, 5 op/s 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3160523291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:56.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:56 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3326273126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:56.583 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: from='client.14901 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: from='client.14907 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: from='client.14913 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: pgmap v302: 129 pgs: 129 active+clean; 583 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s rd, 2.1 KiB/s wr, 5 op/s 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3160523291' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:16:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:56 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3326273126' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:16:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:57 smithi069 ceph-mon[30123]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:57 smithi163 ceph-mon[34354]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:57 smithi103 ceph-mon[34421]: from='client.14931 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:58 smithi069 ceph-mon[30123]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:58 smithi069 ceph-mon[30123]: pgmap v303: 129 pgs: 129 active+clean; 619 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-05-09T09:16:58.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 -- 172.21.15.69:0/1416325136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 msgr2=0x7f8ee8074730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 --2- 172.21.15.69:0/1416325136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee8074730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8ed8009990 tx=0x7f8ed8009ca0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 -- 172.21.15.69:0/1416325136 shutdown_connections 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 --2- 172.21.15.69:0/1416325136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee810c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 --2- 172.21.15.69:0/1416325136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee8074730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 --2- 172.21.15.69:0/1416325136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 0x7f8ee8075180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 -- 172.21.15.69:0/1416325136 >> 172.21.15.69:0/1416325136 conn(0x7f8ee80fd970 msgr2=0x7f8ee80ffdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 -- 172.21.15.69:0/1416325136 shutdown_connections 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.469+0000 7f8eed765700 1 -- 172.21.15.69:0/1416325136 wait complete. 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 Processor -- start 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 -- start start 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 0x7f8ee819a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:58.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee81a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee81101f0 con 0x7f8ee80762d0 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8110070 con 0x7f8ee8074d00 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8eed765700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8ee8110370 con 0x7f8ee810a2d0 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8edffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8edffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55318/0 (socket says 172.21.15.69:55318) 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8edffff700 1 -- 172.21.15.69:0/400661009 learned_addr learned my addr 172.21.15.69:0/400661009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.470+0000 7f8ee6ffd700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 0x7f8ee819a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8ee77fe700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee81a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 -- 172.21.15.69:0/400661009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 msgr2=0x7f8ee81a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee81a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 -- 172.21.15.69:0/400661009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 msgr2=0x7f8ee819a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:58.473 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 0x7f8ee819a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 -- 172.21.15.69:0/400661009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ee81a2310 con 0x7f8ee80762d0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8ee77fe700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee81a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8edffff700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000c9e0 tx=0x7f8ed000ccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0007ac0 con 0x7f8ee80762d0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ed0007c60 con 0x7f8ee80762d0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ee81a2600 con 0x7f8ee80762d0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ed0018730 con 0x7f8ee80762d0 2024-05-09T09:16:58.474 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.471+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ee81a2bc0 con 0x7f8ee80762d0 2024-05-09T09:16:58.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.472+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8ee8050890 con 0x7f8ee80762d0 2024-05-09T09:16:58.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.473+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f8ed00188d0 con 0x7f8ee80762d0 2024-05-09T09:16:58.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.474+0000 7f8ee4ff9700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 0x7f8ed40a5220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:16:58.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.474+0000 7f8ee6ffd700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 0x7f8ed40a5220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:16:58.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.474+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f8ed00cf3e0 con 0x7f8ee80762d0 2024-05-09T09:16:58.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.474+0000 7f8ee6ffd700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 0x7f8ed40a5220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800b5c0 tx=0x7f8ed8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:16:58.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.479+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8ed0014030 con 0x7f8ee80762d0 2024-05-09T09:16:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:58 smithi163 ceph-mon[34354]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:58 smithi163 ceph-mon[34354]: pgmap v303: 129 pgs: 129 active+clean; 619 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-05-09T09:16:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:58 smithi103 ceph-mon[34421]: from='client.14937 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:16:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:58 smithi103 ceph-mon[34421]: pgmap v303: 129 pgs: 129 active+clean; 619 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 5.8 KiB/s wr, 10 op/s 2024-05-09T09:16:58.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.755+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) -- 0x7f8ee802dbb0 con 0x7f8ee80762d0 2024-05-09T09:16:58.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.758+0000 7f8ee4ff9700 1 -- 172.21.15.69:0/400661009 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) ==== 170+0+59 (secure 0 0 0) 0x7f8ed009bfa0 con 0x7f8ee80762d0 2024-05-09T09:16:58.758 INFO:teuthology.orchestra.run.smithi069.stdout:[client.0] 2024-05-09T09:16:58.758 INFO:teuthology.orchestra.run.smithi069.stdout: key = AQCKlDxmsYAtLRAA0XdEOZ/jWwZl3ime1wJIrA== 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 msgr2=0x7f8ed40a5220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 0x7f8ed40a5220 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f8ed800b5c0 tx=0x7f8ed8005fd0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 msgr2=0x7f8ee819abf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f8ed000c9e0 tx=0x7f8ed000ccf0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 shutdown_connections 2024-05-09T09:16:58.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ee810a2d0 0x7f8ee81a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ed40a2d60 0x7f8ed40a5220 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ee8074d00 0x7f8ee819a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 --2- 172.21.15.69:0/400661009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ee80762d0 0x7f8ee819abf0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 >> 172.21.15.69:0/400661009 conn(0x7f8ee80fd970 msgr2=0x7f8ee8106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.761+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 shutdown_connections 2024-05-09T09:16:58.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:16:58.762+0000 7f8eed765700 1 -- 172.21.15.69:0/400661009 wait complete. 2024-05-09T09:16:59.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:59 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T09:16:59.452 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:16:59 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T09:16:59.479 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:16:59.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-05-09T09:16:59.479 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-05-09T09:16:59.517 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-05-09T09:16:59.517 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-05-09T09:16:59.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph mgr dump --format=json 2024-05-09T09:16:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:59 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T09:16:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:16:59 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T09:16:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:59 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-05-09T09:16:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:16:59 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/400661009' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-05-09T09:16:59.797 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:00 smithi069 ceph-mon[30123]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:00 smithi163 ceph-mon[34354]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:00.616 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:00 smithi103 ceph-mon[34421]: pgmap v304: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:01.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- 172.21.15.69:0/2891752218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 msgr2=0x7fcec010f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:01.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 --2- 172.21.15.69:0/2891752218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec010f230 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fceb800b0a0 tx=0x7fceb800b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- 172.21.15.69:0/2891752218 shutdown_connections 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 --2- 172.21.15.69:0/2891752218 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec010f230 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 --2- 172.21.15.69:0/2891752218 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 0x7fcec01029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 --2- 172.21.15.69:0/2891752218 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 0x7fcec0106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- 172.21.15.69:0/2891752218 >> 172.21.15.69:0/2891752218 conn(0x7fcec0079480 msgr2=0x7fcec0079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- 172.21.15.69:0/2891752218 shutdown_connections 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- 172.21.15.69:0/2891752218 wait complete. 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 Processor -- start 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.451+0000 7fcec5f97700 1 -- start start 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 0x7fcec0074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 0x7fcec0074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcec0112640 con 0x7fcec0107660 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcec01124c0 con 0x7fcec0102f80 2024-05-09T09:17:01.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcec01127c0 con 0x7fcec01025d0 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55336/0 (socket says 172.21.15.69:55336) 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 -- 172.21.15.69:0/1791084565 learned_addr learned my addr 172.21.15.69:0/1791084565 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebf7fe700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 0x7fcec0074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebeffd700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 0x7fcec0074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 -- 172.21.15.69:0/1791084565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 msgr2=0x7fcec0074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 0x7fcec0074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 -- 172.21.15.69:0/1791084565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 msgr2=0x7fcec0074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 0x7fcec0074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 -- 172.21.15.69:0/1791084565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcec01a7040 con 0x7fcec0107660 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebffff700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fceb80062a0 tx=0x7fceb8004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb8021070 con 0x7fcec0107660 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcec01a72d0 con 0x7fcec0107660 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fceb8007c00 con 0x7fcec0107660 2024-05-09T09:17:01.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.452+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fceb801d450 con 0x7fcec0107660 2024-05-09T09:17:01.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.453+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcec01a77d0 con 0x7fcec0107660 2024-05-09T09:17:01.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.453+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcec0069010 con 0x7fcec0107660 2024-05-09T09:17:01.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.454+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fceb8004020 con 0x7fcec0107660 2024-05-09T09:17:01.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.455+0000 7fcebcff9700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 0x7fcea80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:01.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.455+0000 7fcebf7fe700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 0x7fcea80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:01.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.455+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fceb800f020 con 0x7fcec0107660 2024-05-09T09:17:01.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.456+0000 7fcebf7fe700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 0x7fcea80a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fceb0004630 tx=0x7fceb000b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:01.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.459+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fceb802a020 con 0x7fcec0107660 2024-05-09T09:17:01.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.710+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) -- 0x7fcec002db80 con 0x7fcec0107660 2024-05-09T09:17:01.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.715+0000 7fcebcff9700 1 -- 172.21.15.69:0/1791084565 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) ==== 74+0+263201 (secure 0 0 0) 0x7fceb809d640 con 0x7fcec0107660 2024-05-09T09:17:01.717 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 msgr2=0x7fcea80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 0x7fcea80a5170 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fceb0004630 tx=0x7fceb000b3c0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 msgr2=0x7fcec00750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fceb80062a0 tx=0x7fceb8004d70 comp rx=0 tx=0).stop 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 shutdown_connections 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcec0107660 0x7fcec00750a0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcec01025d0 0x7fcec0074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcea80a2cb0 0x7fcea80a5170 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 --2- 172.21.15.69:0/1791084565 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcec0102f80 0x7fcec0074b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:01.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 >> 172.21.15.69:0/1791084565 conn(0x7fcec0079480 msgr2=0x7fcec0100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:01.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 shutdown_connections 2024-05-09T09:17:01.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:01.718+0000 7fcec5f97700 1 -- 172.21.15.69:0/1791084565 wait complete. 2024-05-09T09:17:02.454 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:02 smithi069 ceph-mon[30123]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-09T09:17:02.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:02 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1791084565' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T09:17:02.482 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6800","nonce":3550836912},{"type":"v1","addr":"172.21.15.69:6801","nonce":3550836912}]},"active_addr":"172.21.15.69:6801/3550836912","active_change":"2024-05-09T09:09:29.020203+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24109,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24113,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.1","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.69:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2046838104}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":1282800614}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":2399833747}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":3930888378}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.69:0","nonce":4056003851}]}]} 2024-05-09T09:17:02.487 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-05-09T09:17:02.487 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-05-09T09:17:02.487 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T09:17:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:02 smithi163 ceph-mon[34354]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-09T09:17:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:02 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1791084565' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T09:17:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:02 smithi103 ceph-mon[34421]: pgmap v305: 129 pgs: 129 active+clean; 620 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 6.0 KiB/s wr, 12 op/s 2024-05-09T09:17:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:02 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1791084565' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-05-09T09:17:02.728 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 -- 172.21.15.69:0/3041682612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45281083b0 msgr2=0x7f4528108790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 --2- 172.21.15.69:0/3041682612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45281083b0 0x7f4528108790 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f4518009990 tx=0x7f4518009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 -- 172.21.15.69:0/3041682612 shutdown_connections 2024-05-09T09:17:04.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 --2- 172.21.15.69:0/3041682612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45281012b0 0x7f452810f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 --2- 172.21.15.69:0/3041682612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f45281083b0 0x7f4528108790 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 --2- 172.21.15.69:0/3041682612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4528100910 0x7f4528100d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 -- 172.21.15.69:0/3041682612 >> 172.21.15.69:0/3041682612 conn(0x7f4528077890 msgr2=0x7f45280fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 -- 172.21.15.69:0/3041682612 shutdown_connections 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.464+0000 7f452e3ac700 1 -- 172.21.15.69:0/3041682612 wait complete. 2024-05-09T09:17:04.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 Processor -- start 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 -- start start 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 0x7f4528102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45281083b0 0x7f4528102570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4528112610 con 0x7f4528100910 2024-05-09T09:17:04.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4528112490 con 0x7f45281012b0 2024-05-09T09:17:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f452e3ac700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4528112790 con 0x7f45281083b0 2024-05-09T09:17:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f4527fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f4527fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55368/0 (socket says 172.21.15.69:55368) 2024-05-09T09:17:04.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f4527fff700 1 -- 172.21.15.69:0/2218069197 learned_addr learned my addr 172.21.15.69:0/2218069197 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f451ffff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 0x7f4528102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f4527fff700 1 -- 172.21.15.69:0/2218069197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45281083b0 msgr2=0x7f4528102570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.465+0000 7f4527fff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45281083b0 0x7f4528102570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4527fff700 1 -- 172.21.15.69:0/2218069197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 msgr2=0x7f4528102030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4527fff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 0x7f4528102030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4527fff700 1 -- 172.21.15.69:0/2218069197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f45281a6fd0 con 0x7f4528100910 2024-05-09T09:17:04.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f451ffff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 0x7f4528102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:04.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4527fff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f451800bb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:04.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4518027070 con 0x7f4528100910 2024-05-09T09:17:04.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f45281a7260 con 0x7f4528100910 2024-05-09T09:17:04.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4518022470 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f451800f6a0 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.466+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45281a76b0 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.467+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4528103c30 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.468+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f451802a070 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.469+0000 7f4525ffb700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 0x7f45140a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.469+0000 7f451ffff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 0x7f45140a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.469+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f45180d0460 con 0x7f4528100910 2024-05-09T09:17:04.484 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.469+0000 7f451ffff700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 0x7f45140a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4528107a90 tx=0x7f4510009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:04.485 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.473+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4518013070 con 0x7f4528100910 2024-05-09T09:17:04.485 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:04 smithi069 ceph-mon[30123]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T09:17:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:04 smithi163 ceph-mon[34354]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T09:17:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:04 smithi103 ceph-mon[34421]: pgmap v306: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 6.5 KiB/s wr, 14 op/s 2024-05-09T09:17:04.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.668+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f452802dc20 con 0x7f4528100910 2024-05-09T09:17:04.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.670+0000 7f4525ffb700 1 -- 172.21.15.69:0/2218069197 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f451809d0b0 con 0x7f4528100910 2024-05-09T09:17:04.669 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:17:04.669 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:17:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 msgr2=0x7f45140a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 0x7f45140a5170 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4528107a90 tx=0x7f4510009260 comp rx=0 tx=0).stop 2024-05-09T09:17:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 msgr2=0x7f4528101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:04.671 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f4518000c00 tx=0x7f451800bb60 comp rx=0 tx=0).stop 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 shutdown_connections 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f45281083b0 0x7f4528102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4528100910 0x7f4528101af0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45140a2cb0 0x7f45140a5170 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 --2- 172.21.15.69:0/2218069197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f45281012b0 0x7f4528102030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 >> 172.21.15.69:0/2218069197 conn(0x7f4528077890 msgr2=0x7f45280fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 shutdown_connections 2024-05-09T09:17:04.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:04.672+0000 7f452e3ac700 1 -- 172.21.15.69:0/2218069197 wait complete. 2024-05-09T09:17:05.408 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:05 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2218069197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:05.441 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-05-09T09:17:05.441 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd dump --format=json 2024-05-09T09:17:05.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:05 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2218069197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:05 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2218069197' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:05.685 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:06 smithi069 ceph-mon[30123]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-05-09T09:17:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:06 smithi163 ceph-mon[34354]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-05-09T09:17:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:06 smithi103 ceph-mon[34421]: pgmap v307: 129 pgs: 129 active+clean; 625 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 7.8 KiB/s rd, 5.5 KiB/s wr, 13 op/s 2024-05-09T09:17:07.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.442+0000 7f7d188ed700 1 -- 172.21.15.69:0/1250399964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14107660 msgr2=0x7f7d1410f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.442+0000 7f7d188ed700 1 --2- 172.21.15.69:0/1250399964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14107660 0x7f7d1410f230 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f7d0c00b0a0 tx=0x7f7d0c00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 -- 172.21.15.69:0/1250399964 shutdown_connections 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 --2- 172.21.15.69:0/1250399964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14107660 0x7f7d1410f230 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 --2- 172.21.15.69:0/1250399964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 0x7f7d141029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 --2- 172.21.15.69:0/1250399964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14102f80 0x7f7d14106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 -- 172.21.15.69:0/1250399964 >> 172.21.15.69:0/1250399964 conn(0x7f7d14079480 msgr2=0x7f7d14079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 -- 172.21.15.69:0/1250399964 shutdown_connections 2024-05-09T09:17:07.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 -- 172.21.15.69:0/1250399964 wait complete. 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 Processor -- start 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.443+0000 7f7d188ed700 1 -- start start 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 0x7f7d1419e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14107660 0x7f7d141988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7d141126a0 con 0x7f7d14102f80 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7d14112520 con 0x7f7d14107660 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7d14112820 con 0x7f7d141025d0 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:07.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55400/0 (socket says 172.21.15.69:55400) 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 -- 172.21.15.69:0/246584231 learned_addr learned my addr 172.21.15.69:0/246584231 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d1259c700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 0x7f7d1419e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d12d9d700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14107660 0x7f7d141988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 -- 172.21.15.69:0/246584231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 msgr2=0x7f7d1419e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 0x7f7d1419e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 -- 172.21.15.69:0/246584231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14107660 msgr2=0x7f7d141988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14107660 0x7f7d141988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 -- 172.21.15.69:0/246584231 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7d14199080 con 0x7f7d14102f80 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d11d9b700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400b500 tx=0x7f7d0400b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d04015020 con 0x7f7d14102f80 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7d14199370 con 0x7f7d14102f80 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d04011d10 con 0x7f7d14102f80 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.444+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d0400f2e0 con 0x7f7d14102f80 2024-05-09T09:17:07.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.445+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7d141a5530 con 0x7f7d14102f80 2024-05-09T09:17:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.445+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7d14103a60 con 0x7f7d14102f80 2024-05-09T09:17:07.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.446+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7d04018020 con 0x7f7d14102f80 2024-05-09T09:17:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.447+0000 7f7d037fe700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 0x7f7cfc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.448+0000 7f7d1259c700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 0x7f7cfc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.448+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f7d040ceb50 con 0x7f7d14102f80 2024-05-09T09:17:07.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.448+0000 7f7d1259c700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 0x7f7cfc0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005ea0 tx=0x7f7d08005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:07.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.452+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7d0401f020 con 0x7f7d14102f80 2024-05-09T09:17:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:07 smithi069 ceph-mon[30123]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:07.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:07 smithi163 ceph-mon[34354]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:07 smithi103 ceph-mon[34421]: pgmap v308: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.3 KiB/s rd, 5.9 KiB/s wr, 12 op/s 2024-05-09T09:17:07.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.649+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f7d1402db80 con 0x7f7d14102f80 2024-05-09T09:17:07.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.650+0000 7f7d037fe700 1 -- 172.21.15.69:0/246584231 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) ==== 74+0+24116 (secure 0 0 0) 0x7f7d0409b820 con 0x7f7d14102f80 2024-05-09T09:17:07.649 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:17:07.649 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":89,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:16:37.254456+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:17:07.653 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 msgr2=0x7f7cfc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 0x7f7cfc0a5170 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f7d08005ea0 tx=0x7f7d08005e30 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 msgr2=0x7f7d1419ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7d0400b500 tx=0x7f7d0400b8c0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 shutdown_connections 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7cfc0a2cb0 0x7f7cfc0a5170 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7d14107660 0x7f7d141988a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7d141025d0 0x7f7d1419e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 --2- 172.21.15.69:0/246584231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7d14102f80 0x7f7d1419ec10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 >> 172.21.15.69:0/246584231 conn(0x7f7d14079480 msgr2=0x7f7d14100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:07.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 shutdown_connections 2024-05-09T09:17:07.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:07.653+0000 7f7d188ed700 1 -- 172.21.15.69:0/246584231 wait complete. 2024-05-09T09:17:08.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-05-09T09:17:08.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-05-09T09:17:08.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-05-09T09:17:08.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-05-09T09:17:08.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-05-09T09:17:08.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-05-09T09:17:08.319 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.6 flush_pg_stats 2024-05-09T09:17:08.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.7 flush_pg_stats 2024-05-09T09:17:08.320 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.8 flush_pg_stats 2024-05-09T09:17:08.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.9 flush_pg_stats 2024-05-09T09:17:08.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.10 flush_pg_stats 2024-05-09T09:17:08.321 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph tell osd.11 flush_pg_stats 2024-05-09T09:17:08.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:08 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/246584231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:08.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:08 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/246584231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:08 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/246584231' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:17:08.989 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:08.993 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:08.996 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:08.996 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:09.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:09 smithi069 ceph-mon[30123]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-09T09:17:09.546 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:09 smithi103 ceph-mon[34421]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-09T09:17:09.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:09 smithi163 ceph-mon[34354]: pgmap v309: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 2.2 KiB/s wr, 7 op/s 2024-05-09T09:17:11.402 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:11.403 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:11.403 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:11.403 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:11.404 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:12.440 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:12 smithi069 ceph-mon[30123]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-09T09:17:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:12 smithi163 ceph-mon[34354]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-09T09:17:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:12 smithi103 ceph-mon[34421]: pgmap v310: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.8 KiB/s wr, 6 op/s 2024-05-09T09:17:14.070 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:14.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:14 smithi069 ceph-mon[30123]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:14 smithi163 ceph-mon[34354]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:14.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:14 smithi103 ceph-mon[34421]: pgmap v311: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:14.759 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:16.448 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:16 smithi069 ceph-mon[30123]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T09:17:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:16 smithi163 ceph-mon[34354]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T09:17:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:16 smithi103 ceph-mon[34421]: pgmap v312: 129 pgs: 129 active+clean; 637 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 1.2 KiB/s wr, 5 op/s 2024-05-09T09:17:17.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 -- 172.21.15.69:0/1481066627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c101210 msgr2=0x7f607c101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 --2- 172.21.15.69:0/1481066627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c101210 0x7f607c101690 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f606c009990 tx=0x7f606c009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 -- 172.21.15.69:0/1481066627 shutdown_connections 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 --2- 172.21.15.69:0/1481066627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 0x7f607c10e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 --2- 172.21.15.69:0/1481066627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f607c107280 0x7f607c107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 --2- 172.21.15.69:0/1481066627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c101210 0x7f607c101690 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.115+0000 7f60838da700 1 -- 172.21.15.69:0/1481066627 >> 172.21.15.69:0/1481066627 conn(0x7f607c078190 msgr2=0x7f607c0785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- 172.21.15.69:0/1481066627 shutdown_connections 2024-05-09T09:17:17.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- 172.21.15.69:0/1481066627 wait complete. 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 Processor -- start 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- start start 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f607c101210 0x7f607c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 0x7f607c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 0x7f607c1a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:17.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f607c111570 con 0x7f607c107280 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1113f0 con 0x7f607c101bd0 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.116+0000 7f60838da700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f607c1116f0 con 0x7f607c101210 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 0x7f607c1a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 0x7f607c1a4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55900/0 (socket says 172.21.15.69:55900) 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 -- 172.21.15.69:0/2905575808 learned_addr learned my addr 172.21.15.69:0/2905575808 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081676700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f607c101210 0x7f607c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6080e75700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 0x7f607c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 -- 172.21.15.69:0/2905575808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f607c101210 msgr2=0x7f607c199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f607c101210 0x7f607c199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 -- 172.21.15.69:0/2905575808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 msgr2=0x7f607c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 0x7f607c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f607c1a4f50 con 0x7f607c107280 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6080e75700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f607c101bd0 0x7f607c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:17.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f6081e77700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 0x7f607c1a4a10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f607400ada0 tx=0x7f607400c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074017020 con 0x7f607c107280 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f607c1a51e0 con 0x7f607c107280 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f607400a0d0 con 0x7f607c107280 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6074005480 con 0x7f607c107280 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.117+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f607c1a56f0 con 0x7f607c107280 2024-05-09T09:17:17.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.118+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f607c062a30 con 0x7f607c107280 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.119+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6074005620 con 0x7f607c107280 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f60727fc700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60680a2cb0 0x7f60680a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f6081676700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60680a2cb0 0x7f60680a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f60740cf570 con 0x7f607c107280 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f60727fc700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] conn(0x7f60680ad970 0x7f60680afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f607401ee50 con 0x7f60680ad970 2024-05-09T09:17:17.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f60740cf9c0 con 0x7f607c107280 2024-05-09T09:17:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f6081676700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60680a2cb0 0x7f60680a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6078009a10 tx=0x7f6078009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.120+0000 7f6080e75700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] conn(0x7f60680ad970 0x7f60680afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.121+0000 7f6080e75700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] conn(0x7f60680ad970 0x7f60680afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:17.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.121+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== osd.3 v2:172.21.15.69:6810/3308296525 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f607401ee50 con 0x7f60680ad970 2024-05-09T09:17:17.136 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 --> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f607c062e30 con 0x7f60680ad970 2024-05-09T09:17:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60727fc700 1 -- 172.21.15.69:0/2905575808 <== osd.3 v2:172.21.15.69:6810/3308296525 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f607c062e30 con 0x7f60680ad970 2024-05-09T09:17:17.137 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] conn(0x7f60680ad970 msgr2=0x7f60680afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] conn(0x7f60680ad970 0x7f60680afdb0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60680a2cb0 msgr2=0x7f60680a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60680a2cb0 0x7f60680a5170 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f6078009a10 tx=0x7f6078009190 comp rx=0 tx=0).stop 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 msgr2=0x7f607c1a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 --2- 172.21.15.69:0/2905575808 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f607c107280 0x7f607c1a4a10 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f607400ada0 tx=0x7f607400c330 comp rx=0 tx=0).stop 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f6081e77700 1 -- 172.21.15.69:0/2905575808 reap_dead start 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 shutdown_connections 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.138+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 >> 172.21.15.69:0/2905575808 conn(0x7f607c078190 msgr2=0x7f607c0ffe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.139+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 shutdown_connections 2024-05-09T09:17:17.138 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:17.139+0000 7f60838da700 1 -- 172.21.15.69:0/2905575808 wait complete. 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- 172.21.15.69:0/1059889389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406aab0 msgr2=0x7f33e406af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1059889389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406aab0 0x7f33e406af10 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0099c0 tx=0x7f33cc009cd0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- 172.21.15.69:0/1059889389 shutdown_connections 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1059889389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406b450 0x7f33e410f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1059889389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33e41083b0 0x7f33e4108790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1059889389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406aab0 0x7f33e406af10 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- 172.21.15.69:0/1059889389 >> 172.21.15.69:0/1059889389 conn(0x7f33e4078150 msgr2=0x7f33e4078560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- 172.21.15.69:0/1059889389 shutdown_connections 2024-05-09T09:17:18.373 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- 172.21.15.69:0/1059889389 wait complete. 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 Processor -- start 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.374+0000 7f33ea27e700 1 -- start start 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406aab0 0x7f33e4101330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 0x7f33e4101870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33e41083b0 0x7f33e4101db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4112610 con 0x7f33e406b450 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4112490 con 0x7f33e406aab0 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33ea27e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33e4112790 con 0x7f33e41083b0 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 0x7f33e4101870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:18.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 0x7f33e4101870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45602/0 (socket says 172.21.15.69:45602) 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 -- 172.21.15.69:0/1525548092 learned_addr learned my addr 172.21.15.69:0/1525548092 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33e3fff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406aab0 0x7f33e4101330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 -- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33e41083b0 msgr2=0x7f33e4101db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f33e41083b0 0x7f33e4101db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 -- 172.21.15.69:0/1525548092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406aab0 msgr2=0x7f33e4101330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406aab0 0x7f33e4101330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f33e41a7010 con 0x7f33e406b450 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33e3fff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f33e406aab0 0x7f33e4101330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.375+0000 7f33dbfff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 0x7f33e4101870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0098e0 tx=0x7f33cc004580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.376+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc027070 con 0x7f33e406b450 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.376+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33cc004170 con 0x7f33e406b450 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.376+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f33e41a72a0 con 0x7f33e406b450 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.376+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33cc00f460 con 0x7f33e406b450 2024-05-09T09:17:18.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.376+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f33e41a76f0 con 0x7f33e406b450 2024-05-09T09:17:18.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.377+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f33e4062a30 con 0x7f33e406b450 2024-05-09T09:17:18.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.378+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f33cc02a070 con 0x7f33e406b450 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e1ffb700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33d00a2cb0 0x7f33d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e3fff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33d00a2cb0 0x7f33d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f33cc0d09a0 con 0x7f33e406b450 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e1ffb700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] conn(0x7f33d00ad970 0x7f33d00afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f33cc0043d0 con 0x7f33d00ad970 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f33cc0d0df0 con 0x7f33e406b450 2024-05-09T09:17:18.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.379+0000 7f33e3fff700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33d00a2cb0 0x7f33d00a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33d4005e10 tx=0x7f33d4005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:18.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.380+0000 7f33e881b700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] conn(0x7f33d00ad970 0x7f33d00afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:18.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.380+0000 7f33e881b700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] conn(0x7f33d00ad970 0x7f33d00afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:18.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.381+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== osd.8 v2:172.21.15.163:6816/2914036945 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f33cc0043d0 con 0x7f33d00ad970 2024-05-09T09:17:18.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.398+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 --> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f33e4062eb0 con 0x7f33d00ad970 2024-05-09T09:17:18.397 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33e1ffb700 1 -- 172.21.15.69:0/1525548092 <== osd.8 v2:172.21.15.163:6816/2914036945 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f33e4062eb0 con 0x7f33d00ad970 2024-05-09T09:17:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] conn(0x7f33d00ad970 msgr2=0x7f33d00afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] conn(0x7f33d00ad970 0x7f33d00afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33d00a2cb0 msgr2=0x7f33d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33d00a2cb0 0x7f33d00a5170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f33d4005e10 tx=0x7f33d4005d80 comp rx=0 tx=0).stop 2024-05-09T09:17:18.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 msgr2=0x7f33e4101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 --2- 172.21.15.69:0/1525548092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f33e406b450 0x7f33e4101870 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f33cc0098e0 tx=0x7f33cc004580 comp rx=0 tx=0).stop 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33e881b700 1 -- 172.21.15.69:0/1525548092 reap_dead start 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 shutdown_connections 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 >> 172.21.15.69:0/1525548092 conn(0x7f33e4078150 msgr2=0x7f33e4100020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 shutdown_connections 2024-05-09T09:17:18.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:18.399+0000 7f33ea27e700 1 -- 172.21.15.69:0/1525548092 wait complete. 2024-05-09T09:17:18.474 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:18 smithi069 ceph-mon[30123]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:18 smithi163 ceph-mon[34354]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:18 smithi103 ceph-mon[34421]: pgmap v313: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 7 op/s 2024-05-09T09:17:20.505 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:20 smithi069 ceph-mon[30123]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:17:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:20 smithi163 ceph-mon[34354]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:17:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:20 smithi103 ceph-mon[34421]: pgmap v314: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 -- 172.21.15.69:0/1062664017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff600101860 msgr2=0x7ff600101c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 --2- 172.21.15.69:0/1062664017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff600101860 0x7ff600101c40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff5f00099c0 tx=0x7ff5f0009cd0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 -- 172.21.15.69:0/1062664017 shutdown_connections 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 --2- 172.21.15.69:0/1062664017 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6001065b0 0x7ff60010e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 --2- 172.21.15.69:0/1062664017 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff600101860 0x7ff600101c40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 --2- 172.21.15.69:0/1062664017 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff600102210 0x7ff600105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 -- 172.21.15.69:0/1062664017 >> 172.21.15.69:0/1062664017 conn(0x7ff6000781d0 msgr2=0x7ff6000785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 -- 172.21.15.69:0/1062664017 shutdown_connections 2024-05-09T09:17:20.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.727+0000 7ff6058d6700 1 -- 172.21.15.69:0/1062664017 wait complete. 2024-05-09T09:17:20.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 Processor -- start 2024-05-09T09:17:20.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 -- start start 2024-05-09T09:17:20.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff600101860 0x7ff600199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:20.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff600102210 0x7ff600199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:20.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 0x7ff6001a4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff600111590 con 0x7ff6001065b0 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff600111410 con 0x7ff600101860 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff6058d6700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff600111710 con 0x7ff600102210 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff5ff7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 0x7ff6001a4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff5ff7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 0x7ff6001a4a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45628/0 (socket says 172.21.15.69:45628) 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff5feffd700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff600101860 0x7ff600199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff5ff7fe700 1 -- 172.21.15.69:0/3470920783 learned_addr learned my addr 172.21.15.69:0/3470920783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.728+0000 7ff5fe7fc700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff600102210 0x7ff600199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 -- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff600102210 msgr2=0x7ff600199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff600102210 0x7ff600199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 -- 172.21.15.69:0/3470920783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff600101860 msgr2=0x7ff600199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff600101860 0x7ff600199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6001a4f50 con 0x7ff6001065b0 2024-05-09T09:17:20.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5fe7fc700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff600102210 0x7ff600199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff5ff7fe700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 0x7ff6001a4a10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800ada0 tx=0x7ff5f800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8017020 con 0x7ff6001065b0 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6001a5180 con 0x7ff6001065b0 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.729+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5f800a0d0 con 0x7ff6001065b0 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.730+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5f8005480 con 0x7ff6001065b0 2024-05-09T09:17:20.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.730+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6001a56e0 con 0x7ff6001065b0 2024-05-09T09:17:20.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.730+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ff6000629f0 con 0x7ff6001065b0 2024-05-09T09:17:20.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.731+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ff5f8005620 con 0x7ff6001065b0 2024-05-09T09:17:20.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff6048d4700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5e80a2d30 0x7ff5e80a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:20.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff5feffd700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5e80a2d30 0x7ff5e80a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:20.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7ff5f80cea50 con 0x7ff6001065b0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff6048d4700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] conn(0x7ff5e80ad9f0 0x7ff5e80afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ff5e80b0500 con 0x7ff5e80ad9f0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7ff5f80d32c0 con 0x7ff6001065b0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.732+0000 7ff5feffd700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5e80a2d30 0x7ff5e80a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000c00 tx=0x7ff5f0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.733+0000 7ff5fe7fc700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] conn(0x7ff5e80ad9f0 0x7ff5e80afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.733+0000 7ff5fe7fc700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] conn(0x7ff5e80ad9f0 0x7ff5e80afe30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:20.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.734+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== osd.11 v2:172.21.15.163:6824/208079664 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ff5e80b0500 con 0x7ff5e80ad9f0 2024-05-09T09:17:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.750+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 --> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ff600062ea0 con 0x7ff5e80ad9f0 2024-05-09T09:17:20.750 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6048d4700 1 -- 172.21.15.69:0/3470920783 <== osd.11 v2:172.21.15.163:6824/208079664 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ff600062ea0 con 0x7ff5e80ad9f0 2024-05-09T09:17:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] conn(0x7ff5e80ad9f0 msgr2=0x7ff5e80afe30 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] conn(0x7ff5e80ad9f0 0x7ff5e80afe30 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5e80a2d30 msgr2=0x7ff5e80a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5e80a2d30 0x7ff5e80a51f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7ff5f0000c00 tx=0x7ff5f0005fb0 comp rx=0 tx=0).stop 2024-05-09T09:17:20.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 msgr2=0x7ff6001a4a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 --2- 172.21.15.69:0/3470920783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6001065b0 0x7ff6001a4a10 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7ff5f800ada0 tx=0x7ff5f800c330 comp rx=0 tx=0).stop 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff5ff7fe700 1 -- 172.21.15.69:0/3470920783 reap_dead start 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 shutdown_connections 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 >> 172.21.15.69:0/3470920783 conn(0x7ff6000781d0 msgr2=0x7ff600107700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 shutdown_connections 2024-05-09T09:17:20.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:20.751+0000 7ff6058d6700 1 -- 172.21.15.69:0/3470920783 wait complete. 2024-05-09T09:17:21.720 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 -- 172.21.15.69:0/943857801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 msgr2=0x7f03ec1087c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 --2- 172.21.15.69:0/943857801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec1087c0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f03e00099c0 tx=0x7f03e0009cd0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 -- 172.21.15.69:0/943857801 shutdown_connections 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 --2- 172.21.15.69:0/943857801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03ec102d90 0x7f03ec10f200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 --2- 172.21.15.69:0/943857801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec1087c0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 --2- 172.21.15.69:0/943857801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03ec1023d0 0x7f03ec102850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 -- 172.21.15.69:0/943857801 >> 172.21.15.69:0/943857801 conn(0x7f03ec079480 msgr2=0x7f03ec079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 -- 172.21.15.69:0/943857801 shutdown_connections 2024-05-09T09:17:21.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.728+0000 7f03f245f700 1 -- 172.21.15.69:0/943857801 wait complete. 2024-05-09T09:17:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 Processor -- start 2024-05-09T09:17:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 -- start start 2024-05-09T09:17:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03ec1023d0 0x7f03ec074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03ec102d90 0x7f03ec074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:21.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec0750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec112660 con 0x7f03ec1083e0 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec1124e0 con 0x7f03ec102d90 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f245f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f03ec1127e0 con 0x7f03ec1023d0 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f09fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec0750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f09fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec0750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45646/0 (socket says 172.21.15.69:45646) 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03f09fc700 1 -- 172.21.15.69:0/1156737041 learned_addr learned my addr 172.21.15.69:0/1156737041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03ebfff700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03ec1023d0 0x7f03ec074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.729+0000 7f03eb7fe700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03ec102d90 0x7f03ec074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 -- 172.21.15.69:0/1156737041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03ec1023d0 msgr2=0x7f03ec074620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f03ec1023d0 0x7f03ec074620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 -- 172.21.15.69:0/1156737041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03ec102d90 msgr2=0x7f03ec074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f03ec102d90 0x7f03ec074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f03ec1a6f60 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f09fc700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec0750a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f03e400cca0 tx=0x7f03e400a060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e4016020 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f03ec1a7250 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f03e4012420 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f03e40115e0 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.730+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f03ec1a7740 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.731+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f03ec062a30 con 0x7f03ec1083e0 2024-05-09T09:17:21.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.733+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f03e4019020 con 0x7f03ec1083e0 2024-05-09T09:17:21.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.733+0000 7f03e97fa700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f03d40a2cb0 0x7f03d40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:21.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.733+0000 7f03ebfff700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f03d40a2cb0 0x7f03d40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:21.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.733+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f03e40cf660 con 0x7f03ec1083e0 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03e97fa700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] conn(0x7f03d40ad970 0x7f03d40afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f03e4011ec0 con 0x7f03d40ad970 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f03e40cfab0 con 0x7f03ec1083e0 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03eb7fe700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] conn(0x7f03d40ad970 0x7f03d40afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03ebfff700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f03d40a2cb0 0x7f03d40a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f03e0000c00 tx=0x7f03e0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:21.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.734+0000 7f03eb7fe700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] conn(0x7f03d40ad970 0x7f03d40afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:21.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.735+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== osd.0 v2:172.21.15.69:6802/1965346268 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f03e4011ec0 con 0x7f03d40ad970 2024-05-09T09:17:21.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.753+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 --> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f03ec062e30 con 0x7f03d40ad970 2024-05-09T09:17:21.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03e97fa700 1 -- 172.21.15.69:0/1156737041 <== osd.0 v2:172.21.15.69:6802/1965346268 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f03ec062e30 con 0x7f03d40ad970 2024-05-09T09:17:21.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] conn(0x7f03d40ad970 msgr2=0x7f03d40afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] conn(0x7f03d40ad970 0x7f03d40afdb0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f03d40a2cb0 msgr2=0x7f03d40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f03d40a2cb0 0x7f03d40a5170 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f03e0000c00 tx=0x7f03e0005fb0 comp rx=0 tx=0).stop 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 msgr2=0x7f03ec0750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 --2- 172.21.15.69:0/1156737041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f03ec1083e0 0x7f03ec0750a0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f03e400cca0 tx=0x7f03e400a060 comp rx=0 tx=0).stop 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f09fc700 1 -- 172.21.15.69:0/1156737041 reap_dead start 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 shutdown_connections 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 >> 172.21.15.69:0/1156737041 conn(0x7f03ec079480 msgr2=0x7f03ec100ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 shutdown_connections 2024-05-09T09:17:21.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:21.754+0000 7f03f245f700 1 -- 172.21.15.69:0/1156737041 wait complete. 2024-05-09T09:17:22.370 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:22 smithi069 ceph-mon[30123]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:22 smithi163 ceph-mon[34354]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:22 smithi103 ceph-mon[34421]: pgmap v315: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:22.769 INFO:teuthology.orchestra.run.smithi069.stdout:107374182463 2024-05-09T09:17:22.769 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-05-09T09:17:23.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 -- 172.21.15.69:0/139932196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 msgr2=0x7f221010e0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 --2- 172.21.15.69:0/139932196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f221010e0d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f220800b0a0 tx=0x7f220800b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 -- 172.21.15.69:0/139932196 shutdown_connections 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 --2- 172.21.15.69:0/139932196 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f221010e0d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 --2- 172.21.15.69:0/139932196 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 0x7f22101076b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 --2- 172.21.15.69:0/139932196 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2210101250 0x7f22101016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 -- 172.21.15.69:0/139932196 >> 172.21.15.69:0/139932196 conn(0x7f2210077290 msgr2=0x7f22100776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 -- 172.21.15.69:0/139932196 shutdown_connections 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.326+0000 7f22148f4700 1 -- 172.21.15.69:0/139932196 wait complete. 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 Processor -- start 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 -- start start 2024-05-09T09:17:23.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2210101250 0x7f2210103270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f22101037b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 0x7f2210103cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f22101115f0 con 0x7f2210101c10 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2210111470 con 0x7f22101072d0 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f22148f4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2210111770 con 0x7f2210101250 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.327+0000 7f220dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f22101037b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f22101037b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45670/0 (socket says 172.21.15.69:45670) 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 -- 172.21.15.69:0/1769745591 learned_addr learned my addr 172.21.15.69:0/1769745591 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220ed9d700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 0x7f2210103cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:23.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220e59c700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2210101250 0x7f2210103270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 -- 172.21.15.69:0/1769745591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2210101250 msgr2=0x7f2210103270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2210101250 0x7f2210103270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 -- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 msgr2=0x7f2210103cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 0x7f2210103cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f22101a7100 con 0x7f2210101c10 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220ed9d700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f22101072d0 0x7f2210103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f220dd9b700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f22101037b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f220400b530 tx=0x7f220400b8f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:23.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204015020 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2204011d10 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f22101a73f0 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2204010e50 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.328+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f22101a78e0 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.329+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f221006ad60 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.330+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f2204018020 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f21ff7fe700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f21f80a2e50 0x7f21f80a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f220e59c700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f21f80a2e50 0x7f21f80a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f22040cf1e0 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f21ff7fe700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] conn(0x7f21f80adb10 0x7f21f80aff50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f2204013ec0 con 0x7f21f80adb10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f22040cf630 con 0x7f2210101c10 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f220e59c700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f21f80a2e50 0x7f21f80a5310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2200006fd0 tx=0x7f2200008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:23.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.331+0000 7f220ed9d700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] conn(0x7f21f80adb10 0x7f21f80aff50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.332+0000 7f220ed9d700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] conn(0x7f21f80adb10 0x7f21f80aff50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.333+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== osd.7 v2:172.21.15.103:6816/4108396337 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f2204013ec0 con 0x7f21f80adb10 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.349+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 --> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f2210102940 con 0x7f21f80adb10 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.349+0000 7f21ff7fe700 1 -- 172.21.15.69:0/1769745591 <== osd.7 v2:172.21.15.103:6816/4108396337 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f2210102940 con 0x7f21f80adb10 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] conn(0x7f21f80adb10 msgr2=0x7f21f80aff50 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] conn(0x7f21f80adb10 0x7f21f80aff50 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f21f80a2e50 msgr2=0x7f21f80a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f21f80a2e50 0x7f21f80a5310 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2200006fd0 tx=0x7f2200008040 comp rx=0 tx=0).stop 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 msgr2=0x7f22101037b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 --2- 172.21.15.69:0/1769745591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2210101c10 0x7f22101037b0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f220400b530 tx=0x7f220400b8f0 comp rx=0 tx=0).stop 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f220ed9d700 1 -- 172.21.15.69:0/1769745591 reap_dead start 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 shutdown_connections 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 >> 172.21.15.69:0/1769745591 conn(0x7f2210077290 msgr2=0x7f2210105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 shutdown_connections 2024-05-09T09:17:23.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:23.350+0000 7f22148f4700 1 -- 172.21.15.69:0/1769745591 wait complete. 2024-05-09T09:17:24.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:24 smithi069 ceph-mon[30123]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:17:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:24 smithi163 ceph-mon[34354]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:17:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:24 smithi103 ceph-mon[34421]: pgmap v316: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.096+0000 7f0f3a3af700 1 -- 172.21.15.69:0/3060873171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f341083a0 msgr2=0x7f0f34108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.096+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/3060873171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f341083a0 0x7f0f34108780 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f0f24009990 tx=0x7f0f24009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.096+0000 7f0f3a3af700 1 -- 172.21.15.69:0/3060873171 shutdown_connections 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.096+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/3060873171 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f3406a9d0 0x7f0f3410f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.096+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/3060873171 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f341083a0 0x7f0f34108780 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/3060873171 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f3406a030 0x7f0f3406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 -- 172.21.15.69:0/3060873171 >> 172.21.15.69:0/3060873171 conn(0x7f0f340781d0 msgr2=0x7f0f340785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 -- 172.21.15.69:0/3060873171 shutdown_connections 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 -- 172.21.15.69:0/3060873171 wait complete. 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 Processor -- start 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 -- start start 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f3406a030 0x7f0f3419e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.097+0000 7f0f3a3af700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 0x7f0f3419ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f341083a0 0x7f0f341989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:24.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0f341125e0 con 0x7f0f3406a9d0 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112460 con 0x7f0f341083a0 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0f34112760 con 0x7f0f3406a030 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 0x7f0f3419ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 0x7f0f3419ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45676/0 (socket says 172.21.15.69:45676) 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 -- 172.21.15.69:0/33360761 learned_addr learned my addr 172.21.15.69:0/33360761 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f33fff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f3406a030 0x7f0f3419e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3894c700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f341083a0 0x7f0f341989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 -- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f3406a030 msgr2=0x7f0f3419e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0f3406a030 0x7f0f3419e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 -- 172.21.15.69:0/33360761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f341083a0 msgr2=0x7f0f341989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f341083a0 0x7f0f341989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0f34199120 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3894c700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0f341083a0 0x7f0f341989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f2bfff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 0x7f0f3419ed40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00d950 tx=0x7f0f1c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c016070 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0f34199410 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0f1c009e20 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0f1c00f620 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.098+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0f341a56d0 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.099+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f0f34062a30 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.101+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f0f1c019070 con 0x7f0f3406a9d0 2024-05-09T09:17:24.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f31ffb700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0f200a2cb0 0x7f0f200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f0f1c0cf560 con 0x7f0f3406a9d0 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f33fff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0f200a2cb0 0x7f0f200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f31ffb700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] conn(0x7f0f200ad970 0x7f0f200afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f0f1c010e50 con 0x7f0f200ad970 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f0f1c0cf910 con 0x7f0f3406a9d0 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f33fff700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0f200a2cb0 0x7f0f200a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400b5c0 tx=0x7f0f24005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:24.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.102+0000 7f0f3894c700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] conn(0x7f0f200ad970 0x7f0f200afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.103+0000 7f0f3894c700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] conn(0x7f0f200ad970 0x7f0f200afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.104+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== osd.2 v2:172.21.15.163:6800/250759774 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f0f1c010e50 con 0x7f0f200ad970 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.120+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 --> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f0f34062eb0 con 0x7f0f200ad970 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.121+0000 7f0f31ffb700 1 -- 172.21.15.69:0/33360761 <== osd.2 v2:172.21.15.163:6800/250759774 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f0f34062eb0 con 0x7f0f200ad970 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] conn(0x7f0f200ad970 msgr2=0x7f0f200afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] conn(0x7f0f200ad970 0x7f0f200afdb0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0f200a2cb0 msgr2=0x7f0f200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0f200a2cb0 0x7f0f200a5170 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f0f2400b5c0 tx=0x7f0f24005960 comp rx=0 tx=0).stop 2024-05-09T09:17:24.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 msgr2=0x7f0f3419ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 --2- 172.21.15.69:0/33360761 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0f3406a9d0 0x7f0f3419ed40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f0f1c00d950 tx=0x7f0f1c00dc60 comp rx=0 tx=0).stop 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3894c700 1 -- 172.21.15.69:0/33360761 reap_dead start 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 shutdown_connections 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 >> 172.21.15.69:0/33360761 conn(0x7f0f340781d0 msgr2=0x7f0f34100000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 shutdown_connections 2024-05-09T09:17:24.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:24.122+0000 7f0f3a3af700 1 -- 172.21.15.69:0/33360761 wait complete. 2024-05-09T09:17:25.341 INFO:teuthology.orchestra.run.smithi069.stdout:77309411398 2024-05-09T09:17:25.341 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 -- 172.21.15.69:0/44335591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238107660 msgr2=0x7fa23810f230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/44335591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238107660 0x7fa23810f230 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fa23000b0a0 tx=0x7fa23000b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 -- 172.21.15.69:0/44335591 shutdown_connections 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/44335591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238107660 0x7fa23810f230 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/44335591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa2381025d0 0x7fa2381029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/44335591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa238102f80 0x7fa238106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 -- 172.21.15.69:0/44335591 >> 172.21.15.69:0/44335591 conn(0x7fa238079480 msgr2=0x7fa238079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:26.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 -- 172.21.15.69:0/44335591 shutdown_connections 2024-05-09T09:17:26.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.208+0000 7fa23c8e4700 1 -- 172.21.15.69:0/44335591 wait complete. 2024-05-09T09:17:26.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 Processor -- start 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 -- start start 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2381025d0 0x7fa23819e720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 0x7fa23819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa238107660 0x7fa2381988e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa238112640 con 0x7fa238102f80 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa2381124c0 con 0x7fa238107660 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa23c8e4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa2381127c0 con 0x7fa2381025d0 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa235d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 0x7fa23819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa235d9b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 0x7fa23819ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45688/0 (socket says 172.21.15.69:45688) 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.209+0000 7fa235d9b700 1 -- 172.21.15.69:0/2074918020 learned_addr learned my addr 172.21.15.69:0/2074918020 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa23659c700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2381025d0 0x7fa23819e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa236d9d700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa238107660 0x7fa2381988e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 -- 172.21.15.69:0/2074918020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2381025d0 msgr2=0x7fa23819e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa2381025d0 0x7fa23819e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 -- 172.21.15.69:0/2074918020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa238107660 msgr2=0x7fa2381988e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa238107660 0x7fa2381988e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa238199040 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa235d9b700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 0x7fa23819ec60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00d950 tx=0x7fa22c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c017070 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa22c00de30 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa238199330 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa22c0054c0 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.210+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa2381a55f0 con 0x7fa238102f80 2024-05-09T09:17:26.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.211+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fa238062a70 con 0x7fa238102f80 2024-05-09T09:17:26.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.212+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa22c005660 con 0x7fa238102f80 2024-05-09T09:17:26.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa2277fe700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2200a2cb0 0x7fa2200a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa23659c700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2200a2cb0 0x7fa2200a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fa22c0cf0f0 con 0x7fa238102f80 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa2277fe700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] conn(0x7fa2200ad970 0x7fa2200afdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fa2200b0480 con 0x7fa2200ad970 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fa22c0cf540 con 0x7fa238102f80 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa236d9d700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] conn(0x7fa2200ad970 0x7fa2200afdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa23659c700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2200a2cb0 0x7fa2200a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa228005950 tx=0x7fa2280058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.213+0000 7fa236d9d700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] conn(0x7fa2200ad970 0x7fa2200afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.214+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== osd.6 v2:172.21.15.69:6818/3160153338 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fa2200b0480 con 0x7fa2200ad970 2024-05-09T09:17:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.230+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 --> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fa238062ef0 con 0x7fa2200ad970 2024-05-09T09:17:26.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa2277fe700 1 -- 172.21.15.69:0/2074918020 <== osd.6 v2:172.21.15.69:6818/3160153338 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fa238062ef0 con 0x7fa2200ad970 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] conn(0x7fa2200ad970 msgr2=0x7fa2200afdb0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] conn(0x7fa2200ad970 0x7fa2200afdb0 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2200a2cb0 msgr2=0x7fa2200a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2200a2cb0 0x7fa2200a5170 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fa228005950 tx=0x7fa2280058e0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 msgr2=0x7fa23819ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 --2- 172.21.15.69:0/2074918020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa238102f80 0x7fa23819ec60 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fa22c00d950 tx=0x7fa22c00dd10 comp rx=0 tx=0).stop 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa236d9d700 1 -- 172.21.15.69:0/2074918020 reap_dead start 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 shutdown_connections 2024-05-09T09:17:26.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 >> 172.21.15.69:0/2074918020 conn(0x7fa238079480 msgr2=0x7fa238100420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:26.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 shutdown_connections 2024-05-09T09:17:26.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.231+0000 7fa23c8e4700 1 -- 172.21.15.69:0/2074918020 wait complete. 2024-05-09T09:17:26.506 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:26 smithi069 ceph-mon[30123]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:26 smithi163 ceph-mon[34354]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:26 smithi103 ceph-mon[34421]: pgmap v317: 129 pgs: 129 active+clean; 641 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 -- 172.21.15.69:0/1136653053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 msgr2=0x7f09c8101890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 --2- 172.21.15.69:0/1136653053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c8101890 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f09b80099c0 tx=0x7f09b8009cd0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 -- 172.21.15.69:0/1136653053 shutdown_connections 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 --2- 172.21.15.69:0/1136653053 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09c8106540 0x7f09c810e110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 --2- 172.21.15.69:0/1136653053 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c8101890 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 --2- 172.21.15.69:0/1136653053 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8101e60 0x7f09c8105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.855+0000 7f09ccb83700 1 -- 172.21.15.69:0/1136653053 >> 172.21.15.69:0/1136653053 conn(0x7f09c80781d0 msgr2=0x7f09c80785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.856+0000 7f09ccb83700 1 -- 172.21.15.69:0/1136653053 shutdown_connections 2024-05-09T09:17:26.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.856+0000 7f09ccb83700 1 -- 172.21.15.69:0/1136653053 wait complete. 2024-05-09T09:17:26.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.856+0000 7f09ccb83700 1 Processor -- start 2024-05-09T09:17:26.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.856+0000 7f09ccb83700 1 -- start start 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c819e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09c8101e60 0x7f09c819ecf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8106540 0x7f09c8198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f09c8111540 con 0x7f09c81014b0 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81113c0 con 0x7f09c8101e60 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09ccb83700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f09c81116c0 con 0x7f09c8106540 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c819e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c819e7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45708/0 (socket says 172.21.15.69:45708) 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 -- 172.21.15.69:0/100493495 learned_addr learned my addr 172.21.15.69:0/100493495 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c5d9b700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09c8101e60 0x7f09c819ecf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c6d9d700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8106540 0x7f09c8198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 -- 172.21.15.69:0/100493495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8106540 msgr2=0x7f09c8198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8106540 0x7f09c8198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 -- 172.21.15.69:0/100493495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09c8101e60 msgr2=0x7f09c819ecf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f09c8101e60 0x7f09c819ecf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f09c81990d0 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c6d9d700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f09c8106540 0x7f09c8198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09c659c700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c819e7b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09b8006010 tx=0x7f09b8009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.857+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8027070 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.858+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f09b8003780 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.858+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f09c8199360 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.858+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f09b8022da0 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.858+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f09c81a5680 con 0x7f09c81014b0 2024-05-09T09:17:26.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.858+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f09c8062ab0 con 0x7f09c81014b0 2024-05-09T09:17:26.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.860+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f09b802a070 con 0x7f09c81014b0 2024-05-09T09:17:26.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.860+0000 7f09b77fe700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09b00a2cb0 0x7f09b00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09c5d9b700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09b00a2cb0 0x7f09b00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7f09b80d0660 con 0x7f09c81014b0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09b77fe700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] conn(0x7f09b00ad960 0x7f09b00afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f09b8001d30 con 0x7f09b00ad960 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7f09b80d0ab0 con 0x7f09c81014b0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09c6d9d700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] conn(0x7f09b00ad960 0x7f09b00afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09c5d9b700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09b00a2cb0 0x7f09b00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c8101120 tx=0x7f09bc00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.861+0000 7f09c6d9d700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] conn(0x7f09b00ad960 0x7f09b00afda0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:26.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.862+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== osd.9 v2:172.21.15.69:6826/2042751079 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f09b8001d30 con 0x7f09b00ad960 2024-05-09T09:17:26.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.878+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 --> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f09c8062f30 con 0x7f09b00ad960 2024-05-09T09:17:26.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.878+0000 7f09b77fe700 1 -- 172.21.15.69:0/100493495 <== osd.9 v2:172.21.15.69:6826/2042751079 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f09c8062f30 con 0x7f09b00ad960 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] conn(0x7f09b00ad960 msgr2=0x7f09b00afda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] conn(0x7f09b00ad960 0x7f09b00afda0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09b00a2cb0 msgr2=0x7f09b00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f09b00a2cb0 0x7f09b00a5170 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f09c8101120 tx=0x7f09bc00a300 comp rx=0 tx=0).stop 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 msgr2=0x7f09c819e7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 --2- 172.21.15.69:0/100493495 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f09c81014b0 0x7f09c819e7b0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f09b8006010 tx=0x7f09b8009f80 comp rx=0 tx=0).stop 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09c6d9d700 1 -- 172.21.15.69:0/100493495 reap_dead start 2024-05-09T09:17:26.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 shutdown_connections 2024-05-09T09:17:26.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 >> 172.21.15.69:0/100493495 conn(0x7f09c80781d0 msgr2=0x7f09c806ae30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:26.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 shutdown_connections 2024-05-09T09:17:26.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:26.879+0000 7f09ccb83700 1 -- 172.21.15.69:0/100493495 wait complete. 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.066+0000 7fb2dec03700 1 -- 172.21.15.69:0/2668898986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8101bd0 msgr2=0x7fb2d810e0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.066+0000 7fb2dec03700 1 --2- 172.21.15.69:0/2668898986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8101bd0 0x7fb2d810e0c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fb2d000b0a0 tx=0x7fb2d000b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 -- 172.21.15.69:0/2668898986 shutdown_connections 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 --2- 172.21.15.69:0/2668898986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8101bd0 0x7fb2d810e0c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 --2- 172.21.15.69:0/2668898986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2d8107280 0x7fb2d8107660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 --2- 172.21.15.69:0/2668898986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 0x7fb2d8101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 -- 172.21.15.69:0/2668898986 >> 172.21.15.69:0/2668898986 conn(0x7fb2d8078190 msgr2=0x7fb2d80785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:27.067 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 -- 172.21.15.69:0/2668898986 shutdown_connections 2024-05-09T09:17:27.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 -- 172.21.15.69:0/2668898986 wait complete. 2024-05-09T09:17:27.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 Processor -- start 2024-05-09T09:17:27.068 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.067+0000 7fb2dec03700 1 -- start start 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 0x7fb2d819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2d8101bd0 0x7fb2d819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 0x7fb2d81988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8111510 con 0x7fb2d8107280 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8111390 con 0x7fb2d8101210 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dec03700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2d8111690 con 0x7fb2d8101bd0 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 0x7fb2d81988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:27.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 0x7fb2d81988a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45718/0 (socket says 172.21.15.69:45718) 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 -- 172.21.15.69:0/4103559783 learned_addr learned my addr 172.21.15.69:0/4103559783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2d7fff700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2d8101bd0 0x7fb2d819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dc99f700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 0x7fb2d819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 -- 172.21.15.69:0/4103559783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2d8101bd0 msgr2=0x7fb2d819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2d8101bd0 0x7fb2d819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 -- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 msgr2=0x7fb2d819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 0x7fb2d819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2d8199080 con 0x7fb2d8107280 2024-05-09T09:17:27.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dc99f700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2d8101210 0x7fb2d819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2dd1a0700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 0x7fb2d81988a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00062a0 tx=0x7fb2d0004d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.068+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d0021070 con 0x7fb2d8107280 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.069+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2d8199310 con 0x7fb2d8107280 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.069+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2d0007c00 con 0x7fb2d8107280 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.069+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2d001d450 con 0x7fb2d8107280 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.069+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2d81a5530 con 0x7fb2d8107280 2024-05-09T09:17:27.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.070+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7fb2d806ac20 con 0x7fb2d8107280 2024-05-09T09:17:27.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.071+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fb2d0004000 con 0x7fb2d8107280 2024-05-09T09:17:27.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.071+0000 7fb2d5ffb700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2c80a2cb0 0x7fb2c80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(89..89 src has 1..89) ==== 8802+0+0 (secure 0 0 0) 0x7fb2d00d04c0 con 0x7fb2d8107280 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2dc99f700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2c80a2cb0 0x7fb2c80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d5ffb700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] conn(0x7fb2c80ad960 0x7fb2c80afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7fb2d000ce60 con 0x7fb2c80ad960 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) ==== 24+0+0 (secure 0 0 0) 0x7fb2d00d0910 con 0x7fb2d8107280 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2dc99f700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2c80a2cb0 0x7fb2c80a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc005e10 tx=0x7fb2cc005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d7fff700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] conn(0x7fb2c80ad960 0x7fb2c80afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.072+0000 7fb2d7fff700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] conn(0x7fb2c80ad960 0x7fb2c80afda0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:27.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.073+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== osd.4 v2:172.21.15.103:6808/613528366 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7fb2d000ce60 con 0x7fb2c80ad960 2024-05-09T09:17:27.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.089+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 --> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7fb2d8102930 con 0x7fb2c80ad960 2024-05-09T09:17:27.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2d5ffb700 1 -- 172.21.15.69:0/4103559783 <== osd.4 v2:172.21.15.103:6808/613528366 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7fb2d8102930 con 0x7fb2c80ad960 2024-05-09T09:17:27.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] conn(0x7fb2c80ad960 msgr2=0x7fb2c80afda0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] conn(0x7fb2c80ad960 0x7fb2c80afda0 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:27.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2c80a2cb0 msgr2=0x7fb2c80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2c80a2cb0 0x7fb2c80a5170 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fb2cc005e10 tx=0x7fb2cc005d80 comp rx=0 tx=0).stop 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 msgr2=0x7fb2d81988a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 --2- 172.21.15.69:0/4103559783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2d8107280 0x7fb2d81988a0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fb2d00062a0 tx=0x7fb2d0004d70 comp rx=0 tx=0).stop 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dd1a0700 1 -- 172.21.15.69:0/4103559783 reap_dead start 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 shutdown_connections 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.090+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 >> 172.21.15.69:0/4103559783 conn(0x7fb2d8078190 msgr2=0x7fb2d8100460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.091+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 shutdown_connections 2024-05-09T09:17:27.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:27.091+0000 7fb2dec03700 1 -- 172.21.15.69:0/4103559783 wait complete. 2024-05-09T09:17:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:28 smithi069 ceph-mon[30123]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T09:17:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:28 smithi163 ceph-mon[34354]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T09:17:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:28 smithi103 ceph-mon[34421]: pgmap v318: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 597 B/s wr, 7 op/s 2024-05-09T09:17:28.619 INFO:teuthology.orchestra.run.smithi069.stdout:253403070502 2024-05-09T09:17:28.619 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.8 2024-05-09T09:17:29.312 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T09:17:29.312 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-09T09:17:29.312 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]: dispatch 2024-05-09T09:17:29.312 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-09T09:17:29.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-09T09:17:29.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.313 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.337 INFO:teuthology.orchestra.run.smithi069.stdout:343597383703 2024-05-09T09:17:29.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.11 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]: dispatch 2024-05-09T09:17:29.700 INFO:teuthology.orchestra.run.smithi069.stdout:141733920828 2024-05-09T09:17:29.701 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-05-09T09:17:30.102 INFO:teuthology.orchestra.run.smithi069.stdout:34359738447 2024-05-09T09:17:30.103 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-09T09:17:30.423 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-05-09T09:17:30.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T09:17:30.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:30 smithi069 ceph-mon[30123]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-09T09:17:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-05-09T09:17:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T09:17:30.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:30 smithi163 ceph-mon[34354]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T09:17:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: pgmap v319: 129 pgs: 129 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 170 B/s wr, 5 op/s 2024-05-09T09:17:30.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-05-09T09:17:30.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1b", "id": [2, 4]}]': finished 2024-05-09T09:17:30.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11, 2, 1]}]': finished 2024-05-09T09:17:30.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.0", "id": [5, 9]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3", "id": [4, 9]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.34", "id": [4, 9]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.17", "id": [9, 8]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1f", "id": [9, 8]}]': finished 2024-05-09T09:17:30.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:30 smithi103 ceph-mon[34421]: osdmap e90: 12 total, 12 up, 12 in 2024-05-09T09:17:30.731 INFO:teuthology.orchestra.run.smithi069.stdout:287762808867 2024-05-09T09:17:30.732 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.9 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 -- 172.21.15.69:0/2842230281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f81072d0 msgr2=0x7f98f81076b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 --2- 172.21.15.69:0/2842230281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f81072d0 0x7f98f81076b0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f98e8009990 tx=0x7f98e8009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 -- 172.21.15.69:0/2842230281 shutdown_connections 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 --2- 172.21.15.69:0/2842230281 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101c10 0x7f98f810e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 --2- 172.21.15.69:0/2842230281 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f81072d0 0x7f98f81076b0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 --2- 172.21.15.69:0/2842230281 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98f8101250 0x7f98f81016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 -- 172.21.15.69:0/2842230281 >> 172.21.15.69:0/2842230281 conn(0x7f98f8077290 msgr2=0x7f98f80776a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 -- 172.21.15.69:0/2842230281 shutdown_connections 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.947+0000 7f98fda45700 1 -- 172.21.15.69:0/2842230281 wait complete. 2024-05-09T09:17:30.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 Processor -- start 2024-05-09T09:17:30.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 -- start start 2024-05-09T09:17:30.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101250 0x7f98f81a0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:30.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 0x7f98f81a0e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98f81072d0 0x7f98f819ab00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f98f81114f0 con 0x7f98f8101c10 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f98f8111370 con 0x7f98f8101250 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98fda45700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f98f8111670 con 0x7f98f81072d0 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98f67fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 0x7f98f81a0e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:30.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98f67fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 0x7f98f81a0e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36466/0 (socket says 172.21.15.69:36466) 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98f67fc700 1 -- 172.21.15.69:0/4066056268 learned_addr learned my addr 172.21.15.69:0/4066056268 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98f77fe700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98f81072d0 0x7f98f819ab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.948+0000 7f98f6ffd700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101250 0x7f98f81a0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 -- 172.21.15.69:0/4066056268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98f81072d0 msgr2=0x7f98f819ab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f98f81072d0 0x7f98f819ab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 -- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101250 msgr2=0x7f98f81a0940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101250 0x7f98f81a0940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f98f819b260 con 0x7f98f8101c10 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f6ffd700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f98f8101250 0x7f98f81a0940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98f67fc700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 0x7f98f81a0e80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d980 tx=0x7f98ec00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec017030 con 0x7f98f8101c10 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98ec004d10 con 0x7f98f8101c10 2024-05-09T09:17:30.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f98f819b550 con 0x7f98f8101c10 2024-05-09T09:17:30.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98ec009a90 con 0x7f98f8101c10 2024-05-09T09:17:30.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.949+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f98f81a7810 con 0x7f98f8101c10 2024-05-09T09:17:30.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.950+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f98f81006c0 con 0x7f98f8101c10 2024-05-09T09:17:30.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.951+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f98ec01a070 con 0x7f98f8101c10 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98fca43700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98e00a2cb0 0x7f98e00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98f6ffd700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98e00a2cb0 0x7f98e00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f98ec0cef10 con 0x7f98f8101c10 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98fca43700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] conn(0x7f98e00add90 0x7f98e00b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f98ec018e50 con 0x7f98e00add90 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f98ec0d15d0 con 0x7f98f8101c10 2024-05-09T09:17:30.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98f6ffd700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98e00a2cb0 0x7f98e00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98e8006010 tx=0x7f98e800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:30.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.952+0000 7f98f77fe700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] conn(0x7f98e00add90 0x7f98e00b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:30.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.953+0000 7f98f77fe700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] conn(0x7f98e00add90 0x7f98e00b0230 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:30.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.954+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== osd.1 v2:172.21.15.103:6800/4055201011 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f98ec018e50 con 0x7f98e00add90 2024-05-09T09:17:30.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.970+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 --> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f98f8102940 con 0x7f98e00add90 2024-05-09T09:17:30.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.970+0000 7f98fca43700 1 -- 172.21.15.69:0/4066056268 <== osd.1 v2:172.21.15.103:6800/4055201011 2 ==== command_reply(tid 2: 0 ) ==== 8+0+11 (crc 0 0 0) 0x7f98f8102940 con 0x7f98e00add90 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] conn(0x7f98e00add90 msgr2=0x7f98e00b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] conn(0x7f98e00add90 0x7f98e00b0230 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98e00a2cb0 msgr2=0x7f98e00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98e00a2cb0 0x7f98e00a5170 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f98e8006010 tx=0x7f98e800b540 comp rx=0 tx=0).stop 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 msgr2=0x7f98f81a0e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:30.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 --2- 172.21.15.69:0/4066056268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f98f8101c10 0x7f98f81a0e80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f98ec00d980 tx=0x7f98ec00dc90 comp rx=0 tx=0).stop 2024-05-09T09:17:30.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98f77fe700 1 -- 172.21.15.69:0/4066056268 reap_dead start 2024-05-09T09:17:30.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 shutdown_connections 2024-05-09T09:17:30.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 >> 172.21.15.69:0/4066056268 conn(0x7f98f8077290 msgr2=0x7f98f80ffca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:30.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 shutdown_connections 2024-05-09T09:17:30.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:30.971+0000 7f98fda45700 1 -- 172.21.15.69:0/4066056268 wait complete. 2024-05-09T09:17:31.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:31 smithi069 ceph-mon[30123]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T09:17:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:31 smithi163 ceph-mon[34354]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T09:17:31.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:31 smithi103 ceph-mon[34421]: osdmap e91: 12 total, 12 up, 12 in 2024-05-09T09:17:32.164 INFO:teuthology.orchestra.run.smithi069.stdout:55834574923 2024-05-09T09:17:32.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-05-09T09:17:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:32 smithi069 ceph-mon[30123]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-09T09:17:32.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:32 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:17:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:32 smithi163 ceph-mon[34354]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-09T09:17:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:32 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:17:32.563 INFO:teuthology.orchestra.run.smithi069.stdout:219043332140 2024-05-09T09:17:32.563 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.7 2024-05-09T09:17:32.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:32 smithi103 ceph-mon[34421]: pgmap v322: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 255 B/s wr, 4 op/s 2024-05-09T09:17:32.596 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:32 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:17:33.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/1605591238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58107280 msgr2=0x7ffa58107660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/1605591238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58107280 0x7ffa58107660 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c009990 tx=0x7ffa4c009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/1605591238 shutdown_connections 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/1605591238 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5810e0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/1605591238 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58107280 0x7ffa58107660 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/1605591238 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa58101210 0x7ffa58101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/1605591238 >> 172.21.15.69:0/1605591238 conn(0x7ffa58078190 msgr2=0x7ffa580785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/1605591238 shutdown_connections 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.055+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/1605591238 wait complete. 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 Processor -- start 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 -- start start 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58101210 0x7ffa5819e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa58107280 0x7ffa581988a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa58111520 con 0x7ffa58101210 2024-05-09T09:17:33.059 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581113a0 con 0x7ffa58107280 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.056+0000 7ffa5d8a4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa581116a0 con 0x7ffa58101bd0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa56ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58101210 0x7ffa5819e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39814/0 (socket says 172.21.15.69:39814) 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 -- 172.21.15.69:0/2558300780 learned_addr learned my addr 172.21.15.69:0/2558300780 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa577fe700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa58107280 0x7ffa581988a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 -- 172.21.15.69:0/2558300780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa58107280 msgr2=0x7ffa581988a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa58107280 0x7ffa581988a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 -- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58101210 msgr2=0x7ffa5819e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58101210 0x7ffa5819e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa58199000 con 0x7ffa58101bd0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa56ffd700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa58101210 0x7ffa5819e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa567fc700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5819ec20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800e9a0 tx=0x7ffa4800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.060 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa4800ca10 con 0x7ffa58101bd0 2024-05-09T09:17:33.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa4800cbb0 con 0x7ffa58101bd0 2024-05-09T09:17:33.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa581992f0 con 0x7ffa58101bd0 2024-05-09T09:17:33.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.057+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa48018680 con 0x7ffa58101bd0 2024-05-09T09:17:33.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.058+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa581a55b0 con 0x7ffa58101bd0 2024-05-09T09:17:33.061 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.058+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7ffa58105340 con 0x7ffa58101bd0 2024-05-09T09:17:33.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.060+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7ffa48010430 con 0x7ffa58101bd0 2024-05-09T09:17:33.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.060+0000 7ffa5c8a2700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa400a2cb0 0x7ffa400a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.062 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa56ffd700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa400a2cb0 0x7ffa400a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7ffa480d0280 con 0x7ffa58101bd0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa5c8a2700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] conn(0x7ffa400add90 0x7ffa400b0230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7ffa4801bec0 con 0x7ffa400add90 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== mon.1 v2:172.21.15.163:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7ffa480d06d0 con 0x7ffa58101bd0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa577fe700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] conn(0x7ffa400add90 0x7ffa400b0230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.061+0000 7ffa56ffd700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa400a2cb0 0x7ffa400a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b2e0 tx=0x7ffa4c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.062+0000 7ffa577fe700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] conn(0x7ffa400add90 0x7ffa400b0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.063 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.062+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== osd.5 v2:172.21.15.163:6808/3212961219 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7ffa4801bec0 con 0x7ffa400add90 2024-05-09T09:17:33.079 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.079+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 --> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7ffa58102930 con 0x7ffa400add90 2024-05-09T09:17:33.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5c8a2700 1 -- 172.21.15.69:0/2558300780 <== osd.5 v2:172.21.15.163:6808/3212961219 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7ffa58102930 con 0x7ffa400add90 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] conn(0x7ffa400add90 msgr2=0x7ffa400b0230 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] conn(0x7ffa400add90 0x7ffa400b0230 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa400a2cb0 msgr2=0x7ffa400a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa400a2cb0 0x7ffa400a5170 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7ffa4c00b2e0 tx=0x7ffa4c005960 comp rx=0 tx=0).stop 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 msgr2=0x7ffa5819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 --2- 172.21.15.69:0/2558300780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa58101bd0 0x7ffa5819ec20 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7ffa4800e9a0 tx=0x7ffa4800ecb0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa577fe700 1 -- 172.21.15.69:0/2558300780 reap_dead start 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 shutdown_connections 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.080+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 >> 172.21.15.69:0/2558300780 conn(0x7ffa58078190 msgr2=0x7ffa58100500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.081+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 shutdown_connections 2024-05-09T09:17:33.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.081+0000 7ffa5d8a4700 1 -- 172.21.15.69:0/2558300780 wait complete. 2024-05-09T09:17:33.361 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:33 smithi069 ceph-mon[30123]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-05-09T09:17:33.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:33 smithi069 ceph-mon[30123]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-09T09:17:33.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 -- 172.21.15.69:0/2520740780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 msgr2=0x7f4fc810f250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 --2- 172.21.15.69:0/2520740780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc810f250 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f4fc000b0a0 tx=0x7f4fc000b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 -- 172.21.15.69:0/2520740780 shutdown_connections 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 --2- 172.21.15.69:0/2520740780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc810f250 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 --2- 172.21.15.69:0/2520740780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc8108450 0x7f4fc8108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 --2- 172.21.15.69:0/2520740780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fc81023b0 0x7f4fc8102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.464+0000 7f4fce38d700 1 -- 172.21.15.69:0/2520740780 >> 172.21.15.69:0/2520740780 conn(0x7f4fc8079510 msgr2=0x7f4fc8079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.465+0000 7f4fce38d700 1 -- 172.21.15.69:0/2520740780 shutdown_connections 2024-05-09T09:17:33.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.465+0000 7f4fce38d700 1 -- 172.21.15.69:0/2520740780 wait complete. 2024-05-09T09:17:33.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.465+0000 7f4fce38d700 1 Processor -- start 2024-05-09T09:17:33.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.465+0000 7f4fce38d700 1 -- start start 2024-05-09T09:17:33.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc81023b0 0x7f4fc819e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc819ec90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fc8108450 0x7f4fc8198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8112690 con 0x7f4fc8102d70 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8112510 con 0x7f4fc81023b0 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4fc8112810 con 0x7f4fc8108450 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc819ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc819ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36490/0 (socket says 172.21.15.69:36490) 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 -- 172.21.15.69:0/327221138 learned_addr learned my addr 172.21.15.69:0/327221138 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fc7fff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc81023b0 0x7f4fc819e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fcc92a700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fc8108450 0x7f4fc8198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 -- 172.21.15.69:0/327221138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fc8108450 msgr2=0x7f4fc8198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4fc8108450 0x7f4fc8198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 -- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc81023b0 msgr2=0x7f4fc819e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4fc81023b0 0x7f4fc819e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4fc8199070 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fbffff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc819ec90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000d980 tx=0x7f4fb000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb0017070 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4fb0005020 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4fc8199360 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.466+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4fb000fbf0 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.467+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4fc81a5620 con 0x7f4fc8102d70 2024-05-09T09:17:33.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.467+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=osdmap handle=1) -- 0x7f4fc8062a30 con 0x7f4fc8102d70 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.469+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f4fb00051d0 con 0x7f4fc8102d70 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.469+0000 7f4fc5ffb700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4fb40a2d30 0x7f4fb40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.469+0000 7f4fc7fff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4fb40a2d30 0x7f4fb40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.469+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f4fb00cf810 con 0x7f4fc8102d70 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.469+0000 7f4fc5ffb700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] conn(0x7f4fb40ade10 0x7f4fb40b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.470+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] -- command(tid 1: {"prefix": "get_command_descriptions"}) -- 0x7f4fb40b0980 con 0x7f4fb40ade10 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.470+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) ==== 24+0+0 (secure 0 0 0) 0x7f4fb00cfc60 con 0x7f4fc8102d70 2024-05-09T09:17:33.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.470+0000 7f4fc7fff700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4fb40a2d30 0x7f4fb40a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8199a50 tx=0x7f4fb8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.470+0000 7f4fcc92a700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] conn(0x7f4fb40ade10 0x7f4fb40b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:33.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.470+0000 7f4fcc92a700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] conn(0x7f4fb40ade10 0x7f4fb40b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:33.472 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.471+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== osd.10 v2:172.21.15.103:6824/955568662 1 ==== command_reply(tid 1: 0 ) ==== 8+0+27193 (crc 0 0 0) 0x7f4fb40b0980 con 0x7f4fb40ade10 2024-05-09T09:17:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.487+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 --> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] -- command(tid 2: {"prefix": "flush_pg_stats"}) -- 0x7f4fc8062eb0 con 0x7f4fb40ade10 2024-05-09T09:17:33.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.487+0000 7f4fc5ffb700 1 -- 172.21.15.69:0/327221138 <== osd.10 v2:172.21.15.103:6824/955568662 2 ==== command_reply(tid 2: 0 ) ==== 8+0+12 (crc 0 0 0) 0x7f4fc8062eb0 con 0x7f4fb40ade10 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] conn(0x7f4fb40ade10 msgr2=0x7f4fb40b02b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] conn(0x7f4fb40ade10 0x7f4fb40b02b0 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4fb40a2d30 msgr2=0x7f4fb40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4fb40a2d30 0x7f4fb40a51f0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f4fc8199a50 tx=0x7f4fb8009260 comp rx=0 tx=0).stop 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 msgr2=0x7f4fc819ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:33.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 --2- 172.21.15.69:0/327221138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4fc8102d70 0x7f4fc819ec90 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4fb000d980 tx=0x7f4fb000dc90 comp rx=0 tx=0).stop 2024-05-09T09:17:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fcc92a700 1 -- 172.21.15.69:0/327221138 reap_dead start 2024-05-09T09:17:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 shutdown_connections 2024-05-09T09:17:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 >> 172.21.15.69:0/327221138 conn(0x7f4fc8079510 msgr2=0x7f4fc8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 shutdown_connections 2024-05-09T09:17:33.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:33.488+0000 7f4fce38d700 1 -- 172.21.15.69:0/327221138 wait complete. 2024-05-09T09:17:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:33 smithi163 ceph-mon[34354]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-05-09T09:17:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:33 smithi163 ceph-mon[34354]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-09T09:17:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:33 smithi103 ceph-mon[34421]: pgmap v323: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 6 op/s; 126 B/s, 0 objects/s recovering 2024-05-09T09:17:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:33 smithi103 ceph-mon[34421]: Health check failed: Reduced data availability: 1 pg inactive, 1 pg peering (PG_AVAILABILITY) 2024-05-09T09:17:33.841 INFO:teuthology.orchestra.run.smithi069.stdout:197568495666 2024-05-09T09:17:33.841 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.6 2024-05-09T09:17:34.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.455 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.482 INFO:teuthology.orchestra.run.smithi069.stdout:163208757304 2024-05-09T09:17:34.483 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-05-09T09:17:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.739 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:34.739 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:34.741 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:34.746 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:34.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:34.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:35.113 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:35.126 INFO:teuthology.orchestra.run.smithi069.stdout:309237645343 2024-05-09T09:17:35.127 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph osd last-stat-seq osd.10 2024-05-09T09:17:35.131 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:35.454 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:35 smithi069 ceph-mon[30123]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:35 smithi103 ceph-mon[34421]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:35.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:35 smithi163 ceph-mon[34354]: pgmap v324: 129 pgs: 2 peering, 127 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:37.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:37 smithi069 ceph-mon[30123]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-09T09:17:37.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:37 smithi069 ceph-mon[30123]: Cluster is now healthy 2024-05-09T09:17:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:37 smithi163 ceph-mon[34354]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-09T09:17:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:37 smithi163 ceph-mon[34354]: Cluster is now healthy 2024-05-09T09:17:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:37 smithi103 ceph-mon[34421]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive, 1 pg peering) 2024-05-09T09:17:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:37 smithi103 ceph-mon[34421]: Cluster is now healthy 2024-05-09T09:17:38.403 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:38 smithi069 ceph-mon[30123]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:38 smithi103 ceph-mon[34421]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:38 smithi163 ceph-mon[34354]: pgmap v325: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 129 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:38.606 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:39.470 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:39.471 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:40.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 -- 172.21.15.69:0/2422232162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 msgr2=0x7f54bc10e0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:40.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 --2- 172.21.15.69:0/2422232162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc10e0f0 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f54b4009990 tx=0x7f54b4009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 -- 172.21.15.69:0/2422232162 shutdown_connections 2024-05-09T09:17:40.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 --2- 172.21.15.69:0/2422232162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc10e0f0 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 --2- 172.21.15.69:0/2422232162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc106250 0x7f54bc106630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 --2- 172.21.15.69:0/2422232162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54bc101dd0 0x7f54bc102250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 -- 172.21.15.69:0/2422232162 >> 172.21.15.69:0/2422232162 conn(0x7f54bc0765c0 msgr2=0x7f54bc0769d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 -- 172.21.15.69:0/2422232162 shutdown_connections 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.429+0000 7f54c3259700 1 -- 172.21.15.69:0/2422232162 wait complete. 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 Processor -- start 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 -- start start 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 0x7f54bc19e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54bc106250 0x7f54bc198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc111600 con 0x7f54bc102790 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc111480 con 0x7f54bc101dd0 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54c3259700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f54bc111780 con 0x7f54bc106250 2024-05-09T09:17:40.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54b3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54b3fff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50806/0 (socket says 172.21.15.69:50806) 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.430+0000 7f54b3fff700 1 -- 172.21.15.69:0/3576741374 learned_addr learned my addr 172.21.15.69:0/3576741374 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54c0ff5700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 0x7f54bc19e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 -- 172.21.15.69:0/3576741374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54bc106250 msgr2=0x7f54bc198890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54bc106250 0x7f54bc198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 -- 172.21.15.69:0/3576741374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 msgr2=0x7f54bc19e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 0x7f54bc19e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 -- 172.21.15.69:0/3576741374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f54bc198ff0 con 0x7f54bc102790 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54c0ff5700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 0x7f54bc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b3fff700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00c9b0 tx=0x7f54ac00cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac007b00 con 0x7f54bc102790 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f54ac007ca0 con 0x7f54bc102790 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f54bc1992e0 con 0x7f54bc102790 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f54ac018780 con 0x7f54bc102790 2024-05-09T09:17:40.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.431+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f54bc1a55a0 con 0x7f54bc102790 2024-05-09T09:17:40.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.432+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f54bc069010 con 0x7f54bc102790 2024-05-09T09:17:40.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.433+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f54ac00f450 con 0x7f54bc102790 2024-05-09T09:17:40.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.434+0000 7f54b1ffb700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 0x7f54a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:40.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.434+0000 7f54c0ff5700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 0x7f54a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:40.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.434+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f54ac0cf580 con 0x7f54bc102790 2024-05-09T09:17:40.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.435+0000 7f54c0ff5700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 0x7f54a40a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f54bc1999d0 tx=0x7f54b80073e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:40.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.442+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f54ac014030 con 0x7f54bc102790 2024-05-09T09:17:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:40 smithi069 ceph-mon[30123]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 104 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:40.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:40 smithi163 ceph-mon[34354]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 104 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:40 smithi103 ceph-mon[34421]: pgmap v326: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s; 104 B/s, 1 keys/s, 0 objects/s recovering 2024-05-09T09:17:40.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.644+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) -- 0x7f54bc1a5950 con 0x7f54bc102790 2024-05-09T09:17:40.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.644+0000 7f54b1ffb700 1 -- 172.21.15.69:0/3576741374 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f54ac09c040 con 0x7f54bc102790 2024-05-09T09:17:40.645 INFO:teuthology.orchestra.run.smithi069.stdout:34359738450 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 msgr2=0x7f54a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 0x7f54a40a5170 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f54bc1999d0 tx=0x7f54b80073e0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 msgr2=0x7f54bc19ec10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f54ac00c9b0 tx=0x7f54ac00cd70 comp rx=0 tx=0).stop 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 shutdown_connections 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f54bc106250 0x7f54bc198890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f54a40a2cb0 0x7f54a40a5170 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f54bc101dd0 0x7f54bc19e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 --2- 172.21.15.69:0/3576741374 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f54bc102790 0x7f54bc19ec10 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:40.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 >> 172.21.15.69:0/3576741374 conn(0x7f54bc0765c0 msgr2=0x7f54bc100c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:40.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 shutdown_connections 2024-05-09T09:17:40.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:40.646+0000 7f54c3259700 1 -- 172.21.15.69:0/3576741374 wait complete. 2024-05-09T09:17:41.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:41 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3576741374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T09:17:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:41 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3576741374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T09:17:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:41 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3576741374' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-05-09T09:17:42.482 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:42 smithi069 ceph-mon[30123]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 94 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:42.517 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738447 got 34359738450 for osd.0 2024-05-09T09:17:42.517 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:42 smithi163 ceph-mon[34354]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 94 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:42 smithi103 ceph-mon[34421]: pgmap v327: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s; 94 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:44.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:44 smithi069 ceph-mon[30123]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 86 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:44 smithi163 ceph-mon[34354]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 86 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:44 smithi103 ceph-mon[34421]: pgmap v328: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s; 86 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 -- 172.21.15.69:0/1680645657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 msgr2=0x7f5840108830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 --2- 172.21.15.69:0/1680645657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840108830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f583c009990 tx=0x7f583c009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 -- 172.21.15.69:0/1680645657 shutdown_connections 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 --2- 172.21.15.69:0/1680645657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584010f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 --2- 172.21.15.69:0/1680645657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840108830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 --2- 172.21.15.69:0/1680645657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f5840102830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 -- 172.21.15.69:0/1680645657 >> 172.21.15.69:0/1680645657 conn(0x7f5840079510 msgr2=0x7f5840079920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 -- 172.21.15.69:0/1680645657 shutdown_connections 2024-05-09T09:17:44.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.878+0000 7f5848019700 1 -- 172.21.15.69:0/1680645657 wait complete. 2024-05-09T09:17:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 Processor -- start 2024-05-09T09:17:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 -- start start 2024-05-09T09:17:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f584019e760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:44.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584019eca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5840112670 con 0x7f5840108450 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58401124f0 con 0x7f5840102d70 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5848019700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58401127f0 con 0x7f58401023b0 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f58465b6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f58455b4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584019eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f58465b6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50828/0 (socket says 172.21.15.69:50828) 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f58465b6700 1 -- 172.21.15.69:0/574415931 learned_addr learned my addr 172.21.15.69:0/574415931 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.879+0000 7f5845db5700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f584019e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 -- 172.21.15.69:0/574415931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 msgr2=0x7f584019e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f584019e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 -- 172.21.15.69:0/574415931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 msgr2=0x7f584019eca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:44.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584019eca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 -- 172.21.15.69:0/574415931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58401990d0 con 0x7f5840108450 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5845db5700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f584019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58455b4700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f58465b6700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5838007ee0 tx=0x7f583800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f583800dbd0 con 0x7f5840108450 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58401993c0 con 0x7f5840108450 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f583800dd70 con 0x7f5840108450 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58401a5630 con 0x7f5840108450 2024-05-09T09:17:44.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.880+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5838005a60 con 0x7f5840108450 2024-05-09T09:17:44.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.881+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f584010a910 con 0x7f5840108450 2024-05-09T09:17:44.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.883+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f583801a020 con 0x7f5840108450 2024-05-09T09:17:44.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.883+0000 7f5836ffd700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 0x7f582c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:44.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.883+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f58380cebe0 con 0x7f5840108450 2024-05-09T09:17:44.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.884+0000 7f5845db5700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 0x7f582c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:44.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.884+0000 7f5845db5700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 0x7f582c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098e0 tx=0x7f583c00b560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:44.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:44.888+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5838010070 con 0x7f5840108450 2024-05-09T09:17:45.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.096+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) -- 0x7f5840199e50 con 0x7f5840108450 2024-05-09T09:17:45.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.096+0000 7f5836ffd700 1 -- 172.21.15.69:0/574415931 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f583809b6a0 con 0x7f5840108450 2024-05-09T09:17:45.095 INFO:teuthology.orchestra.run.smithi069.stdout:343597383707 2024-05-09T09:17:45.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 msgr2=0x7f582c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 0x7f582c0a5170 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f583c0098e0 tx=0x7f583c00b560 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 msgr2=0x7f5840198970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f5838007ee0 tx=0x7f583800d3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 shutdown_connections 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5840108450 0x7f5840198970 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58401023b0 0x7f584019e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f582c0a2cb0 0x7f582c0a5170 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 --2- 172.21.15.69:0/574415931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5840102d70 0x7f584019eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.098+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 >> 172.21.15.69:0/574415931 conn(0x7f5840079510 msgr2=0x7f58401010e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.099+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 shutdown_connections 2024-05-09T09:17:45.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.099+0000 7f5848019700 1 -- 172.21.15.69:0/574415931 wait complete. 2024-05-09T09:17:45.285 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:45.492 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/574415931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T09:17:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/574415931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T09:17:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/574415931' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 -- 172.21.15.69:0/497920798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec106990 msgr2=0x7f89ec106d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 --2- 172.21.15.69:0/497920798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec106990 0x7f89ec106d70 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f89dc00aaa0 tx=0x7f89dc00adb0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 -- 172.21.15.69:0/497920798 shutdown_connections 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 --2- 172.21.15.69:0/497920798 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec1013a0 0x7f89ec10d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 --2- 172.21.15.69:0/497920798 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec106990 0x7f89ec106d70 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 --2- 172.21.15.69:0/497920798 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1009e0 0x7f89ec100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.882+0000 7f89f2487700 1 -- 172.21.15.69:0/497920798 >> 172.21.15.69:0/497920798 conn(0x7f89ec077890 msgr2=0x7f89ec0fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 -- 172.21.15.69:0/497920798 shutdown_connections 2024-05-09T09:17:45.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 -- 172.21.15.69:0/497920798 wait complete. 2024-05-09T09:17:45.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 Processor -- start 2024-05-09T09:17:45.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 -- start start 2024-05-09T09:17:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.883+0000 7f89f2487700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1013a0 0x7f89ec0fd240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec106990 0x7f89ec0fd780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:45.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec110440 con 0x7f89ec1009e0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec1102c0 con 0x7f89ec1013a0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f89ec1105c0 con 0x7f89ec106990 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50840/0 (socket says 172.21.15.69:50840) 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 -- 172.21.15.69:0/3149856279 learned_addr learned my addr 172.21.15.69:0/3149856279 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89eb7fe700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1013a0 0x7f89ec0fd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f0a24700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec106990 0x7f89ec0fd780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 -- 172.21.15.69:0/3149856279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec106990 msgr2=0x7f89ec0fd780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec106990 0x7f89ec0fd780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 -- 172.21.15.69:0/3149856279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1013a0 msgr2=0x7f89ec0fd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1013a0 0x7f89ec0fd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 -- 172.21.15.69:0/3149856279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89ec1a6f90 con 0x7f89ec1009e0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89ebfff700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f89dc006010 tx=0x7f89dc0039a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:45.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc023020 con 0x7f89ec1009e0 2024-05-09T09:17:45.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89ec1a7220 con 0x7f89ec1009e0 2024-05-09T09:17:45.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89dc003bd0 con 0x7f89ec1009e0 2024-05-09T09:17:45.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89dc016680 con 0x7f89ec1009e0 2024-05-09T09:17:45.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.884+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89ec1a76d0 con 0x7f89ec1009e0 2024-05-09T09:17:45.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.885+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f89ec1020d0 con 0x7f89ec1009e0 2024-05-09T09:17:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.887+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f89dc026020 con 0x7f89ec1009e0 2024-05-09T09:17:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.887+0000 7f89e97fa700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 0x7f89d40a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.887+0000 7f89eb7fe700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 0x7f89d40a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.887+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f89dc0d0f50 con 0x7f89ec1009e0 2024-05-09T09:17:45.887 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.888+0000 7f89eb7fe700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 0x7f89d40a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89ec105190 tx=0x7f89e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:45.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:45.891+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f89dc02d020 con 0x7f89ec1009e0 2024-05-09T09:17:46.100 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.101+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) -- 0x7f89ec02db80 con 0x7f89ec1009e0 2024-05-09T09:17:46.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.102+0000 7f89e97fa700 1 -- 172.21.15.69:0/3149856279 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f89dc09daa0 con 0x7f89ec1009e0 2024-05-09T09:17:46.101 INFO:teuthology.orchestra.run.smithi069.stdout:141733920832 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 msgr2=0x7f89d40a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 0x7f89d40a51f0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f89ec105190 tx=0x7f89e000b410 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 msgr2=0x7f89ec0fcd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f89dc006010 tx=0x7f89dc0039a0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 shutdown_connections 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f89ec106990 0x7f89ec0fd780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f89ec1009e0 0x7f89ec0fcd00 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89d40a2d30 0x7f89d40a51f0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 --2- 172.21.15.69:0/3149856279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f89ec1013a0 0x7f89ec0fd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 >> 172.21.15.69:0/3149856279 conn(0x7f89ec077890 msgr2=0x7f89ec0ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:46.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 shutdown_connections 2024-05-09T09:17:46.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.104+0000 7f89f2487700 1 -- 172.21.15.69:0/3149856279 wait complete. 2024-05-09T09:17:46.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:46 smithi069 ceph-mon[30123]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:46.441 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3149856279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T09:17:46.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 -- 172.21.15.69:0/3576069340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c1083a0 msgr2=0x7f6c1c108780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 --2- 172.21.15.69:0/3576069340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c108780 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f6c10009990 tx=0x7f6c10009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 -- 172.21.15.69:0/3576069340 shutdown_connections 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 --2- 172.21.15.69:0/3576069340 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 0x7f6c1c10f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 --2- 172.21.15.69:0/3576069340 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c108780 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.544+0000 7f6c22f49700 1 --2- 172.21.15.69:0/3576069340 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c06aed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- 172.21.15.69:0/3576069340 >> 172.21.15.69:0/3576069340 conn(0x7f6c1c0781f0 msgr2=0x7f6c1c078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- 172.21.15.69:0/3576069340 shutdown_connections 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- 172.21.15.69:0/3576069340 wait complete. 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 Processor -- start 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- start start 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 0x7f6c1c19ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c112660 con 0x7f6c1c06aa70 2024-05-09T09:17:46.546 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c1124e0 con 0x7f6c1c06b410 2024-05-09T09:17:46.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.545+0000 7f6c22f49700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6c1c1127e0 con 0x7f6c1c1083a0 2024-05-09T09:17:46.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:46.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50856/0 (socket says 172.21.15.69:50856) 2024-05-09T09:17:46.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 -- 172.21.15.69:0/4155574871 learned_addr learned my addr 172.21.15.69:0/4155574871 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:46.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c214e6700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:46.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 -- 172.21.15.69:0/4155574871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 msgr2=0x7f6c1c198960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c1bfff700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 0x7f6c1c19ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:46.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c198960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 -- 172.21.15.69:0/4155574871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 msgr2=0x7f6c1c19ece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 0x7f6c1c19ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 -- 172.21.15.69:0/4155574871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6c1c1990c0 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c214e6700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c20ce5700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000bff0 tx=0x7f6c10005640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c10027070 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6c10004150 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6c1000f460 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.546+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6c1c199350 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.547+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6c1c1a5670 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.547+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6c1c069010 con 0x7f6c1c06aa70 2024-05-09T09:17:46.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.548+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6c1002a070 con 0x7f6c1c06aa70 2024-05-09T09:17:46.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.549+0000 7f6c19ffb700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 0x7f6c0c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:46.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.549+0000 7f6c1bfff700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 0x7f6c0c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:46.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.549+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f6c100d0b50 con 0x7f6c1c06aa70 2024-05-09T09:17:46.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.550+0000 7f6c1bfff700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 0x7f6c0c0a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c078860 tx=0x7f6c08005cf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:46.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.553+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6c10031020 con 0x7f6c1c06aa70 2024-05-09T09:17:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:46 smithi163 ceph-mon[34354]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3149856279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T09:17:46.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:46 smithi103 ceph-mon[34421]: pgmap v329: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s; 1 B/s, 0 keys/s, 0 objects/s recovering 2024-05-09T09:17:46.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3149856279' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-05-09T09:17:46.753 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.753+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) -- 0x7f6c1c02db80 con 0x7f6c1c06aa70 2024-05-09T09:17:46.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.754+0000 7f6c19ffb700 1 -- 172.21.15.69:0/4155574871 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f6c10013070 con 0x7f6c1c06aa70 2024-05-09T09:17:46.754 INFO:teuthology.orchestra.run.smithi069.stdout:287762808870 2024-05-09T09:17:46.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 msgr2=0x7f6c0c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 0x7f6c0c0a51f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f6c1c078860 tx=0x7f6c08005cf0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 msgr2=0x7f6c1c19e7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f6c1000bff0 tx=0x7f6c10005640 comp rx=0 tx=0).stop 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 shutdown_connections 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6c1c1083a0 0x7f6c1c198960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6c1c06aa70 0x7f6c1c19e7a0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.756+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6c0c0a2d30 0x7f6c0c0a51f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.757+0000 7f6c22f49700 1 --2- 172.21.15.69:0/4155574871 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6c1c06b410 0x7f6c1c19ece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:46.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.757+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 >> 172.21.15.69:0/4155574871 conn(0x7f6c1c0781f0 msgr2=0x7f6c1c0fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:46.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.757+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 shutdown_connections 2024-05-09T09:17:46.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:46.757+0000 7f6c22f49700 1 -- 172.21.15.69:0/4155574871 wait complete. 2024-05-09T09:17:47.140 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383703 got 343597383707 for osd.11 2024-05-09T09:17:47.141 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:47.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4155574871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T09:17:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:47 smithi069 ceph-mon[30123]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:17:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4155574871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T09:17:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:47 smithi163 ceph-mon[34354]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:17:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4155574871' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-05-09T09:17:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:47 smithi103 ceph-mon[34421]: pgmap v330: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:17:49.412 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920832 for osd.4 2024-05-09T09:17:49.412 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:50.003 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808867 got 287762808870 for osd.9 2024-05-09T09:17:50.004 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:50.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:50 smithi069 ceph-mon[30123]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:50.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3232903700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T09:17:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:50 smithi163 ceph-mon[34354]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3232903700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T09:17:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:50 smithi103 ceph-mon[34421]: pgmap v331: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3232903700' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 -- 172.21.15.69:0/821263424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 msgr2=0x7fa728108770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 --2- 172.21.15.69:0/821263424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728108770 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fa718009990 tx=0x7fa718009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 -- 172.21.15.69:0/821263424 shutdown_connections 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 --2- 172.21.15.69:0/821263424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72810f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 --2- 172.21.15.69:0/821263424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728108770 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 --2- 172.21.15.69:0/821263424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 0x7fa72806ad20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 -- 172.21.15.69:0/821263424 >> 172.21.15.69:0/821263424 conn(0x7fa728078190 msgr2=0x7fa7280785a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:50.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 -- 172.21.15.69:0/821263424 shutdown_connections 2024-05-09T09:17:50.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.626+0000 7fa72d23f700 1 -- 172.21.15.69:0/821263424 wait complete. 2024-05-09T09:17:50.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.627+0000 7fa72d23f700 1 Processor -- start 2024-05-09T09:17:50.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.627+0000 7fa72d23f700 1 -- start start 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.627+0000 7fa72d23f700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 0x7fa72819e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.627+0000 7fa72d23f700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72819ec10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.627+0000 7fa72d23f700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72d23f700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7281125f0 con 0x7fa728108390 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72d23f700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa728112470 con 0x7fa72806a8c0 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72d23f700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa728112770 con 0x7fa72806b260 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36652/0 (socket says 172.21.15.69:36652) 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 -- 172.21.15.69:0/1641614570 learned_addr learned my addr 172.21.15.69:0/1641614570 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72659c700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72819ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa726d9d700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 0x7fa72819e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 -- 172.21.15.69:0/1641614570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 msgr2=0x7fa72819ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72819ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 -- 172.21.15.69:0/1641614570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 msgr2=0x7fa72819e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 0x7fa72819e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 -- 172.21.15.69:0/1641614570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa728198ff0 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72659c700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa72759e700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa720007ee0 tx=0x7fa72000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.628+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa72000dbd0 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.629+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa72000dd70 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.629+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7281992e0 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.629+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa720005a60 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.629+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7281a55a0 con 0x7fa728108390 2024-05-09T09:17:50.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.630+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa728069010 con 0x7fa728108390 2024-05-09T09:17:50.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.631+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fa72001a020 con 0x7fa728108390 2024-05-09T09:17:50.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.631+0000 7fa70ffff700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 0x7fa7100a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.631+0000 7fa726d9d700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 0x7fa7100a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.630 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.632+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fa7200ceba0 con 0x7fa728108390 2024-05-09T09:17:50.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.632+0000 7fa726d9d700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 0x7fa7100a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa718006010 tx=0x7fa71800b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:50.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.636+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa720010070 con 0x7fa728108390 2024-05-09T09:17:50.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.107+0000 7f41f320e700 1 -- 172.21.15.69:0/1755125209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 msgr2=0x7f41ec1029b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.107+0000 7f41f320e700 1 --2- 172.21.15.69:0/1755125209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec1029b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f41e0009990 tx=0x7f41e0009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- 172.21.15.69:0/1755125209 shutdown_connections 2024-05-09T09:17:50.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- 172.21.15.69:0/1755125209 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec107660 0x7f41ec10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- 172.21.15.69:0/1755125209 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec1029b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- 172.21.15.69:0/1755125209 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec102f80 0x7f41ec106f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- 172.21.15.69:0/1755125209 >> 172.21.15.69:0/1755125209 conn(0x7f41ec079480 msgr2=0x7f41ec079890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- 172.21.15.69:0/1755125209 shutdown_connections 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- 172.21.15.69:0/1755125209 wait complete. 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 Processor -- start 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- start start 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 0x7f41ec19ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107660 0x7f41ec1989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec1126a0 con 0x7f41ec1025d0 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec112520 con 0x7f41ec102f80 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.108+0000 7f41f320e700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f41ec112820 con 0x7f41ec107660 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36632/0 (socket says 172.21.15.69:36632) 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 -- 172.21.15.69:0/3232903700 learned_addr learned my addr 172.21.15.69:0/3232903700 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41ebfff700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 0x7f41ec19ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f17ab700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107660 0x7f41ec1989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stdout:77309411403 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 -- 172.21.15.69:0/3232903700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107660 msgr2=0x7f41ec1989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107660 0x7f41ec1989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 -- 172.21.15.69:0/3232903700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 msgr2=0x7f41ec19ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 0x7f41ec19ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 -- 172.21.15.69:0/3232903700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f41ec199120 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41ebfff700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 0x7f41ec19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f0faa700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f41e0000c00 tx=0x7f41e0003730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e0027070 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41e0004190 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f41ec1993b0 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41e000f460 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.109+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f41ec1a56d0 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.110+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f41ec103a60 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.112+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f41e002a070 con 0x7f41ec1025d0 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.113+0000 7f41e9ffb700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 0x7f41dc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:50.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.113+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f41e00d08a0 con 0x7f41ec1025d0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.113+0000 7f41ebfff700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 0x7f41dc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.113+0000 7f41ebfff700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 0x7f41dc0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1070e0 tx=0x7f41d8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.116+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41e0013070 con 0x7f41ec1025d0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.320+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) -- 0x7f41ec02db80 con 0x7f41ec1025d0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.320+0000 7f41e9ffb700 1 -- 172.21.15.69:0/3232903700 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f41e009d3f0 con 0x7f41ec1025d0 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 msgr2=0x7f41dc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 0x7f41dc0a5170 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f41ec1070e0 tx=0x7f41d8008040 comp rx=0 tx=0).stop 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 msgr2=0x7f41ec19e800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f41e0000c00 tx=0x7f41e0003730 comp rx=0 tx=0).stop 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 shutdown_connections 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f41ec107660 0x7f41ec1989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f41ec1025d0 0x7f41ec19e800 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41dc0a2cb0 0x7f41dc0a5170 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 --2- 172.21.15.69:0/3232903700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f41ec102f80 0x7f41ec19ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 >> 172.21.15.69:0/3232903700 conn(0x7f41ec079480 msgr2=0x7f41ec101b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 shutdown_connections 2024-05-09T09:17:50.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:49.323+0000 7f41f320e700 1 -- 172.21.15.69:0/3232903700 wait complete. 2024-05-09T09:17:50.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.840+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) -- 0x7fa72802db80 con 0x7fa728108390 2024-05-09T09:17:50.839 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.840+0000 7fa70ffff700 1 -- 172.21.15.69:0/1641614570 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fa72009b660 con 0x7fa728108390 2024-05-09T09:17:50.839 INFO:teuthology.orchestra.run.smithi069.stdout:253403070508 2024-05-09T09:17:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.842+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 msgr2=0x7fa7100a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.841 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.842+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 0x7fa7100a5170 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fa718006010 tx=0x7fa71800b540 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.842+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 msgr2=0x7fa728198890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.842+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa720007ee0 tx=0x7fa72000d3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 shutdown_connections 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa728108390 0x7fa728198890 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa7100a2cb0 0x7fa7100a5170 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa72806a8c0 0x7fa72819e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 --2- 172.21.15.69:0/1641614570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa72806b260 0x7fa72819ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 >> 172.21.15.69:0/1641614570 conn(0x7fa728078190 msgr2=0x7fa7280ffe30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 shutdown_connections 2024-05-09T09:17:50.842 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:50.843+0000 7fa72d23f700 1 -- 172.21.15.69:0/1641614570 wait complete. 2024-05-09T09:17:50.940 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:51.339 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411398 got 77309411403 for osd.2 2024-05-09T09:17:51.339 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:51.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1641614570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T09:17:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1641614570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T09:17:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1641614570' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-05-09T09:17:52.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:52 smithi069 ceph-mon[30123]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:17:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 -- 172.21.15.69:0/474112622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 msgr2=0x7f5a2010e1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 --2- 172.21.15.69:0/474112622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a2010e1b0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f5a1800b0d0 tx=0x7f5a1800b3e0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 -- 172.21.15.69:0/474112622 shutdown_connections 2024-05-09T09:17:52.486 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 --2- 172.21.15.69:0/474112622 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a2010e1b0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 --2- 172.21.15.69:0/474112622 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20101c80 0x7f5a20102060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 --2- 172.21.15.69:0/474112622 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20102630 0x7f5a20105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 -- 172.21.15.69:0/474112622 >> 172.21.15.69:0/474112622 conn(0x7f5a20077890 msgr2=0x7f5a200fec40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 -- 172.21.15.69:0/474112622 shutdown_connections 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.486+0000 7f5a2756c700 1 -- 172.21.15.69:0/474112622 wait complete. 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 Processor -- start 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 -- start start 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20101c80 0x7f5a2019d6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 0x7f5a2019dc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5a20111660 con 0x7f5a201065a0 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5a201114e0 con 0x7f5a20102630 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a2756c700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5a201117e0 con 0x7f5a20101c80 2024-05-09T09:17:52.487 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36660/0 (socket says 172.21.15.69:36660) 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25308700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20101c80 0x7f5a2019d6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 -- 172.21.15.69:0/1508809341 learned_addr learned my addr 172.21.15.69:0/1508809341 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a24b07700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 0x7f5a2019dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 -- 172.21.15.69:0/1508809341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20101c80 msgr2=0x7f5a2019d6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20101c80 0x7f5a2019d6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 -- 172.21.15.69:0/1508809341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 msgr2=0x7f5a2019dc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 0x7f5a2019dc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a25b09700 1 -- 172.21.15.69:0/1508809341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5a20198060 con 0x7f5a201065a0 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.487+0000 7f5a24b07700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 0x7f5a2019dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:17:52.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a25b09700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5a18006070 tx=0x7f5a18004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18021070 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5a201982f0 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5a18007b80 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5a18005330 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.488+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5a201a45c0 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.489+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5a20198de0 con 0x7f5a201065a0 2024-05-09T09:17:52.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.490+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f5a18024070 con 0x7f5a201065a0 2024-05-09T09:17:52.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.491+0000 7f5a167fc700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 0x7f5a0c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.490 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.491+0000 7f5a25308700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 0x7f5a0c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.491+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f5a1800f020 con 0x7f5a201065a0 2024-05-09T09:17:52.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.492+0000 7f5a25308700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 0x7f5a0c0a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a201001d0 tx=0x7f5a1c009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:52.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.495+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5a1802b020 con 0x7f5a201065a0 2024-05-09T09:17:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:52 smithi163 ceph-mon[34354]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:17:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:52 smithi103 ceph-mon[34421]: pgmap v332: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 -- 172.21.15.69:0/4115720625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 msgr2=0x7f705410f1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 --2- 172.21.15.69:0/4115720625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f705410f1e0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f704c00b0a0 tx=0x7f704c00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 -- 172.21.15.69:0/4115720625 shutdown_connections 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 --2- 172.21.15.69:0/4115720625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f705410f1e0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 --2- 172.21.15.69:0/4115720625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054108780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 --2- 172.21.15.69:0/4115720625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 0x7f705406a490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 -- 172.21.15.69:0/4115720625 >> 172.21.15.69:0/4115720625 conn(0x7f70540781d0 msgr2=0x7f70540785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 -- 172.21.15.69:0/4115720625 shutdown_connections 2024-05-09T09:17:52.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 -- 172.21.15.69:0/4115720625 wait complete. 2024-05-09T09:17:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.704+0000 7f70592c1700 1 Processor -- start 2024-05-09T09:17:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 -- start start 2024-05-09T09:17:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 0x7f70541012f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.705 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054101d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7054112620 con 0x7f705406a9d0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70541124a0 con 0x7f705406a030 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70592c1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70541127a0 con 0x7f70541083a0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36688/0 (socket says 172.21.15.69:36688) 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 -- 172.21.15.69:0/1253676681 learned_addr learned my addr 172.21.15.69:0/1253676681 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f7052ffd700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 0x7f70541012f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70537fe700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054101d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 -- 172.21.15.69:0/1253676681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 msgr2=0x7f7054101d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054101d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 -- 172.21.15.69:0/1253676681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 msgr2=0x7f70541012f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 0x7f70541012f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70527fc700 1 -- 172.21.15.69:0/1253676681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70541a7010 con 0x7f705406a9d0 2024-05-09T09:17:52.706 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.705+0000 7f70537fe700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f70527fc700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f704400d950 tx=0x7f704400dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7044016070 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7044009e20 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70541a7300 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f704400f620 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.706+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70541a7890 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.707+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7054069010 con 0x7f705406a9d0 2024-05-09T09:17:52.707 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.709+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f7044019070 con 0x7f705406a9d0 2024-05-09T09:17:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.709+0000 7f703bfff700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 0x7f703c0a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.710+0000 7f7052ffd700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 0x7f703c0a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:52.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.710+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f70440cf060 con 0x7f705406a9d0 2024-05-09T09:17:52.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.710+0000 7f7052ffd700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 0x7f703c0a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7048009be0 tx=0x7f7048009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:52.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.714+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f704409bb20 con 0x7f705406a9d0 2024-05-09T09:17:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.719+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) -- 0x7f5a20102a10 con 0x7f5a201065a0 2024-05-09T09:17:52.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.719+0000 7f5a167fc700 1 -- 172.21.15.69:0/1508809341 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f5a1809da20 con 0x7f5a201065a0 2024-05-09T09:17:52.719 INFO:teuthology.orchestra.run.smithi069.stdout:107374182470 2024-05-09T09:17:52.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 msgr2=0x7f5a0c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 0x7f5a0c0a5170 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f5a201001d0 tx=0x7f5a1c009190 comp rx=0 tx=0).stop 2024-05-09T09:17:52.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 msgr2=0x7f5a20197900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f5a18006070 tx=0x7f5a18004d00 comp rx=0 tx=0).stop 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 shutdown_connections 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5a201065a0 0x7f5a20197900 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5a20101c80 0x7f5a2019d6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5a0c0a2cb0 0x7f5a0c0a5170 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 --2- 172.21.15.69:0/1508809341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5a20102630 0x7f5a2019dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.721+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 >> 172.21.15.69:0/1508809341 conn(0x7f5a20077890 msgr2=0x7f5a20104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.722+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 shutdown_connections 2024-05-09T09:17:52.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.722+0000 7f5a2756c700 1 -- 172.21.15.69:0/1508809341 wait complete. 2024-05-09T09:17:52.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.923+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) -- 0x7f705402db80 con 0x7f705406a9d0 2024-05-09T09:17:52.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.923+0000 7f703bfff700 1 -- 172.21.15.69:0/1253676681 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f704409b420 con 0x7f705406a9d0 2024-05-09T09:17:52.923 INFO:teuthology.orchestra.run.smithi069.stdout:219043332146 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 msgr2=0x7f703c0a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 0x7f703c0a51f0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f7048009be0 tx=0x7f7048009260 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 msgr2=0x7f7054101830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f704400d950 tx=0x7f704400dc60 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 shutdown_connections 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70541083a0 0x7f7054101d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f703c0a2d30 0x7f703c0a51f0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f705406a030 0x7f70541012f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 --2- 172.21.15.69:0/1253676681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f705406a9d0 0x7f7054101830 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 >> 172.21.15.69:0/1253676681 conn(0x7f70540781d0 msgr2=0x7f70540ffda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 shutdown_connections 2024-05-09T09:17:52.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:52.926+0000 7f70592c1700 1 -- 172.21.15.69:0/1253676681 wait complete. 2024-05-09T09:17:53.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1508809341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T09:17:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1253676681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T09:17:53.545 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070502 got 253403070508 for osd.8 2024-05-09T09:17:53.545 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1508809341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T09:17:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1253676681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T09:17:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1508809341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-05-09T09:17:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1253676681' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-05-09T09:17:54.463 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:54 smithi069 ceph-mon[30123]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:17:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:54 smithi163 ceph-mon[34354]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:17:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:54 smithi103 ceph-mon[34421]: pgmap v333: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 -- 172.21.15.69:0/1434588205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c107290 msgr2=0x7fed4c107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 --2- 172.21.15.69:0/1434588205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c107290 0x7fed4c107670 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7fed40009990 tx=0x7fed40009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 -- 172.21.15.69:0/1434588205 shutdown_connections 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 --2- 172.21.15.69:0/1434588205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c10e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 --2- 172.21.15.69:0/1434588205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c107290 0x7fed4c107670 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 --2- 172.21.15.69:0/1434588205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed4c101220 0x7fed4c1016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 -- 172.21.15.69:0/1434588205 >> 172.21.15.69:0/1434588205 conn(0x7fed4c0781d0 msgr2=0x7fed4c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:54.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 -- 172.21.15.69:0/1434588205 shutdown_connections 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.567+0000 7fed52f56700 1 -- 172.21.15.69:0/1434588205 wait complete. 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 Processor -- start 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 -- start start 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c19ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed4c107290 0x7fed4c198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111500 con 0x7fed4c101220 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111380 con 0x7fed4c107290 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed52f56700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fed4c111680 con 0x7fed4c101be0 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36706/0 (socket says 172.21.15.69:36706) 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 -- 172.21.15.69:0/119768342 learned_addr learned my addr 172.21.15.69:0/119768342 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:54.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed4bfff700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c19ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 -- 172.21.15.69:0/119768342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 msgr2=0x7fed4c19ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c19ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 -- 172.21.15.69:0/119768342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed4c107290 msgr2=0x7fed4c198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed4c107290 0x7fed4c198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed50cf2700 1 -- 172.21.15.69:0/119768342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fed4c1990b0 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.568+0000 7fed4bfff700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed50cf2700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed40027070 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fed40004050 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fed4c199340 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fed4000f460 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.569+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fed4c1a5610 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.570+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fed4c102940 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.571+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7fed4002a070 con 0x7fed4c101220 2024-05-09T09:17:54.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.572+0000 7fed49ffb700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 0x7fed3c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.572+0000 7fed4bfff700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 0x7fed3c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.572+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fed400d0480 con 0x7fed4c101220 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.572+0000 7fed4bfff700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 0x7fed3c0a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fed38005fd0 tx=0x7fed38005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.576+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fed4005e740 con 0x7fed4c101220 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.791+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) -- 0x7fed4c1a57e0 con 0x7fed4c101220 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.791+0000 7fed49ffb700 1 -- 172.21.15.69:0/119768342 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7fed40013070 con 0x7fed4c101220 2024-05-09T09:17:54.894 INFO:teuthology.orchestra.run.smithi069.stdout:197568495671 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 msgr2=0x7fed3c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 0x7fed3c0a5170 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fed38005fd0 tx=0x7fed38005ee0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 msgr2=0x7fed4c19e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fed40000c00 tx=0x7fed400047b0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 shutdown_connections 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fed3c0a2cb0 0x7fed3c0a5170 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fed4c107290 0x7fed4c198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fed4c101220 0x7fed4c19e740 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 --2- 172.21.15.69:0/119768342 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fed4c101be0 0x7fed4c19ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.794+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 >> 172.21.15.69:0/119768342 conn(0x7fed4c0781d0 msgr2=0x7fed4c105600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.795+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 shutdown_connections 2024-05-09T09:17:54.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:54.795+0000 7fed52f56700 1 -- 172.21.15.69:0/119768342 wait complete. 2024-05-09T09:17:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/119768342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T09:17:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:55 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/119768342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T09:17:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:55 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/119768342' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-05-09T09:17:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:55.913 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:17:55.938 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332140 got 219043332146 for osd.7 2024-05-09T09:17:55.939 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:56.347 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182463 got 107374182470 for osd.3 2024-05-09T09:17:56.347 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:56.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:56 smithi069 ceph-mon[30123]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:56.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:17:56.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:17:56.461 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:56 smithi163 ceph-mon[34354]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:17:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:17:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:56 smithi103 ceph-mon[34421]: pgmap v334: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:17:56.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:17:56.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:17:56.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:17:56.994 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495671 for osd.6 2024-05-09T09:17:56.994 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 -- 172.21.15.69:0/4114810145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c06aa20 msgr2=0x7f6f0c06ae80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 --2- 172.21.15.69:0/4114810145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c06ae80 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6f00009990 tx=0x7f6f00009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 -- 172.21.15.69:0/4114810145 shutdown_connections 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 --2- 172.21.15.69:0/4114810145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c10f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 --2- 172.21.15.69:0/4114810145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c1087b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 --2- 172.21.15.69:0/4114810145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c06ae80 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 -- 172.21.15.69:0/4114810145 >> 172.21.15.69:0/4114810145 conn(0x7f6f0c078260 msgr2=0x7f6f0c078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 -- 172.21.15.69:0/4114810145 shutdown_connections 2024-05-09T09:17:57.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.308+0000 7f6f1092a700 1 -- 172.21.15.69:0/4114810145 wait complete. 2024-05-09T09:17:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 Processor -- start 2024-05-09T09:17:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 -- start start 2024-05-09T09:17:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c19ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c19ef90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:57.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c112760 con 0x7f6f0c1083d0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1125e0 con 0x7f6f0c06aa20 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f1092a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f0c1128e0 con 0x7f6f0c06b3c0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36736/0 (socket says 172.21.15.69:36736) 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 -- 172.21.15.69:0/680435619 learned_addr learned my addr 172.21.15.69:0/680435619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0a59c700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c19ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f09d9b700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c19ef90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 -- 172.21.15.69:0/680435619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 msgr2=0x7f6f0c19ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c19ef90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 -- 172.21.15.69:0/680435619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06aa20 msgr2=0x7f6f0c19ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c19ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f0ad9d700 1 -- 172.21.15.69:0/680435619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f0c199400 con 0x7f6f0c1083d0 2024-05-09T09:17:57.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.309+0000 7f6f09d9b700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6f0ad9d700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b970 tx=0x7f6f0400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04019070 con 0x7f6f0c1083d0 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f04014440 con 0x7f6f0c1083d0 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f0c1996f0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f040075d0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.310+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f0c1a59d0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.311+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6f0c103a80 con 0x7f6f0c1083d0 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.312+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f6f0401c070 con 0x7f6f0c1083d0 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.313+0000 7f6efb7fe700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 0x7f6ef40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.313+0000 7f6f0a59c700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 0x7f6ef40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.313+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f6f040cecf0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.313+0000 7f6f0a59c700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 0x7f6ef40a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006fd0 tx=0x7f6efc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:57.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.316+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6f0409b8c0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.515+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) -- 0x7f6f0c199fe0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.515+0000 7f6efb7fe700 1 -- 172.21.15.69:0/680435619 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) ==== 74+0+12 (secure 0 0 0) 0x7f6f0409b1c0 con 0x7f6f0c1083d0 2024-05-09T09:17:57.515 INFO:teuthology.orchestra.run.smithi069.stdout:55834574928 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 msgr2=0x7f6ef40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 0x7f6ef40a5170 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f6efc006fd0 tx=0x7f6efc008040 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 msgr2=0x7f6f0c198b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400b970 tx=0x7f6f0400bd30 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 shutdown_connections 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f0c1083d0 0x7f6f0c198b20 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6ef40a2cb0 0x7f6ef40a5170 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f0c06aa20 0x7f6f0c19ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 --2- 172.21.15.69:0/680435619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f0c06b3c0 0x7f6f0c19ef90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 >> 172.21.15.69:0/680435619 conn(0x7f6f0c078260 msgr2=0x7f6f0c0fff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 shutdown_connections 2024-05-09T09:17:57.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:57.518+0000 7f6f1092a700 1 -- 172.21.15.69:0/680435619 wait complete. 2024-05-09T09:17:58.429 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:58 smithi069 ceph-mon[30123]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:17:58.429 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:58 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/680435619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 -- 172.21.15.69:0/1440480817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418100ce0 msgr2=0x7f2418101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 --2- 172.21.15.69:0/1440480817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418100ce0 0x7f2418101160 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f2400009990 tx=0x7f2400009ca0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 -- 172.21.15.69:0/1440480817 shutdown_connections 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 --2- 172.21.15.69:0/1440480817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 0x7f241810d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 --2- 172.21.15.69:0/1440480817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418106a90 0x7f2418106e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 --2- 172.21.15.69:0/1440480817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418100ce0 0x7f2418101160 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 -- 172.21.15.69:0/1440480817 >> 172.21.15.69:0/1440480817 conn(0x7f2418076540 msgr2=0x7f2418076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:58.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 -- 172.21.15.69:0/1440480817 shutdown_connections 2024-05-09T09:17:58.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.533+0000 7f241cea4700 1 -- 172.21.15.69:0/1440480817 wait complete. 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 Processor -- start 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 -- start start 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 0x7f241819d630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 0x7f241819db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2418110d60 con 0x7f2418106a90 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2418110be0 con 0x7f24181016a0 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241cea4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2418110ee0 con 0x7f2418100ce0 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51814/0 (socket says 172.21.15.69:51814) 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 -- 172.21.15.69:0/2905072955 learned_addr learned my addr 172.21.15.69:0/2905072955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f240ffff700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 0x7f241819db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:58.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241659c700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 0x7f241819d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 -- 172.21.15.69:0/2905072955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 msgr2=0x7f241819d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 0x7f241819d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 -- 172.21.15.69:0/2905072955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 msgr2=0x7f241819db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 0x7f241819db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f2416d9d700 1 -- 172.21.15.69:0/2905072955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2418197f90 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.534+0000 7f241659c700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 0x7f241819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f2416d9d700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2410007ee0 tx=0x7f241000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f241000dbd0 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2418198280 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f241000dd70 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2410005a60 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.535+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24181a4490 con 0x7f2418106a90 2024-05-09T09:17:58.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.536+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2418102120 con 0x7f2418106a90 2024-05-09T09:17:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.537+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 17) ==== 137884+0+0 (secure 0 0 0) 0x7f241001a020 con 0x7f2418106a90 2024-05-09T09:17:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.537+0000 7f240f7fe700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 0x7f24040a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:17:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.538+0000 7f241659c700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 0x7f24040a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:17:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.538+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f24100cec40 con 0x7f2418106a90 2024-05-09T09:17:58.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.538+0000 7f241659c700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 0x7f24040a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2408007900 tx=0x7f2408008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:17:58.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.542+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f24100d5080 con 0x7f2418106a90 2024-05-09T09:17:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:58 smithi163 ceph-mon[34354]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:17:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:58 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/680435619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T09:17:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:58 smithi103 ceph-mon[34421]: pgmap v335: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:17:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:58 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/680435619' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-05-09T09:17:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.738+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) -- 0x7f24181a46c0 con 0x7f2418106a90 2024-05-09T09:17:58.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.739+0000 7f240f7fe700 1 -- 172.21.15.69:0/2905072955 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) ==== 74+0+13 (secure 0 0 0) 0x7f2410010070 con 0x7f2418106a90 2024-05-09T09:17:58.738 INFO:teuthology.orchestra.run.smithi069.stdout:163208757309 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 msgr2=0x7f24040a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 0x7f24040a5170 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2408007900 tx=0x7f2408008040 comp rx=0 tx=0).stop 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 msgr2=0x7f24181977b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f2410007ee0 tx=0x7f241000d3b0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 shutdown_connections 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2418106a90 0x7f24181977b0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2418100ce0 0x7f241819d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24040a2cb0 0x7f24040a5170 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 --2- 172.21.15.69:0/2905072955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24181016a0 0x7f241819db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:17:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 >> 172.21.15.69:0/2905072955 conn(0x7f2418076540 msgr2=0x7f24181044f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:17:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 shutdown_connections 2024-05-09T09:17:58.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:17:58.741+0000 7f241cea4700 1 -- 172.21.15.69:0/2905072955 wait complete. 2024-05-09T09:17:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:17:59 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2905072955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T09:17:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:17:59 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2905072955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T09:17:59.588 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574923 got 55834574928 for osd.1 2024-05-09T09:17:59.588 DEBUG:teuthology.parallel:result is None 2024-05-09T09:17:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:17:59 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2905072955' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-05-09T09:18:00.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:00 smithi069 ceph-mon[30123]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:00 smithi069 ceph-mon[30123]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T09:18:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:00 smithi163 ceph-mon[34354]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:00 smithi163 ceph-mon[34354]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T09:18:00.610 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757304 got 163208757309 for osd.5 2024-05-09T09:18:00.610 DEBUG:teuthology.parallel:result is None 2024-05-09T09:18:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:00 smithi103 ceph-mon[34421]: pgmap v336: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:00 smithi103 ceph-mon[34421]: mgrmap e18: x(active, since 8m), standbys: z, y 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 -- 172.21.15.69:0/3779243905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 msgr2=0x7f2d2c1076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 --2- 172.21.15.69:0/3779243905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1076a0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c009990 tx=0x7f2d1c009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 -- 172.21.15.69:0/3779243905 shutdown_connections 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 --2- 172.21.15.69:0/3779243905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 0x7f2d2c10e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 --2- 172.21.15.69:0/3779243905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1076a0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 --2- 172.21.15.69:0/3779243905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 0x7f2d2c1016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 -- 172.21.15.69:0/3779243905 >> 172.21.15.69:0/3779243905 conn(0x7f2d2c078260 msgr2=0x7f2d2c078670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 -- 172.21.15.69:0/3779243905 shutdown_connections 2024-05-09T09:18:01.166 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 -- 172.21.15.69:0/3779243905 wait complete. 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.167+0000 7f2d31599700 1 Processor -- start 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 -- start start 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 0x7f2d2c199980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 0x7f2d2c199ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c111560 con 0x7f2d2c1072c0 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1113e0 con 0x7f2d2c101c10 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d31599700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2d2c1116e0 con 0x7f2d2c101250 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51834/0 (socket says 172.21.15.69:51834) 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 -- 172.21.15.69:0/2581067667 learned_addr learned my addr 172.21.15.69:0/2581067667 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:01.169 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2a7fc700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 0x7f2d2c199ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2affd700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 0x7f2d2c199980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 -- 172.21.15.69:0/2581067667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 msgr2=0x7f2d2c199980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 0x7f2d2c199980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 -- 172.21.15.69:0/2581067667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 msgr2=0x7f2d2c199ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 0x7f2d2c199ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.168+0000 7f2d2b7fe700 1 -- 172.21.15.69:0/2581067667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2d2c1a4f40 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d2b7fe700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007ee0 tx=0x7f2d2400d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24004bb0 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2d2c1a51d0 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2d24004d50 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2d24005960 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.169+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2d2c1a56e0 con 0x7f2d2c1072c0 2024-05-09T09:18:01.170 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.170+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2d2c102970 con 0x7f2d2c1072c0 2024-05-09T09:18:01.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.171+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2d2401a020 con 0x7f2d2c1072c0 2024-05-09T09:18:01.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.172+0000 7f2d13fff700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 0x7f2d140a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:01.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.172+0000 7f2d2affd700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 0x7f2d140a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.172+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f2d240cea30 con 0x7f2d2c1072c0 2024-05-09T09:18:01.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.172+0000 7f2d2affd700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 0x7f2d140a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b5c0 tx=0x7f2d1c005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:01.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.176+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2d24010070 con 0x7f2d2c1072c0 2024-05-09T09:18:01.371 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.372+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) -- 0x7f2d2c02db80 con 0x7f2d2c1072c0 2024-05-09T09:18:01.372 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.373+0000 7f2d13fff700 1 -- 172.21.15.69:0/2581067667 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) ==== 75+0+13 (secure 0 0 0) 0x7f2d2409b650 con 0x7f2d2c1072c0 2024-05-09T09:18:01.372 INFO:teuthology.orchestra.run.smithi069.stdout:309237645348 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 msgr2=0x7f2d140a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 0x7f2d140a50e0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2d1c00b5c0 tx=0x7f2d1c005960 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 msgr2=0x7f2d2c1a4a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f2d24007ee0 tx=0x7f2d2400d3b0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 shutdown_connections 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2d2c1072c0 0x7f2d2c1a4a00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2d2c101250 0x7f2d2c199980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2d140a2c20 0x7f2d140a50e0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 --2- 172.21.15.69:0/2581067667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2d2c101c10 0x7f2d2c199ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:01.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 >> 172.21.15.69:0/2581067667 conn(0x7f2d2c078260 msgr2=0x7f2d2c100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:01.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 shutdown_connections 2024-05-09T09:18:01.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:01.375+0000 7f2d31599700 1 -- 172.21.15.69:0/2581067667 wait complete. 2024-05-09T09:18:02.020 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645348 for osd.10 2024-05-09T09:18:02.020 DEBUG:teuthology.parallel:result is None 2024-05-09T09:18:02.021 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-05-09T09:18:02.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T09:18:02.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:02 smithi069 ceph-mon[30123]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-09T09:18:02.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:02 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2581067667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T09:18:02.265 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:02 smithi163 ceph-mon[34354]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-09T09:18:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:02 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2581067667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T09:18:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:02 smithi103 ceph-mon[34421]: pgmap v337: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-05-09T09:18:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:02 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2581067667' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- 172.21.15.69:0/1441494288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 msgr2=0x7f34641077e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 --2- 172.21.15.69:0/1441494288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f34641077e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3460009990 tx=0x7f3460009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- 172.21.15.69:0/1441494288 shutdown_connections 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 --2- 172.21.15.69:0/1441494288 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f346410f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 --2- 172.21.15.69:0/1441494288 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f34641077e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 --2- 172.21.15.69:0/1441494288 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 0x7f3464101160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- 172.21.15.69:0/1441494288 >> 172.21.15.69:0/1441494288 conn(0x7f3464076540 msgr2=0x7f3464076950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- 172.21.15.69:0/1441494288 shutdown_connections 2024-05-09T09:18:03.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- 172.21.15.69:0/1441494288 wait complete. 2024-05-09T09:18:03.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 Processor -- start 2024-05-09T09:18:03.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.899+0000 7f346c5f4700 1 -- start start 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 0x7f3464101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f3464102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3464112660 con 0x7f3464107400 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34641124e0 con 0x7f34641016a0 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34641127e0 con 0x7f3464100ce0 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51856/0 (socket says 172.21.15.69:51856) 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 -- 172.21.15.69:0/2371767944 learned_addr learned my addr 172.21.15.69:0/2371767944 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346a390700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 0x7f3464101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f3469b8f700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f3464102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 -- 172.21.15.69:0/2371767944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 msgr2=0x7f3464101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:03.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 0x7f3464101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 -- 172.21.15.69:0/2371767944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 msgr2=0x7f3464102400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f3464102400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 -- 172.21.15.69:0/2371767944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f34641a7030 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f3469b8f700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f3464102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346ab91700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f345c007ee0 tx=0x7f345c00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c00dbd0 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34641a7320 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f345c00dd70 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.900+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345c005a60 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.901+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34641a78b0 con 0x7f3464107400 2024-05-09T09:18:03.902 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.901+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3464069010 con 0x7f3464107400 2024-05-09T09:18:03.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.903+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f345c01a020 con 0x7f3464107400 2024-05-09T09:18:03.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.903+0000 7f345b7fe700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 0x7f34500a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:03.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.903+0000 7f346a390700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 0x7f34500a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:03.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.903+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f345c0ce9e0 con 0x7f3464107400 2024-05-09T09:18:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.904+0000 7f346a390700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 0x7f34500a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3460006010 tx=0x7f3460005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:03.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:03.908+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f345c010070 con 0x7f3464107400 2024-05-09T09:18:04.098 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.098+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f3464062760 con 0x7f34500a2c20 2024-05-09T09:18:04.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.108+0000 7f345b7fe700 1 -- 172.21.15.69:0/2371767944 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340799 (secure 0 0 0) 0x7f3464062760 con 0x7f34500a2c20 2024-05-09T09:18:04.112 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 msgr2=0x7f34500a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 0x7f34500a50e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f3460006010 tx=0x7f3460005960 comp rx=0 tx=0).stop 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 msgr2=0x7f3464102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f345c007ee0 tx=0x7f345c00d3b0 comp rx=0 tx=0).stop 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 shutdown_connections 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3464107400 0x7f3464102940 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3464100ce0 0x7f3464101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:04.115 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f34500a2c20 0x7f34500a50e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:04.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 --2- 172.21.15.69:0/2371767944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34641016a0 0x7f3464102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:04.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 >> 172.21.15.69:0/2371767944 conn(0x7f3464076540 msgr2=0x7f34640ff7c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:04.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 shutdown_connections 2024-05-09T09:18:04.116 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:04.112+0000 7f346c5f4700 1 -- 172.21.15.69:0/2371767944 wait complete. 2024-05-09T09:18:04.116 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:18:04.424 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:04 smithi069 ceph-mon[30123]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:04.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:04 smithi163 ceph-mon[34354]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:04 smithi103 ceph-mon[34421]: pgmap v338: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:04.770 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":338,"stamp":"2024-05-09T09:18:03.125333+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":559,"num_read_kb":396,"num_write":350,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181771,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":564,"ondisk_log_size":564,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353016,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588808,"statfs":{"total":1151940427776,"available":1151578939392,"internally_reserved":0,"allocated":12443648,"data_stored":7959515,"data_compressed":359364,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":87,"num_read_kb":42,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003643"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201717+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:17:30.201717+0000","last_peered":"2024-05-09T09:17:30.201717+0000","last_clean":"2024-05-09T09:17:30.201717+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:17:30.201717+0000","last_undegraded":"2024-05-09T09:17:30.201717+0000","last_fullsized":"2024-05-09T09:17:30.201717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.118550+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:17:59.118550+0000","last_peered":"2024-05-09T09:17:59.118550+0000","last_clean":"2024-05-09T09:17:59.118550+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:17:59.118550+0000","last_undegraded":"2024-05-09T09:17:59.118550+0000","last_fullsized":"2024-05-09T09:17:59.118550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.138566+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:17:59.138566+0000","last_peered":"2024-05-09T09:17:59.138566+0000","last_clean":"2024-05-09T09:17:59.138566+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:17:59.138566+0000","last_undegraded":"2024-05-09T09:17:59.138566+0000","last_fullsized":"2024-05-09T09:17:59.138566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.118775+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:17:59.118775+0000","last_peered":"2024-05-09T09:17:59.118775+0000","last_clean":"2024-05-09T09:17:59.118775+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:17:59.118775+0000","last_undegraded":"2024-05-09T09:17:59.118775+0000","last_fullsized":"2024-05-09T09:17:59.118775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123283+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:17:59.123283+0000","last_peered":"2024-05-09T09:17:59.123283+0000","last_clean":"2024-05-09T09:17:59.123283+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:17:59.123283+0000","last_undegraded":"2024-05-09T09:17:59.123283+0000","last_fullsized":"2024-05-09T09:17:59.123283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126266+0000","last_change":"2024-05-09T09:17:30.205057+0000","last_active":"2024-05-09T09:17:59.126266+0000","last_peered":"2024-05-09T09:17:59.126266+0000","last_clean":"2024-05-09T09:17:59.126266+0000","last_became_active":"2024-05-09T09:17:30.204736+0000","last_became_peered":"2024-05-09T09:17:30.204736+0000","last_unstale":"2024-05-09T09:17:59.126266+0000","last_undegraded":"2024-05-09T09:17:59.126266+0000","last_fullsized":"2024-05-09T09:17:59.126266+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:00.112812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203716+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:17:30.203716+0000","last_peered":"2024-05-09T09:17:30.203716+0000","last_clean":"2024-05-09T09:17:30.203716+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:17:30.203716+0000","last_undegraded":"2024-05-09T09:17:30.203716+0000","last_fullsized":"2024-05-09T09:17:30.203716+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.138729+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:17:59.138729+0000","last_peered":"2024-05-09T09:17:59.138729+0000","last_clean":"2024-05-09T09:17:59.138729+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:17:59.138729+0000","last_undegraded":"2024-05-09T09:17:59.138729+0000","last_fullsized":"2024-05-09T09:17:59.138729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201778+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:17:30.201778+0000","last_peered":"2024-05-09T09:17:30.201778+0000","last_clean":"2024-05-09T09:17:30.201778+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:17:30.201778+0000","last_undegraded":"2024-05-09T09:17:30.201778+0000","last_fullsized":"2024-05-09T09:17:30.201778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126828+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:17:59.126828+0000","last_peered":"2024-05-09T09:17:59.126828+0000","last_clean":"2024-05-09T09:17:59.126828+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:17:59.126828+0000","last_undegraded":"2024-05-09T09:17:59.126828+0000","last_fullsized":"2024-05-09T09:17:59.126828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.204309+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:17:30.204309+0000","last_peered":"2024-05-09T09:17:30.204309+0000","last_clean":"2024-05-09T09:17:30.204309+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:17:30.204309+0000","last_undegraded":"2024-05-09T09:17:30.204309+0000","last_fullsized":"2024-05-09T09:17:30.204309+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.128543+0000","last_change":"2024-05-09T09:17:30.205228+0000","last_active":"2024-05-09T09:17:59.128543+0000","last_peered":"2024-05-09T09:17:59.128543+0000","last_clean":"2024-05-09T09:17:59.128543+0000","last_became_active":"2024-05-09T09:17:30.204824+0000","last_became_peered":"2024-05-09T09:17:30.204824+0000","last_unstale":"2024-05-09T09:17:59.128543+0000","last_undegraded":"2024-05-09T09:17:59.128543+0000","last_fullsized":"2024-05-09T09:17:59.128543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:28.158394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272698+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:17:30.272698+0000","last_peered":"2024-05-09T09:17:30.272698+0000","last_clean":"2024-05-09T09:17:30.272698+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:17:30.272698+0000","last_undegraded":"2024-05-09T09:17:30.272698+0000","last_fullsized":"2024-05-09T09:17:30.272698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133283+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:17:59.133283+0000","last_peered":"2024-05-09T09:17:59.133283+0000","last_clean":"2024-05-09T09:17:59.133283+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:17:59.133283+0000","last_undegraded":"2024-05-09T09:17:59.133283+0000","last_fullsized":"2024-05-09T09:17:59.133283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134317+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:17:59.134317+0000","last_peered":"2024-05-09T09:17:59.134317+0000","last_clean":"2024-05-09T09:17:59.134317+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:17:59.134317+0000","last_undegraded":"2024-05-09T09:17:59.134317+0000","last_fullsized":"2024-05-09T09:17:59.134317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200323+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:17:30.200323+0000","last_peered":"2024-05-09T09:17:30.200323+0000","last_clean":"2024-05-09T09:17:30.200323+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:17:30.200323+0000","last_undegraded":"2024-05-09T09:17:30.200323+0000","last_fullsized":"2024-05-09T09:17:30.200323+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201448+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:17:30.201448+0000","last_peered":"2024-05-09T09:17:30.201448+0000","last_clean":"2024-05-09T09:17:30.201448+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:17:30.201448+0000","last_undegraded":"2024-05-09T09:17:30.201448+0000","last_fullsized":"2024-05-09T09:17:30.201448+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.128284+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:17:59.128284+0000","last_peered":"2024-05-09T09:17:59.128284+0000","last_clean":"2024-05-09T09:17:59.128284+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:17:59.128284+0000","last_undegraded":"2024-05-09T09:17:59.128284+0000","last_fullsized":"2024-05-09T09:17:59.128284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133825+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:17:59.133825+0000","last_peered":"2024-05-09T09:17:59.133825+0000","last_clean":"2024-05-09T09:17:59.133825+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:17:59.133825+0000","last_undegraded":"2024-05-09T09:17:59.133825+0000","last_fullsized":"2024-05-09T09:17:59.133825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197950+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:17:30.197950+0000","last_peered":"2024-05-09T09:17:30.197950+0000","last_clean":"2024-05-09T09:17:30.197950+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:17:30.197950+0000","last_undegraded":"2024-05-09T09:17:30.197950+0000","last_fullsized":"2024-05-09T09:17:30.197950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197332+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:17:30.197332+0000","last_peered":"2024-05-09T09:17:30.197332+0000","last_clean":"2024-05-09T09:17:30.197332+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:17:30.197332+0000","last_undegraded":"2024-05-09T09:17:30.197332+0000","last_fullsized":"2024-05-09T09:17:30.197332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.124139+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:17:59.124139+0000","last_peered":"2024-05-09T09:17:59.124139+0000","last_clean":"2024-05-09T09:17:59.124139+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:17:59.124139+0000","last_undegraded":"2024-05-09T09:17:59.124139+0000","last_fullsized":"2024-05-09T09:17:59.124139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275512+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:17:30.275512+0000","last_peered":"2024-05-09T09:17:30.275512+0000","last_clean":"2024-05-09T09:17:30.275512+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:17:30.275512+0000","last_undegraded":"2024-05-09T09:17:30.275512+0000","last_fullsized":"2024-05-09T09:17:30.275512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.464422+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:17:30.464422+0000","last_peered":"2024-05-09T09:17:30.464422+0000","last_clean":"2024-05-09T09:17:30.464422+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:17:30.464422+0000","last_undegraded":"2024-05-09T09:17:30.464422+0000","last_fullsized":"2024-05-09T09:17:30.464422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.141206+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:17:59.141206+0000","last_peered":"2024-05-09T09:17:59.141206+0000","last_clean":"2024-05-09T09:17:59.141206+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:17:59.141206+0000","last_undegraded":"2024-05-09T09:17:59.141206+0000","last_fullsized":"2024-05-09T09:17:59.141206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275890+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:17:30.275890+0000","last_peered":"2024-05-09T09:17:30.275890+0000","last_clean":"2024-05-09T09:17:30.275890+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:17:30.275890+0000","last_undegraded":"2024-05-09T09:17:30.275890+0000","last_fullsized":"2024-05-09T09:17:30.275890+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.213737+0000","last_change":"2024-05-09T09:17:30.213383+0000","last_active":"2024-05-09T09:17:30.213737+0000","last_peered":"2024-05-09T09:17:30.213737+0000","last_clean":"2024-05-09T09:17:30.213737+0000","last_became_active":"2024-05-09T09:17:30.205007+0000","last_became_peered":"2024-05-09T09:17:30.205007+0000","last_unstale":"2024-05-09T09:17:30.213737+0000","last_undegraded":"2024-05-09T09:17:30.213737+0000","last_fullsized":"2024-05-09T09:17:30.213737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:36.820105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200587+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:17:30.200587+0000","last_peered":"2024-05-09T09:17:30.200587+0000","last_clean":"2024-05-09T09:17:30.200587+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:17:30.200587+0000","last_undegraded":"2024-05-09T09:17:30.200587+0000","last_fullsized":"2024-05-09T09:17:30.200587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275698+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:17:30.275698+0000","last_peered":"2024-05-09T09:17:30.275698+0000","last_clean":"2024-05-09T09:17:30.275698+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:17:30.275698+0000","last_undegraded":"2024-05-09T09:17:30.275698+0000","last_fullsized":"2024-05-09T09:17:30.275698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202142+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:17:30.202142+0000","last_peered":"2024-05-09T09:17:30.202142+0000","last_clean":"2024-05-09T09:17:30.202142+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:17:30.202142+0000","last_undegraded":"2024-05-09T09:17:30.202142+0000","last_fullsized":"2024-05-09T09:17:30.202142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135980+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:17:59.135980+0000","last_peered":"2024-05-09T09:17:59.135980+0000","last_clean":"2024-05-09T09:17:59.135980+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:17:59.135980+0000","last_undegraded":"2024-05-09T09:17:59.135980+0000","last_fullsized":"2024-05-09T09:17:59.135980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198622+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:17:30.198622+0000","last_peered":"2024-05-09T09:17:30.198622+0000","last_clean":"2024-05-09T09:17:30.198622+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:17:30.198622+0000","last_undegraded":"2024-05-09T09:17:30.198622+0000","last_fullsized":"2024-05-09T09:17:30.198622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.322280+0000","last_change":"2024-05-09T09:17:30.281412+0000","last_active":"2024-05-09T09:18:00.322280+0000","last_peered":"2024-05-09T09:18:00.322280+0000","last_clean":"2024-05-09T09:18:00.322280+0000","last_became_active":"2024-05-09T09:17:30.277253+0000","last_became_peered":"2024-05-09T09:17:30.277253+0000","last_unstale":"2024-05-09T09:18:00.322280+0000","last_undegraded":"2024-05-09T09:18:00.322280+0000","last_fullsized":"2024-05-09T09:18:00.322280+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:48.491205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123482+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:17:59.123482+0000","last_peered":"2024-05-09T09:17:59.123482+0000","last_clean":"2024-05-09T09:17:59.123482+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:17:59.123482+0000","last_undegraded":"2024-05-09T09:17:59.123482+0000","last_fullsized":"2024-05-09T09:17:59.123482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199031+0000","last_change":"2024-05-09T09:16:36.913312+0000","last_active":"2024-05-09T09:17:30.199031+0000","last_peered":"2024-05-09T09:17:30.199031+0000","last_clean":"2024-05-09T09:17:30.199031+0000","last_became_active":"2024-05-09T09:16:36.912908+0000","last_became_peered":"2024-05-09T09:16:36.912908+0000","last_unstale":"2024-05-09T09:17:30.199031+0000","last_undegraded":"2024-05-09T09:17:30.199031+0000","last_fullsized":"2024-05-09T09:17:30.199031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:59.214206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199926+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:17:30.199926+0000","last_peered":"2024-05-09T09:17:30.199926+0000","last_clean":"2024-05-09T09:17:30.199926+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:17:30.199926+0000","last_undegraded":"2024-05-09T09:17:30.199926+0000","last_fullsized":"2024-05-09T09:17:30.199926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133105+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:17:59.133105+0000","last_peered":"2024-05-09T09:17:59.133105+0000","last_clean":"2024-05-09T09:17:59.133105+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:17:59.133105+0000","last_undegraded":"2024-05-09T09:17:59.133105+0000","last_fullsized":"2024-05-09T09:17:59.133105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.277465+0000","last_change":"2024-05-09T09:17:30.277465+0000","last_active":"2024-05-09T09:17:30.277465+0000","last_peered":"2024-05-09T09:17:30.277465+0000","last_clean":"2024-05-09T09:17:30.277465+0000","last_became_active":"2024-05-09T09:17:30.277014+0000","last_became_peered":"2024-05-09T09:17:30.277014+0000","last_unstale":"2024-05-09T09:17:30.277465+0000","last_undegraded":"2024-05-09T09:17:30.277465+0000","last_fullsized":"2024-05-09T09:17:30.277465+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:36:04.346440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272444+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:17:30.272444+0000","last_peered":"2024-05-09T09:17:30.272444+0000","last_clean":"2024-05-09T09:17:30.272444+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:17:30.272444+0000","last_undegraded":"2024-05-09T09:17:30.272444+0000","last_fullsized":"2024-05-09T09:17:30.272444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139286+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:17:59.139286+0000","last_peered":"2024-05-09T09:17:59.139286+0000","last_clean":"2024-05-09T09:17:59.139286+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:17:59.139286+0000","last_undegraded":"2024-05-09T09:17:59.139286+0000","last_fullsized":"2024-05-09T09:17:59.139286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202153+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:17:30.202153+0000","last_peered":"2024-05-09T09:17:30.202153+0000","last_clean":"2024-05-09T09:17:30.202153+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:17:30.202153+0000","last_undegraded":"2024-05-09T09:17:30.202153+0000","last_fullsized":"2024-05-09T09:17:30.202153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200844+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:17:30.200844+0000","last_peered":"2024-05-09T09:17:30.200844+0000","last_clean":"2024-05-09T09:17:30.200844+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:17:30.200844+0000","last_undegraded":"2024-05-09T09:17:30.200844+0000","last_fullsized":"2024-05-09T09:17:30.200844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139719+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:17:59.139719+0000","last_peered":"2024-05-09T09:17:59.139719+0000","last_clean":"2024-05-09T09:17:59.139719+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:17:59.139719+0000","last_undegraded":"2024-05-09T09:17:59.139719+0000","last_fullsized":"2024-05-09T09:17:59.139719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.463926+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:17:30.463926+0000","last_peered":"2024-05-09T09:17:30.463926+0000","last_clean":"2024-05-09T09:17:30.463926+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:17:30.463926+0000","last_undegraded":"2024-05-09T09:17:30.463926+0000","last_fullsized":"2024-05-09T09:17:30.463926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.321726+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:18:00.321726+0000","last_peered":"2024-05-09T09:18:00.321726+0000","last_clean":"2024-05-09T09:18:00.321726+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:18:00.321726+0000","last_undegraded":"2024-05-09T09:18:00.321726+0000","last_fullsized":"2024-05-09T09:18:00.321726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201804+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:17:30.201804+0000","last_peered":"2024-05-09T09:17:30.201804+0000","last_clean":"2024-05-09T09:17:30.201804+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:17:30.201804+0000","last_undegraded":"2024-05-09T09:17:30.201804+0000","last_fullsized":"2024-05-09T09:17:30.201804+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196370+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:17:30.196370+0000","last_peered":"2024-05-09T09:17:30.196370+0000","last_clean":"2024-05-09T09:17:30.196370+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:17:30.196370+0000","last_undegraded":"2024-05-09T09:17:30.196370+0000","last_fullsized":"2024-05-09T09:17:30.196370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203975+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:17:30.203975+0000","last_peered":"2024-05-09T09:17:30.203975+0000","last_clean":"2024-05-09T09:17:30.203975+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:17:30.203975+0000","last_undegraded":"2024-05-09T09:17:30.203975+0000","last_fullsized":"2024-05-09T09:17:30.203975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134161+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:17:59.134161+0000","last_peered":"2024-05-09T09:17:59.134161+0000","last_clean":"2024-05-09T09:17:59.134161+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:17:59.134161+0000","last_undegraded":"2024-05-09T09:17:59.134161+0000","last_fullsized":"2024-05-09T09:17:59.134161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276096+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:17:30.276096+0000","last_peered":"2024-05-09T09:17:30.276096+0000","last_clean":"2024-05-09T09:17:30.276096+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:17:30.276096+0000","last_undegraded":"2024-05-09T09:17:30.276096+0000","last_fullsized":"2024-05-09T09:17:30.276096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203205+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:17:30.203205+0000","last_peered":"2024-05-09T09:17:30.203205+0000","last_clean":"2024-05-09T09:17:30.203205+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:17:30.203205+0000","last_undegraded":"2024-05-09T09:17:30.203205+0000","last_fullsized":"2024-05-09T09:17:30.203205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200856+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:17:30.200856+0000","last_peered":"2024-05-09T09:17:30.200856+0000","last_clean":"2024-05-09T09:17:30.200856+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:17:30.200856+0000","last_undegraded":"2024-05-09T09:17:30.200856+0000","last_fullsized":"2024-05-09T09:17:30.200856+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196970+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:17:30.196970+0000","last_peered":"2024-05-09T09:17:30.196970+0000","last_clean":"2024-05-09T09:17:30.196970+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:17:30.196970+0000","last_undegraded":"2024-05-09T09:17:30.196970+0000","last_fullsized":"2024-05-09T09:17:30.196970+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":207,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:01.323220+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:18:01.323220+0000","last_peered":"2024-05-09T09:18:01.323220+0000","last_clean":"2024-05-09T09:18:01.323220+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:18:01.323220+0000","last_undegraded":"2024-05-09T09:18:01.323220+0000","last_fullsized":"2024-05-09T09:18:01.323220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":135,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.132631+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:17:59.132631+0000","last_peered":"2024-05-09T09:17:59.132631+0000","last_clean":"2024-05-09T09:17:59.132631+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:17:59.132631+0000","last_undegraded":"2024-05-09T09:17:59.132631+0000","last_fullsized":"2024-05-09T09:17:59.132631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203677+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:17:30.203677+0000","last_peered":"2024-05-09T09:17:30.203677+0000","last_clean":"2024-05-09T09:17:30.203677+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:17:30.203677+0000","last_undegraded":"2024-05-09T09:17:30.203677+0000","last_fullsized":"2024-05-09T09:17:30.203677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198267+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:17:30.198267+0000","last_peered":"2024-05-09T09:17:30.198267+0000","last_clean":"2024-05-09T09:17:30.198267+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:17:30.198267+0000","last_undegraded":"2024-05-09T09:17:30.198267+0000","last_fullsized":"2024-05-09T09:17:30.198267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272566+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:17:30.272566+0000","last_peered":"2024-05-09T09:17:30.272566+0000","last_clean":"2024-05-09T09:17:30.272566+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:17:30.272566+0000","last_undegraded":"2024-05-09T09:17:30.272566+0000","last_fullsized":"2024-05-09T09:17:30.272566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200229+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:17:30.200229+0000","last_peered":"2024-05-09T09:17:30.200229+0000","last_clean":"2024-05-09T09:17:30.200229+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:17:30.200229+0000","last_undegraded":"2024-05-09T09:17:30.200229+0000","last_fullsized":"2024-05-09T09:17:30.200229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201267+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:17:30.201267+0000","last_peered":"2024-05-09T09:17:30.201267+0000","last_clean":"2024-05-09T09:17:30.201267+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:17:30.201267+0000","last_undegraded":"2024-05-09T09:17:30.201267+0000","last_fullsized":"2024-05-09T09:17:30.201267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203660+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:17:30.203660+0000","last_peered":"2024-05-09T09:17:30.203660+0000","last_clean":"2024-05-09T09:17:30.203660+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:17:30.203660+0000","last_undegraded":"2024-05-09T09:17:30.203660+0000","last_fullsized":"2024-05-09T09:17:30.203660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.465897+0000","last_change":"2024-05-09T09:17:30.465425+0000","last_active":"2024-05-09T09:17:30.465897+0000","last_peered":"2024-05-09T09:17:30.465897+0000","last_clean":"2024-05-09T09:17:30.465897+0000","last_became_active":"2024-05-09T09:17:30.465043+0000","last_became_peered":"2024-05-09T09:17:30.465043+0000","last_unstale":"2024-05-09T09:17:30.465897+0000","last_undegraded":"2024-05-09T09:17:30.465897+0000","last_fullsized":"2024-05-09T09:17:30.465897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:16:54.706845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.321216+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:18:00.321216+0000","last_peered":"2024-05-09T09:18:00.321216+0000","last_clean":"2024-05-09T09:18:00.321216+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:18:00.321216+0000","last_undegraded":"2024-05-09T09:18:00.321216+0000","last_fullsized":"2024-05-09T09:18:00.321216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203620+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:17:30.203620+0000","last_peered":"2024-05-09T09:17:30.203620+0000","last_clean":"2024-05-09T09:17:30.203620+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:17:30.203620+0000","last_undegraded":"2024-05-09T09:17:30.203620+0000","last_fullsized":"2024-05-09T09:17:30.203620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272921+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:17:30.272921+0000","last_peered":"2024-05-09T09:17:30.272921+0000","last_clean":"2024-05-09T09:17:30.272921+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:17:30.272921+0000","last_undegraded":"2024-05-09T09:17:30.272921+0000","last_fullsized":"2024-05-09T09:17:30.272921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200420+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:17:30.200420+0000","last_peered":"2024-05-09T09:17:30.200420+0000","last_clean":"2024-05-09T09:17:30.200420+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:17:30.200420+0000","last_undegraded":"2024-05-09T09:17:30.200420+0000","last_fullsized":"2024-05-09T09:17:30.200420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135691+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:17:59.135691+0000","last_peered":"2024-05-09T09:17:59.135691+0000","last_clean":"2024-05-09T09:17:59.135691+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:17:59.135691+0000","last_undegraded":"2024-05-09T09:17:59.135691+0000","last_fullsized":"2024-05-09T09:17:59.135691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200158+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:17:30.200158+0000","last_peered":"2024-05-09T09:17:30.200158+0000","last_clean":"2024-05-09T09:17:30.200158+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:17:30.200158+0000","last_undegraded":"2024-05-09T09:17:30.200158+0000","last_fullsized":"2024-05-09T09:17:30.200158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203431+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:17:30.203431+0000","last_peered":"2024-05-09T09:17:30.203431+0000","last_clean":"2024-05-09T09:17:30.203431+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:17:30.203431+0000","last_undegraded":"2024-05-09T09:17:30.203431+0000","last_fullsized":"2024-05-09T09:17:30.203431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200520+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:17:30.200520+0000","last_peered":"2024-05-09T09:17:30.200520+0000","last_clean":"2024-05-09T09:17:30.200520+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:17:30.200520+0000","last_undegraded":"2024-05-09T09:17:30.200520+0000","last_fullsized":"2024-05-09T09:17:30.200520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201202+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:17:30.201202+0000","last_peered":"2024-05-09T09:17:30.201202+0000","last_clean":"2024-05-09T09:17:30.201202+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:17:30.201202+0000","last_undegraded":"2024-05-09T09:17:30.201202+0000","last_fullsized":"2024-05-09T09:17:30.201202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196900+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:17:30.196900+0000","last_peered":"2024-05-09T09:17:30.196900+0000","last_clean":"2024-05-09T09:17:30.196900+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:17:30.196900+0000","last_undegraded":"2024-05-09T09:17:30.196900+0000","last_fullsized":"2024-05-09T09:17:30.196900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.127717+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:17:59.127717+0000","last_peered":"2024-05-09T09:17:59.127717+0000","last_clean":"2024-05-09T09:17:59.127717+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:17:59.127717+0000","last_undegraded":"2024-05-09T09:17:59.127717+0000","last_fullsized":"2024-05-09T09:17:59.127717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272710+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:17:30.272710+0000","last_peered":"2024-05-09T09:17:30.272710+0000","last_clean":"2024-05-09T09:17:30.272710+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:17:30.272710+0000","last_undegraded":"2024-05-09T09:17:30.272710+0000","last_fullsized":"2024-05-09T09:17:30.272710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.204121+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:17:30.204121+0000","last_peered":"2024-05-09T09:17:30.204121+0000","last_clean":"2024-05-09T09:17:30.204121+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:17:30.204121+0000","last_undegraded":"2024-05-09T09:17:30.204121+0000","last_fullsized":"2024-05-09T09:17:30.204121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.131020+0000","last_change":"2024-05-09T09:17:30.203699+0000","last_active":"2024-05-09T09:17:59.131020+0000","last_peered":"2024-05-09T09:17:59.131020+0000","last_clean":"2024-05-09T09:17:59.131020+0000","last_became_active":"2024-05-09T09:17:30.203312+0000","last_became_peered":"2024-05-09T09:17:30.203312+0000","last_unstale":"2024-05-09T09:17:59.131020+0000","last_undegraded":"2024-05-09T09:17:59.131020+0000","last_fullsized":"2024-05-09T09:17:59.131020+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272709+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:17:30.272709+0000","last_peered":"2024-05-09T09:17:30.272709+0000","last_clean":"2024-05-09T09:17:30.272709+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:17:30.272709+0000","last_undegraded":"2024-05-09T09:17:30.272709+0000","last_fullsized":"2024-05-09T09:17:30.272709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.115942+0000","last_change":"2024-05-09T09:17:30.276150+0000","last_active":"2024-05-09T09:17:59.115942+0000","last_peered":"2024-05-09T09:17:59.115942+0000","last_clean":"2024-05-09T09:17:59.115942+0000","last_became_active":"2024-05-09T09:17:30.275868+0000","last_became_peered":"2024-05-09T09:17:30.275868+0000","last_unstale":"2024-05-09T09:17:59.115942+0000","last_undegraded":"2024-05-09T09:17:59.115942+0000","last_fullsized":"2024-05-09T09:17:59.115942+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:18:08.892380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":95,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:56.314718+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:17:56.314718+0000","last_peered":"2024-05-09T09:17:56.314718+0000","last_clean":"2024-05-09T09:17:56.314718+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:17:56.314718+0000","last_undegraded":"2024-05-09T09:17:56.314718+0000","last_fullsized":"2024-05-09T09:17:56.314718+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":65,"num_read_kb":65,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203956+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:17:30.203956+0000","last_peered":"2024-05-09T09:17:30.203956+0000","last_clean":"2024-05-09T09:17:30.203956+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:17:30.203956+0000","last_undegraded":"2024-05-09T09:17:30.203956+0000","last_fullsized":"2024-05-09T09:17:30.203956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.131068+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:17:59.131068+0000","last_peered":"2024-05-09T09:17:59.131068+0000","last_clean":"2024-05-09T09:17:59.131068+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:17:59.131068+0000","last_undegraded":"2024-05-09T09:17:59.131068+0000","last_fullsized":"2024-05-09T09:17:59.131068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276021+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:17:30.276021+0000","last_peered":"2024-05-09T09:17:30.276021+0000","last_clean":"2024-05-09T09:17:30.276021+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:17:30.276021+0000","last_undegraded":"2024-05-09T09:17:30.276021+0000","last_fullsized":"2024-05-09T09:17:30.276021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":351,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203210+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:17:30.203210+0000","last_peered":"2024-05-09T09:17:30.203210+0000","last_clean":"2024-05-09T09:17:30.203210+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:17:30.203210+0000","last_undegraded":"2024-05-09T09:17:30.203210+0000","last_fullsized":"2024-05-09T09:17:30.203210+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203358+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:17:30.203358+0000","last_peered":"2024-05-09T09:17:30.203358+0000","last_clean":"2024-05-09T09:17:30.203358+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:17:30.203358+0000","last_undegraded":"2024-05-09T09:17:30.203358+0000","last_fullsized":"2024-05-09T09:17:30.203358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272341+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:17:30.272341+0000","last_peered":"2024-05-09T09:17:30.272341+0000","last_clean":"2024-05-09T09:17:30.272341+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:17:30.272341+0000","last_undegraded":"2024-05-09T09:17:30.272341+0000","last_fullsized":"2024-05-09T09:17:30.272341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203451+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:17:30.203451+0000","last_peered":"2024-05-09T09:17:30.203451+0000","last_clean":"2024-05-09T09:17:30.203451+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:17:30.203451+0000","last_undegraded":"2024-05-09T09:17:30.203451+0000","last_fullsized":"2024-05-09T09:17:30.203451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200099+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:17:30.200099+0000","last_peered":"2024-05-09T09:17:30.200099+0000","last_clean":"2024-05-09T09:17:30.200099+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:17:30.200099+0000","last_undegraded":"2024-05-09T09:17:30.200099+0000","last_fullsized":"2024-05-09T09:17:30.200099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.208425+0000","last_change":"2024-05-09T09:17:30.208053+0000","last_active":"2024-05-09T09:17:30.208425+0000","last_peered":"2024-05-09T09:17:30.208425+0000","last_clean":"2024-05-09T09:17:30.208425+0000","last_became_active":"2024-05-09T09:17:30.203628+0000","last_became_peered":"2024-05-09T09:17:30.203628+0000","last_unstale":"2024-05-09T09:17:30.208425+0000","last_undegraded":"2024-05-09T09:17:30.208425+0000","last_fullsized":"2024-05-09T09:17:30.208425+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:37.871338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272911+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:17:30.272911+0000","last_peered":"2024-05-09T09:17:30.272911+0000","last_clean":"2024-05-09T09:17:30.272911+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:17:30.272911+0000","last_undegraded":"2024-05-09T09:17:30.272911+0000","last_fullsized":"2024-05-09T09:17:30.272911+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.463898+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:17:30.463898+0000","last_peered":"2024-05-09T09:17:30.463898+0000","last_clean":"2024-05-09T09:17:30.463898+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:17:30.463898+0000","last_undegraded":"2024-05-09T09:17:30.463898+0000","last_fullsized":"2024-05-09T09:17:30.463898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197199+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:17:30.197199+0000","last_peered":"2024-05-09T09:17:30.197199+0000","last_clean":"2024-05-09T09:17:30.197199+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:17:30.197199+0000","last_undegraded":"2024-05-09T09:17:30.197199+0000","last_fullsized":"2024-05-09T09:17:30.197199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139503+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:17:59.139503+0000","last_peered":"2024-05-09T09:17:59.139503+0000","last_clean":"2024-05-09T09:17:59.139503+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:17:59.139503+0000","last_undegraded":"2024-05-09T09:17:59.139503+0000","last_fullsized":"2024-05-09T09:17:59.139503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198637+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:17:30.198637+0000","last_peered":"2024-05-09T09:17:30.198637+0000","last_clean":"2024-05-09T09:17:30.198637+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:17:30.198637+0000","last_undegraded":"2024-05-09T09:17:30.198637+0000","last_fullsized":"2024-05-09T09:17:30.198637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276292+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:17:30.276292+0000","last_peered":"2024-05-09T09:17:30.276292+0000","last_clean":"2024-05-09T09:17:30.276292+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:17:30.276292+0000","last_undegraded":"2024-05-09T09:17:30.276292+0000","last_fullsized":"2024-05-09T09:17:30.276292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139913+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:17:59.139913+0000","last_peered":"2024-05-09T09:17:59.139913+0000","last_clean":"2024-05-09T09:17:59.139913+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:17:59.139913+0000","last_undegraded":"2024-05-09T09:17:59.139913+0000","last_fullsized":"2024-05-09T09:17:59.139913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275865+0000","last_change":"2024-05-09T09:16:36.967460+0000","last_active":"2024-05-09T09:17:30.275865+0000","last_peered":"2024-05-09T09:17:30.275865+0000","last_clean":"2024-05-09T09:17:30.275865+0000","last_became_active":"2024-05-09T09:16:36.966966+0000","last_became_peered":"2024-05-09T09:16:36.966966+0000","last_unstale":"2024-05-09T09:17:30.275865+0000","last_undegraded":"2024-05-09T09:17:30.275865+0000","last_fullsized":"2024-05-09T09:17:30.275865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:14.954608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200436+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:17:30.200436+0000","last_peered":"2024-05-09T09:17:30.200436+0000","last_clean":"2024-05-09T09:17:30.200436+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:17:30.200436+0000","last_undegraded":"2024-05-09T09:17:30.200436+0000","last_fullsized":"2024-05-09T09:17:30.200436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.107450+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:17:59.107450+0000","last_peered":"2024-05-09T09:17:59.107450+0000","last_clean":"2024-05-09T09:17:59.107450+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:17:59.107450+0000","last_undegraded":"2024-05-09T09:17:59.107450+0000","last_fullsized":"2024-05-09T09:17:59.107450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197028+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:17:30.197028+0000","last_peered":"2024-05-09T09:17:30.197028+0000","last_clean":"2024-05-09T09:17:30.197028+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:17:30.197028+0000","last_undegraded":"2024-05-09T09:17:30.197028+0000","last_fullsized":"2024-05-09T09:17:30.197028+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.464027+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:17:30.464027+0000","last_peered":"2024-05-09T09:17:30.464027+0000","last_clean":"2024-05-09T09:17:30.464027+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:17:30.464027+0000","last_undegraded":"2024-05-09T09:17:30.464027+0000","last_fullsized":"2024-05-09T09:17:30.464027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200264+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:17:30.200264+0000","last_peered":"2024-05-09T09:17:30.200264+0000","last_clean":"2024-05-09T09:17:30.200264+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:17:30.200264+0000","last_undegraded":"2024-05-09T09:17:30.200264+0000","last_fullsized":"2024-05-09T09:17:30.200264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276003+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:17:30.276003+0000","last_peered":"2024-05-09T09:17:30.276003+0000","last_clean":"2024-05-09T09:17:30.276003+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:17:30.276003+0000","last_undegraded":"2024-05-09T09:17:30.276003+0000","last_fullsized":"2024-05-09T09:17:30.276003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197934+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:17:30.197934+0000","last_peered":"2024-05-09T09:17:30.197934+0000","last_clean":"2024-05-09T09:17:30.197934+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:17:30.197934+0000","last_undegraded":"2024-05-09T09:17:30.197934+0000","last_fullsized":"2024-05-09T09:17:30.197934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135332+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:17:59.135332+0000","last_peered":"2024-05-09T09:17:59.135332+0000","last_clean":"2024-05-09T09:17:59.135332+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:17:59.135332+0000","last_undegraded":"2024-05-09T09:17:59.135332+0000","last_fullsized":"2024-05-09T09:17:59.135332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275860+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:17:30.275860+0000","last_peered":"2024-05-09T09:17:30.275860+0000","last_clean":"2024-05-09T09:17:30.275860+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:17:30.275860+0000","last_undegraded":"2024-05-09T09:17:30.275860+0000","last_fullsized":"2024-05-09T09:17:30.275860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200902+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:17:30.200902+0000","last_peered":"2024-05-09T09:17:30.200902+0000","last_clean":"2024-05-09T09:17:30.200902+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:17:30.200902+0000","last_undegraded":"2024-05-09T09:17:30.200902+0000","last_fullsized":"2024-05-09T09:17:30.200902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203956+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:17:30.203956+0000","last_peered":"2024-05-09T09:17:30.203956+0000","last_clean":"2024-05-09T09:17:30.203956+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:17:30.203956+0000","last_undegraded":"2024-05-09T09:17:30.203956+0000","last_fullsized":"2024-05-09T09:17:30.203956+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200394+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:17:30.200394+0000","last_peered":"2024-05-09T09:17:30.200394+0000","last_clean":"2024-05-09T09:17:30.200394+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:17:30.200394+0000","last_undegraded":"2024-05-09T09:17:30.200394+0000","last_fullsized":"2024-05-09T09:17:30.200394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199138+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:17:30.199138+0000","last_peered":"2024-05-09T09:17:30.199138+0000","last_clean":"2024-05-09T09:17:30.199138+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:17:30.199138+0000","last_undegraded":"2024-05-09T09:17:30.199138+0000","last_fullsized":"2024-05-09T09:17:30.199138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.204341+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:17:30.204341+0000","last_peered":"2024-05-09T09:17:30.204341+0000","last_clean":"2024-05-09T09:17:30.204341+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:17:30.204341+0000","last_undegraded":"2024-05-09T09:17:30.204341+0000","last_fullsized":"2024-05-09T09:17:30.204341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202235+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:17:30.202235+0000","last_peered":"2024-05-09T09:17:30.202235+0000","last_clean":"2024-05-09T09:17:30.202235+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:17:30.202235+0000","last_undegraded":"2024-05-09T09:17:30.202235+0000","last_fullsized":"2024-05-09T09:17:30.202235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203366+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:17:30.203366+0000","last_peered":"2024-05-09T09:17:30.203366+0000","last_clean":"2024-05-09T09:17:30.203366+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:17:30.203366+0000","last_undegraded":"2024-05-09T09:17:30.203366+0000","last_fullsized":"2024-05-09T09:17:30.203366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203995+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:17:30.203995+0000","last_peered":"2024-05-09T09:17:30.203995+0000","last_clean":"2024-05-09T09:17:30.203995+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:17:30.203995+0000","last_undegraded":"2024-05-09T09:17:30.203995+0000","last_fullsized":"2024-05-09T09:17:30.203995+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200378+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:17:30.200378+0000","last_peered":"2024-05-09T09:17:30.200378+0000","last_clean":"2024-05-09T09:17:30.200378+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:17:30.200378+0000","last_undegraded":"2024-05-09T09:17:30.200378+0000","last_fullsized":"2024-05-09T09:17:30.200378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272924+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:17:30.272924+0000","last_peered":"2024-05-09T09:17:30.272924+0000","last_clean":"2024-05-09T09:17:30.272924+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:17:30.272924+0000","last_undegraded":"2024-05-09T09:17:30.272924+0000","last_fullsized":"2024-05-09T09:17:30.272924+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.204230+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:17:30.204230+0000","last_peered":"2024-05-09T09:17:30.204230+0000","last_clean":"2024-05-09T09:17:30.204230+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:17:30.204230+0000","last_undegraded":"2024-05-09T09:17:30.204230+0000","last_fullsized":"2024-05-09T09:17:30.204230+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200379+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:17:30.200379+0000","last_peered":"2024-05-09T09:17:30.200379+0000","last_clean":"2024-05-09T09:17:30.200379+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:17:30.200379+0000","last_undegraded":"2024-05-09T09:17:30.200379+0000","last_fullsized":"2024-05-09T09:17:30.200379+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.130318+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:17:59.130318+0000","last_peered":"2024-05-09T09:17:59.130318+0000","last_clean":"2024-05-09T09:17:59.130318+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:17:59.130318+0000","last_undegraded":"2024-05-09T09:17:59.130318+0000","last_fullsized":"2024-05-09T09:17:59.130318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133198+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:17:59.133198+0000","last_peered":"2024-05-09T09:17:59.133198+0000","last_clean":"2024-05-09T09:17:59.133198+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:17:59.133198+0000","last_undegraded":"2024-05-09T09:17:59.133198+0000","last_fullsized":"2024-05-09T09:17:59.133198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134304+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:17:59.134304+0000","last_peered":"2024-05-09T09:17:59.134304+0000","last_clean":"2024-05-09T09:17:59.134304+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:17:59.134304+0000","last_undegraded":"2024-05-09T09:17:59.134304+0000","last_fullsized":"2024-05-09T09:17:59.134304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272832+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:17:30.272832+0000","last_peered":"2024-05-09T09:17:30.272832+0000","last_clean":"2024-05-09T09:17:30.272832+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:17:30.272832+0000","last_undegraded":"2024-05-09T09:17:30.272832+0000","last_fullsized":"2024-05-09T09:17:30.272832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200609+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:17:30.200609+0000","last_peered":"2024-05-09T09:17:30.200609+0000","last_clean":"2024-05-09T09:17:30.200609+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:17:30.200609+0000","last_undegraded":"2024-05-09T09:17:30.200609+0000","last_fullsized":"2024-05-09T09:17:30.200609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.145520+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:17:59.145520+0000","last_peered":"2024-05-09T09:17:59.145520+0000","last_clean":"2024-05-09T09:17:59.145520+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:17:59.145520+0000","last_undegraded":"2024-05-09T09:17:59.145520+0000","last_fullsized":"2024-05-09T09:17:59.145520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126935+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:17:59.126935+0000","last_peered":"2024-05-09T09:17:59.126935+0000","last_clean":"2024-05-09T09:17:59.126935+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:17:59.126935+0000","last_undegraded":"2024-05-09T09:17:59.126935+0000","last_fullsized":"2024-05-09T09:17:59.126935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123443+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:17:59.123443+0000","last_peered":"2024-05-09T09:17:59.123443+0000","last_clean":"2024-05-09T09:17:59.123443+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:17:59.123443+0000","last_undegraded":"2024-05-09T09:17:59.123443+0000","last_fullsized":"2024-05-09T09:17:59.123443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139102+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:17:59.139102+0000","last_peered":"2024-05-09T09:17:59.139102+0000","last_clean":"2024-05-09T09:17:59.139102+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:17:59.139102+0000","last_undegraded":"2024-05-09T09:17:59.139102+0000","last_fullsized":"2024-05-09T09:17:59.139102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200770+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:17:30.200770+0000","last_peered":"2024-05-09T09:17:30.200770+0000","last_clean":"2024-05-09T09:17:30.200770+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:17:30.200770+0000","last_undegraded":"2024-05-09T09:17:30.200770+0000","last_fullsized":"2024-05-09T09:17:30.200770+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272623+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:17:30.272623+0000","last_peered":"2024-05-09T09:17:30.272623+0000","last_clean":"2024-05-09T09:17:30.272623+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:17:30.272623+0000","last_undegraded":"2024-05-09T09:17:30.272623+0000","last_fullsized":"2024-05-09T09:17:30.272623+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201328+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:17:30.201328+0000","last_peered":"2024-05-09T09:17:30.201328+0000","last_clean":"2024-05-09T09:17:30.201328+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:17:30.201328+0000","last_undegraded":"2024-05-09T09:17:30.201328+0000","last_fullsized":"2024-05-09T09:17:30.201328+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":280,"ondisk_log_size":280,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":433,"num_read_kb":287,"num_write":167,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1035,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":202851,"data_compressed":4887,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383711,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32156,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712996,"statfs":{"total":95995035648,"available":95962107904,"internally_reserved":0,"allocated":999424,"data_stored":500766,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1151617,"data_compressed":57955,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":505076,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070511,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1007616,"data_stored":500878,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332148,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1060864,"data_stored":1096809,"data_compressed":56326,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495673,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":991232,"data_stored":496156,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574929,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":559422,"data_compressed":22240,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738455,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":497544,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":560399,"data_compressed":22240,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496350,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":501960,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1092538,"data_compressed":56326,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63356,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64333,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2183,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65183,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:18:04.776 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph pg dump --format=json 2024-05-09T09:18:05.021 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:05.393 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:05 smithi069 ceph-mon[30123]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:05 smithi163 ceph-mon[34354]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:05 smithi103 ceph-mon[34421]: from='client.15111 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:06.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:06 smithi069 ceph-mon[30123]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:06 smithi163 ceph-mon[34354]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:06 smithi103 ceph-mon[34421]: pgmap v339: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 -- 172.21.15.69:0/3574050840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c106540 msgr2=0x7f488c10e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 --2- 172.21.15.69:0/3574050840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c106540 0x7f488c10e110 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f488400b0d0 tx=0x7f488400b3e0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 -- 172.21.15.69:0/3574050840 shutdown_connections 2024-05-09T09:18:06.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 --2- 172.21.15.69:0/3574050840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c106540 0x7f488c10e110 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 --2- 172.21.15.69:0/3574050840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c1014b0 0x7f488c101890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 --2- 172.21.15.69:0/3574050840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c101e60 0x7f488c105e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.876+0000 7f4893a1b700 1 -- 172.21.15.69:0/3574050840 >> 172.21.15.69:0/3574050840 conn(0x7f488c0781d0 msgr2=0x7f488c0785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- 172.21.15.69:0/3574050840 shutdown_connections 2024-05-09T09:18:06.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- 172.21.15.69:0/3574050840 wait complete. 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 Processor -- start 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- start start 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 0x7f488c199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 0x7f488c1a4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f488c111620 con 0x7f488c101e60 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1114a0 con 0x7f488c1014b0 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.877+0000 7f4893a1b700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f488c1117a0 con 0x7f488c106540 2024-05-09T09:18:06.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51882/0 (socket says 172.21.15.69:51882) 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 -- 172.21.15.69:0/2647446154 learned_addr learned my addr 172.21.15.69:0/2647446154 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f48917b7700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 0x7f488c199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4891fb8700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 0x7f488c1a4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 -- 172.21.15.69:0/2647446154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 msgr2=0x7f488c1a4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 0x7f488c1a4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 -- 172.21.15.69:0/2647446154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 msgr2=0x7f488c199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 0x7f488c199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:06.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 -- 172.21.15.69:0/2647446154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f488c1a4fe0 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f48917b7700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 0x7f488c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4891fb8700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 0x7f488c1a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4890fb6700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f488800b500 tx=0x7f488800b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4888015020 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f488c1a5270 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4888011d10 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f488800f2e0 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.878+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f488c1a5780 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.879+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f488c102940 con 0x7f488c101e60 2024-05-09T09:18:06.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.880+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4888018020 con 0x7f488c101e60 2024-05-09T09:18:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.881+0000 7f48827fc700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 0x7f48780a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.881+0000 7f48917b7700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 0x7f48780a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.881+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f48880cebb0 con 0x7f488c101e60 2024-05-09T09:18:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.881+0000 7f48917b7700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 0x7f48780a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f487c0099d0 tx=0x7f487c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:06.886 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:06.886+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f488801f020 con 0x7f488c101e60 2024-05-09T09:18:07.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.075+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f488c062760 con 0x7f48780a2c20 2024-05-09T09:18:07.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.084+0000 7f48827fc700 1 -- 172.21.15.69:0/2647446154 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340872 (secure 0 0 0) 0x7f488c062760 con 0x7f48780a2c20 2024-05-09T09:18:07.089 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:07.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 msgr2=0x7f48780a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:07.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 0x7f48780a50e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f487c0099d0 tx=0x7f487c008040 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 msgr2=0x7f488c199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f488800b500 tx=0x7f488800b8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 shutdown_connections 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f488c106540 0x7f488c1a4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f48780a2c20 0x7f48780a50e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f488c1014b0 0x7f488c199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 --2- 172.21.15.69:0/2647446154 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f488c101e60 0x7f488c199f60 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 >> 172.21.15.69:0/2647446154 conn(0x7f488c0781d0 msgr2=0x7f488c0ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 shutdown_connections 2024-05-09T09:18:07.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:07.088+0000 7f4893a1b700 1 -- 172.21.15.69:0/2647446154 wait complete. 2024-05-09T09:18:07.093 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:18:07.825 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":339,"stamp":"2024-05-09T09:18:05.125905+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658349,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":564,"num_read_kb":401,"num_write":350,"num_write_kb":3060,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181771,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":784,"ondisk_log_size":784,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352984,"kb_used_data":12152,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588840,"statfs":{"total":1151940427776,"available":1151578972160,"internally_reserved":0,"allocated":12443648,"data_stored":7959905,"data_compressed":359364,"data_compressed_allocated":491520,"data_compressed_original":2482176,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":14,"apply_latency_ms":14,"commit_latency_ns":14000000,"apply_latency_ns":14000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":67,"num_read_kb":37,"num_write":7,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003709"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.148733+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:17:59.148733+0000","last_peered":"2024-05-09T09:17:59.148733+0000","last_clean":"2024-05-09T09:17:59.148733+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:17:59.148733+0000","last_undegraded":"2024-05-09T09:17:59.148733+0000","last_fullsized":"2024-05-09T09:17:59.148733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.118550+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:17:59.118550+0000","last_peered":"2024-05-09T09:17:59.118550+0000","last_clean":"2024-05-09T09:17:59.118550+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:17:59.118550+0000","last_undegraded":"2024-05-09T09:17:59.118550+0000","last_fullsized":"2024-05-09T09:17:59.118550+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.138566+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:17:59.138566+0000","last_peered":"2024-05-09T09:17:59.138566+0000","last_clean":"2024-05-09T09:17:59.138566+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:17:59.138566+0000","last_undegraded":"2024-05-09T09:17:59.138566+0000","last_fullsized":"2024-05-09T09:17:59.138566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.118775+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:17:59.118775+0000","last_peered":"2024-05-09T09:17:59.118775+0000","last_clean":"2024-05-09T09:17:59.118775+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:17:59.118775+0000","last_undegraded":"2024-05-09T09:17:59.118775+0000","last_fullsized":"2024-05-09T09:17:59.118775+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123283+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:17:59.123283+0000","last_peered":"2024-05-09T09:17:59.123283+0000","last_clean":"2024-05-09T09:17:59.123283+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:17:59.123283+0000","last_undegraded":"2024-05-09T09:17:59.123283+0000","last_fullsized":"2024-05-09T09:17:59.123283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126266+0000","last_change":"2024-05-09T09:17:30.205057+0000","last_active":"2024-05-09T09:17:59.126266+0000","last_peered":"2024-05-09T09:17:59.126266+0000","last_clean":"2024-05-09T09:17:59.126266+0000","last_became_active":"2024-05-09T09:17:30.204736+0000","last_became_peered":"2024-05-09T09:17:30.204736+0000","last_unstale":"2024-05-09T09:17:59.126266+0000","last_undegraded":"2024-05-09T09:17:59.126266+0000","last_fullsized":"2024-05-09T09:17:59.126266+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:00.112812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.132135+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:17:59.132135+0000","last_peered":"2024-05-09T09:17:59.132135+0000","last_clean":"2024-05-09T09:17:59.132135+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:17:59.132135+0000","last_undegraded":"2024-05-09T09:17:59.132135+0000","last_fullsized":"2024-05-09T09:17:59.132135+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.138729+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:17:59.138729+0000","last_peered":"2024-05-09T09:17:59.138729+0000","last_clean":"2024-05-09T09:17:59.138729+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:17:59.138729+0000","last_undegraded":"2024-05-09T09:17:59.138729+0000","last_fullsized":"2024-05-09T09:17:59.138729+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.147140+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:17:59.147140+0000","last_peered":"2024-05-09T09:17:59.147140+0000","last_clean":"2024-05-09T09:17:59.147140+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:17:59.147140+0000","last_undegraded":"2024-05-09T09:17:59.147140+0000","last_fullsized":"2024-05-09T09:17:59.147140+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":41,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126828+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:17:59.126828+0000","last_peered":"2024-05-09T09:17:59.126828+0000","last_clean":"2024-05-09T09:17:59.126828+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:17:59.126828+0000","last_undegraded":"2024-05-09T09:17:59.126828+0000","last_fullsized":"2024-05-09T09:17:59.126828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.146832+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:17:59.146832+0000","last_peered":"2024-05-09T09:17:59.146832+0000","last_clean":"2024-05-09T09:17:59.146832+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:17:59.146832+0000","last_undegraded":"2024-05-09T09:17:59.146832+0000","last_fullsized":"2024-05-09T09:17:59.146832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.128543+0000","last_change":"2024-05-09T09:17:30.205228+0000","last_active":"2024-05-09T09:17:59.128543+0000","last_peered":"2024-05-09T09:17:59.128543+0000","last_clean":"2024-05-09T09:17:59.128543+0000","last_became_active":"2024-05-09T09:17:30.204824+0000","last_became_peered":"2024-05-09T09:17:30.204824+0000","last_unstale":"2024-05-09T09:17:59.128543+0000","last_undegraded":"2024-05-09T09:17:59.128543+0000","last_fullsized":"2024-05-09T09:17:59.128543+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:28.158394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.130105+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:17:59.130105+0000","last_peered":"2024-05-09T09:17:59.130105+0000","last_clean":"2024-05-09T09:17:59.130105+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:17:59.130105+0000","last_undegraded":"2024-05-09T09:17:59.130105+0000","last_fullsized":"2024-05-09T09:17:59.130105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133283+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:17:59.133283+0000","last_peered":"2024-05-09T09:17:59.133283+0000","last_clean":"2024-05-09T09:17:59.133283+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:17:59.133283+0000","last_undegraded":"2024-05-09T09:17:59.133283+0000","last_fullsized":"2024-05-09T09:17:59.133283+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134317+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:17:59.134317+0000","last_peered":"2024-05-09T09:17:59.134317+0000","last_clean":"2024-05-09T09:17:59.134317+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:17:59.134317+0000","last_undegraded":"2024-05-09T09:17:59.134317+0000","last_fullsized":"2024-05-09T09:17:59.134317+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.144677+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:17:59.144677+0000","last_peered":"2024-05-09T09:17:59.144677+0000","last_clean":"2024-05-09T09:17:59.144677+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:17:59.144677+0000","last_undegraded":"2024-05-09T09:17:59.144677+0000","last_fullsized":"2024-05-09T09:17:59.144677+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.148779+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:17:59.148779+0000","last_peered":"2024-05-09T09:17:59.148779+0000","last_clean":"2024-05-09T09:17:59.148779+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:17:59.148779+0000","last_undegraded":"2024-05-09T09:17:59.148779+0000","last_fullsized":"2024-05-09T09:17:59.148779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.128284+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:17:59.128284+0000","last_peered":"2024-05-09T09:17:59.128284+0000","last_clean":"2024-05-09T09:17:59.128284+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:17:59.128284+0000","last_undegraded":"2024-05-09T09:17:59.128284+0000","last_fullsized":"2024-05-09T09:17:59.128284+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133825+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:17:59.133825+0000","last_peered":"2024-05-09T09:17:59.133825+0000","last_clean":"2024-05-09T09:17:59.133825+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:17:59.133825+0000","last_undegraded":"2024-05-09T09:17:59.133825+0000","last_fullsized":"2024-05-09T09:17:59.133825+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197950+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:17:30.197950+0000","last_peered":"2024-05-09T09:17:30.197950+0000","last_clean":"2024-05-09T09:17:30.197950+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:17:30.197950+0000","last_undegraded":"2024-05-09T09:17:30.197950+0000","last_fullsized":"2024-05-09T09:17:30.197950+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":37,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197332+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:17:30.197332+0000","last_peered":"2024-05-09T09:17:30.197332+0000","last_clean":"2024-05-09T09:17:30.197332+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:17:30.197332+0000","last_undegraded":"2024-05-09T09:17:30.197332+0000","last_fullsized":"2024-05-09T09:17:30.197332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.124139+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:17:59.124139+0000","last_peered":"2024-05-09T09:17:59.124139+0000","last_clean":"2024-05-09T09:17:59.124139+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:17:59.124139+0000","last_undegraded":"2024-05-09T09:17:59.124139+0000","last_fullsized":"2024-05-09T09:17:59.124139+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275512+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:17:30.275512+0000","last_peered":"2024-05-09T09:17:30.275512+0000","last_clean":"2024-05-09T09:17:30.275512+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:17:30.275512+0000","last_undegraded":"2024-05-09T09:17:30.275512+0000","last_fullsized":"2024-05-09T09:17:30.275512+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.464422+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:17:30.464422+0000","last_peered":"2024-05-09T09:17:30.464422+0000","last_clean":"2024-05-09T09:17:30.464422+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:17:30.464422+0000","last_undegraded":"2024-05-09T09:17:30.464422+0000","last_fullsized":"2024-05-09T09:17:30.464422+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.141206+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:17:59.141206+0000","last_peered":"2024-05-09T09:17:59.141206+0000","last_clean":"2024-05-09T09:17:59.141206+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:17:59.141206+0000","last_undegraded":"2024-05-09T09:17:59.141206+0000","last_fullsized":"2024-05-09T09:17:59.141206+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275890+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:17:30.275890+0000","last_peered":"2024-05-09T09:17:30.275890+0000","last_clean":"2024-05-09T09:17:30.275890+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:17:30.275890+0000","last_undegraded":"2024-05-09T09:17:30.275890+0000","last_fullsized":"2024-05-09T09:17:30.275890+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.213737+0000","last_change":"2024-05-09T09:17:30.213383+0000","last_active":"2024-05-09T09:17:30.213737+0000","last_peered":"2024-05-09T09:17:30.213737+0000","last_clean":"2024-05-09T09:17:30.213737+0000","last_became_active":"2024-05-09T09:17:30.205007+0000","last_became_peered":"2024-05-09T09:17:30.205007+0000","last_unstale":"2024-05-09T09:17:30.213737+0000","last_undegraded":"2024-05-09T09:17:30.213737+0000","last_fullsized":"2024-05-09T09:17:30.213737+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:36.820105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":34,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.148014+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:17:59.148014+0000","last_peered":"2024-05-09T09:17:59.148014+0000","last_clean":"2024-05-09T09:17:59.148014+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:17:59.148014+0000","last_undegraded":"2024-05-09T09:17:59.148014+0000","last_fullsized":"2024-05-09T09:17:59.148014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275698+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:17:30.275698+0000","last_peered":"2024-05-09T09:17:30.275698+0000","last_clean":"2024-05-09T09:17:30.275698+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:17:30.275698+0000","last_undegraded":"2024-05-09T09:17:30.275698+0000","last_fullsized":"2024-05-09T09:17:30.275698+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202142+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:17:30.202142+0000","last_peered":"2024-05-09T09:17:30.202142+0000","last_clean":"2024-05-09T09:17:30.202142+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:17:30.202142+0000","last_undegraded":"2024-05-09T09:17:30.202142+0000","last_fullsized":"2024-05-09T09:17:30.202142+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":40,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135980+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:17:59.135980+0000","last_peered":"2024-05-09T09:17:59.135980+0000","last_clean":"2024-05-09T09:17:59.135980+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:17:59.135980+0000","last_undegraded":"2024-05-09T09:17:59.135980+0000","last_fullsized":"2024-05-09T09:17:59.135980+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198622+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:17:30.198622+0000","last_peered":"2024-05-09T09:17:30.198622+0000","last_clean":"2024-05-09T09:17:30.198622+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:17:30.198622+0000","last_undegraded":"2024-05-09T09:17:30.198622+0000","last_fullsized":"2024-05-09T09:17:30.198622+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"91'7","reported_seq":118,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.322280+0000","last_change":"2024-05-09T09:17:30.281412+0000","last_active":"2024-05-09T09:18:00.322280+0000","last_peered":"2024-05-09T09:18:00.322280+0000","last_clean":"2024-05-09T09:18:00.322280+0000","last_became_active":"2024-05-09T09:17:30.277253+0000","last_became_peered":"2024-05-09T09:17:30.277253+0000","last_unstale":"2024-05-09T09:18:00.322280+0000","last_undegraded":"2024-05-09T09:18:00.322280+0000","last_fullsized":"2024-05-09T09:18:00.322280+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:48.491205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":71,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123482+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:17:59.123482+0000","last_peered":"2024-05-09T09:17:59.123482+0000","last_clean":"2024-05-09T09:17:59.123482+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:17:59.123482+0000","last_undegraded":"2024-05-09T09:17:59.123482+0000","last_fullsized":"2024-05-09T09:17:59.123482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199031+0000","last_change":"2024-05-09T09:16:36.913312+0000","last_active":"2024-05-09T09:17:30.199031+0000","last_peered":"2024-05-09T09:17:30.199031+0000","last_clean":"2024-05-09T09:17:30.199031+0000","last_became_active":"2024-05-09T09:16:36.912908+0000","last_became_peered":"2024-05-09T09:16:36.912908+0000","last_unstale":"2024-05-09T09:17:30.199031+0000","last_undegraded":"2024-05-09T09:17:30.199031+0000","last_fullsized":"2024-05-09T09:17:30.199031+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:00:59.214206+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199926+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:17:30.199926+0000","last_peered":"2024-05-09T09:17:30.199926+0000","last_clean":"2024-05-09T09:17:30.199926+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:17:30.199926+0000","last_undegraded":"2024-05-09T09:17:30.199926+0000","last_fullsized":"2024-05-09T09:17:30.199926+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133105+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:17:59.133105+0000","last_peered":"2024-05-09T09:17:59.133105+0000","last_clean":"2024-05-09T09:17:59.133105+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:17:59.133105+0000","last_undegraded":"2024-05-09T09:17:59.133105+0000","last_fullsized":"2024-05-09T09:17:59.133105+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.277465+0000","last_change":"2024-05-09T09:17:30.277465+0000","last_active":"2024-05-09T09:17:30.277465+0000","last_peered":"2024-05-09T09:17:30.277465+0000","last_clean":"2024-05-09T09:17:30.277465+0000","last_became_active":"2024-05-09T09:17:30.277014+0000","last_became_peered":"2024-05-09T09:17:30.277014+0000","last_unstale":"2024-05-09T09:17:30.277465+0000","last_undegraded":"2024-05-09T09:17:30.277465+0000","last_fullsized":"2024-05-09T09:17:30.277465+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:36:04.346440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272444+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:17:30.272444+0000","last_peered":"2024-05-09T09:17:30.272444+0000","last_clean":"2024-05-09T09:17:30.272444+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:17:30.272444+0000","last_undegraded":"2024-05-09T09:17:30.272444+0000","last_fullsized":"2024-05-09T09:17:30.272444+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139286+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:17:59.139286+0000","last_peered":"2024-05-09T09:17:59.139286+0000","last_clean":"2024-05-09T09:17:59.139286+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:17:59.139286+0000","last_undegraded":"2024-05-09T09:17:59.139286+0000","last_fullsized":"2024-05-09T09:17:59.139286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202153+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:17:30.202153+0000","last_peered":"2024-05-09T09:17:30.202153+0000","last_clean":"2024-05-09T09:17:30.202153+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:17:30.202153+0000","last_undegraded":"2024-05-09T09:17:30.202153+0000","last_fullsized":"2024-05-09T09:17:30.202153+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200844+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:17:30.200844+0000","last_peered":"2024-05-09T09:17:30.200844+0000","last_clean":"2024-05-09T09:17:30.200844+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:17:30.200844+0000","last_undegraded":"2024-05-09T09:17:30.200844+0000","last_fullsized":"2024-05-09T09:17:30.200844+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139719+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:17:59.139719+0000","last_peered":"2024-05-09T09:17:59.139719+0000","last_clean":"2024-05-09T09:17:59.139719+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:17:59.139719+0000","last_undegraded":"2024-05-09T09:17:59.139719+0000","last_fullsized":"2024-05-09T09:17:59.139719+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.463926+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:17:30.463926+0000","last_peered":"2024-05-09T09:17:30.463926+0000","last_clean":"2024-05-09T09:17:30.463926+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:17:30.463926+0000","last_undegraded":"2024-05-09T09:17:30.463926+0000","last_fullsized":"2024-05-09T09:17:30.463926+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"91'4","reported_seq":99,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.321726+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:18:00.321726+0000","last_peered":"2024-05-09T09:18:00.321726+0000","last_clean":"2024-05-09T09:18:00.321726+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:18:00.321726+0000","last_undegraded":"2024-05-09T09:18:00.321726+0000","last_fullsized":"2024-05-09T09:18:00.321726+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":68,"num_read_kb":68,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.143239+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:17:59.143239+0000","last_peered":"2024-05-09T09:17:59.143239+0000","last_clean":"2024-05-09T09:17:59.143239+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:17:59.143239+0000","last_undegraded":"2024-05-09T09:17:59.143239+0000","last_fullsized":"2024-05-09T09:17:59.143239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196370+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:17:30.196370+0000","last_peered":"2024-05-09T09:17:30.196370+0000","last_clean":"2024-05-09T09:17:30.196370+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:17:30.196370+0000","last_undegraded":"2024-05-09T09:17:30.196370+0000","last_fullsized":"2024-05-09T09:17:30.196370+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203975+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:17:30.203975+0000","last_peered":"2024-05-09T09:17:30.203975+0000","last_clean":"2024-05-09T09:17:30.203975+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:17:30.203975+0000","last_undegraded":"2024-05-09T09:17:30.203975+0000","last_fullsized":"2024-05-09T09:17:30.203975+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134161+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:17:59.134161+0000","last_peered":"2024-05-09T09:17:59.134161+0000","last_clean":"2024-05-09T09:17:59.134161+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:17:59.134161+0000","last_undegraded":"2024-05-09T09:17:59.134161+0000","last_fullsized":"2024-05-09T09:17:59.134161+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276096+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:17:30.276096+0000","last_peered":"2024-05-09T09:17:30.276096+0000","last_clean":"2024-05-09T09:17:30.276096+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:17:30.276096+0000","last_undegraded":"2024-05-09T09:17:30.276096+0000","last_fullsized":"2024-05-09T09:17:30.276096+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203205+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:17:30.203205+0000","last_peered":"2024-05-09T09:17:30.203205+0000","last_clean":"2024-05-09T09:17:30.203205+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:17:30.203205+0000","last_undegraded":"2024-05-09T09:17:30.203205+0000","last_fullsized":"2024-05-09T09:17:30.203205+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.147609+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:17:59.147609+0000","last_peered":"2024-05-09T09:17:59.147609+0000","last_clean":"2024-05-09T09:17:59.147609+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:17:59.147609+0000","last_undegraded":"2024-05-09T09:17:59.147609+0000","last_fullsized":"2024-05-09T09:17:59.147609+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196970+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:17:30.196970+0000","last_peered":"2024-05-09T09:17:30.196970+0000","last_clean":"2024-05-09T09:17:30.196970+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:17:30.196970+0000","last_undegraded":"2024-05-09T09:17:30.196970+0000","last_fullsized":"2024-05-09T09:17:30.196970+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"91'35","reported_seq":207,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:01.323220+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:18:01.323220+0000","last_peered":"2024-05-09T09:18:01.323220+0000","last_clean":"2024-05-09T09:18:01.323220+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:18:01.323220+0000","last_undegraded":"2024-05-09T09:18:01.323220+0000","last_fullsized":"2024-05-09T09:18:01.323220+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63605,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":145,"num_read_kb":135,"num_write":36,"num_write_kb":79,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.132631+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:17:59.132631+0000","last_peered":"2024-05-09T09:17:59.132631+0000","last_clean":"2024-05-09T09:17:59.132631+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:17:59.132631+0000","last_undegraded":"2024-05-09T09:17:59.132631+0000","last_fullsized":"2024-05-09T09:17:59.132631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203677+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:17:30.203677+0000","last_peered":"2024-05-09T09:17:30.203677+0000","last_clean":"2024-05-09T09:17:30.203677+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:17:30.203677+0000","last_undegraded":"2024-05-09T09:17:30.203677+0000","last_fullsized":"2024-05-09T09:17:30.203677+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198267+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:17:30.198267+0000","last_peered":"2024-05-09T09:17:30.198267+0000","last_clean":"2024-05-09T09:17:30.198267+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:17:30.198267+0000","last_undegraded":"2024-05-09T09:17:30.198267+0000","last_fullsized":"2024-05-09T09:17:30.198267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.128463+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:17:59.128463+0000","last_peered":"2024-05-09T09:17:59.128463+0000","last_clean":"2024-05-09T09:17:59.128463+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:17:59.128463+0000","last_undegraded":"2024-05-09T09:17:59.128463+0000","last_fullsized":"2024-05-09T09:17:59.128463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200229+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:17:30.200229+0000","last_peered":"2024-05-09T09:17:30.200229+0000","last_clean":"2024-05-09T09:17:30.200229+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:17:30.200229+0000","last_undegraded":"2024-05-09T09:17:30.200229+0000","last_fullsized":"2024-05-09T09:17:30.200229+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201267+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:17:30.201267+0000","last_peered":"2024-05-09T09:17:30.201267+0000","last_clean":"2024-05-09T09:17:30.201267+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:17:30.201267+0000","last_undegraded":"2024-05-09T09:17:30.201267+0000","last_fullsized":"2024-05-09T09:17:30.201267+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.138217+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:17:59.138217+0000","last_peered":"2024-05-09T09:17:59.138217+0000","last_clean":"2024-05-09T09:17:59.138217+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:17:59.138217+0000","last_undegraded":"2024-05-09T09:17:59.138217+0000","last_fullsized":"2024-05-09T09:17:59.138217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.465897+0000","last_change":"2024-05-09T09:17:30.465425+0000","last_active":"2024-05-09T09:17:30.465897+0000","last_peered":"2024-05-09T09:17:30.465897+0000","last_clean":"2024-05-09T09:17:30.465897+0000","last_became_active":"2024-05-09T09:17:30.465043+0000","last_became_peered":"2024-05-09T09:17:30.465043+0000","last_unstale":"2024-05-09T09:17:30.465897+0000","last_undegraded":"2024-05-09T09:17:30.465897+0000","last_fullsized":"2024-05-09T09:17:30.465897+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:16:54.706845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:00.321216+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:18:00.321216+0000","last_peered":"2024-05-09T09:18:00.321216+0000","last_clean":"2024-05-09T09:18:00.321216+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:18:00.321216+0000","last_undegraded":"2024-05-09T09:18:00.321216+0000","last_fullsized":"2024-05-09T09:18:00.321216+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":72,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133890+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:17:59.133890+0000","last_peered":"2024-05-09T09:17:59.133890+0000","last_clean":"2024-05-09T09:17:59.133890+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:17:59.133890+0000","last_undegraded":"2024-05-09T09:17:59.133890+0000","last_fullsized":"2024-05-09T09:17:59.133890+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272921+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:17:30.272921+0000","last_peered":"2024-05-09T09:17:30.272921+0000","last_clean":"2024-05-09T09:17:30.272921+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:17:30.272921+0000","last_undegraded":"2024-05-09T09:17:30.272921+0000","last_fullsized":"2024-05-09T09:17:30.272921+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":42,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200420+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:17:30.200420+0000","last_peered":"2024-05-09T09:17:30.200420+0000","last_clean":"2024-05-09T09:17:30.200420+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:17:30.200420+0000","last_undegraded":"2024-05-09T09:17:30.200420+0000","last_fullsized":"2024-05-09T09:17:30.200420+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135691+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:17:59.135691+0000","last_peered":"2024-05-09T09:17:59.135691+0000","last_clean":"2024-05-09T09:17:59.135691+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:17:59.135691+0000","last_undegraded":"2024-05-09T09:17:59.135691+0000","last_fullsized":"2024-05-09T09:17:59.135691+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200158+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:17:30.200158+0000","last_peered":"2024-05-09T09:17:30.200158+0000","last_clean":"2024-05-09T09:17:30.200158+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:17:30.200158+0000","last_undegraded":"2024-05-09T09:17:30.200158+0000","last_fullsized":"2024-05-09T09:17:30.200158+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203431+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:17:30.203431+0000","last_peered":"2024-05-09T09:17:30.203431+0000","last_clean":"2024-05-09T09:17:30.203431+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:17:30.203431+0000","last_undegraded":"2024-05-09T09:17:30.203431+0000","last_fullsized":"2024-05-09T09:17:30.203431+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.125336+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:17:59.125336+0000","last_peered":"2024-05-09T09:17:59.125336+0000","last_clean":"2024-05-09T09:17:59.125336+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:17:59.125336+0000","last_undegraded":"2024-05-09T09:17:59.125336+0000","last_fullsized":"2024-05-09T09:17:59.125336+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.201202+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:17:30.201202+0000","last_peered":"2024-05-09T09:17:30.201202+0000","last_clean":"2024-05-09T09:17:30.201202+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:17:30.201202+0000","last_undegraded":"2024-05-09T09:17:30.201202+0000","last_fullsized":"2024-05-09T09:17:30.201202+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.196900+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:17:30.196900+0000","last_peered":"2024-05-09T09:17:30.196900+0000","last_clean":"2024-05-09T09:17:30.196900+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:17:30.196900+0000","last_undegraded":"2024-05-09T09:17:30.196900+0000","last_fullsized":"2024-05-09T09:17:30.196900+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.127717+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:17:59.127717+0000","last_peered":"2024-05-09T09:17:59.127717+0000","last_clean":"2024-05-09T09:17:59.127717+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:17:59.127717+0000","last_undegraded":"2024-05-09T09:17:59.127717+0000","last_fullsized":"2024-05-09T09:17:59.127717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272710+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:17:30.272710+0000","last_peered":"2024-05-09T09:17:30.272710+0000","last_clean":"2024-05-09T09:17:30.272710+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:17:30.272710+0000","last_undegraded":"2024-05-09T09:17:30.272710+0000","last_fullsized":"2024-05-09T09:17:30.272710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.204121+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:17:30.204121+0000","last_peered":"2024-05-09T09:17:30.204121+0000","last_clean":"2024-05-09T09:17:30.204121+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:17:30.204121+0000","last_undegraded":"2024-05-09T09:17:30.204121+0000","last_fullsized":"2024-05-09T09:17:30.204121+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":38,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.131020+0000","last_change":"2024-05-09T09:17:30.203699+0000","last_active":"2024-05-09T09:17:59.131020+0000","last_peered":"2024-05-09T09:17:59.131020+0000","last_clean":"2024-05-09T09:17:59.131020+0000","last_became_active":"2024-05-09T09:17:30.203312+0000","last_became_peered":"2024-05-09T09:17:30.203312+0000","last_unstale":"2024-05-09T09:17:59.131020+0000","last_undegraded":"2024-05-09T09:17:59.131020+0000","last_fullsized":"2024-05-09T09:17:59.131020+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272709+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:17:30.272709+0000","last_peered":"2024-05-09T09:17:30.272709+0000","last_clean":"2024-05-09T09:17:30.272709+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:17:30.272709+0000","last_undegraded":"2024-05-09T09:17:30.272709+0000","last_fullsized":"2024-05-09T09:17:30.272709+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.115942+0000","last_change":"2024-05-09T09:17:30.276150+0000","last_active":"2024-05-09T09:17:59.115942+0000","last_peered":"2024-05-09T09:17:59.115942+0000","last_clean":"2024-05-09T09:17:59.115942+0000","last_became_active":"2024-05-09T09:17:30.275868+0000","last_became_peered":"2024-05-09T09:17:30.275868+0000","last_unstale":"2024-05-09T09:17:59.115942+0000","last_undegraded":"2024-05-09T09:17:59.115942+0000","last_fullsized":"2024-05-09T09:17:59.115942+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:18:08.892380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":100,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:18:01.322008+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:18:01.322008+0000","last_peered":"2024-05-09T09:18:01.322008+0000","last_clean":"2024-05-09T09:18:01.322008+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:18:01.322008+0000","last_undegraded":"2024-05-09T09:18:01.322008+0000","last_fullsized":"2024-05-09T09:18:01.322008+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":70,"num_read_kb":70,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203956+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:17:30.203956+0000","last_peered":"2024-05-09T09:17:30.203956+0000","last_clean":"2024-05-09T09:17:30.203956+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:17:30.203956+0000","last_undegraded":"2024-05-09T09:17:30.203956+0000","last_fullsized":"2024-05-09T09:17:30.203956+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.131068+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:17:59.131068+0000","last_peered":"2024-05-09T09:17:59.131068+0000","last_clean":"2024-05-09T09:17:59.131068+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:17:59.131068+0000","last_undegraded":"2024-05-09T09:17:59.131068+0000","last_fullsized":"2024-05-09T09:17:59.131068+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276021+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:17:30.276021+0000","last_peered":"2024-05-09T09:17:30.276021+0000","last_clean":"2024-05-09T09:17:30.276021+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:17:30.276021+0000","last_undegraded":"2024-05-09T09:17:30.276021+0000","last_fullsized":"2024-05-09T09:17:30.276021+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":352,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203272+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:17:30.203272+0000","last_peered":"2024-05-09T09:17:30.203272+0000","last_clean":"2024-05-09T09:17:30.203272+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:17:30.203272+0000","last_undegraded":"2024-05-09T09:17:30.203272+0000","last_fullsized":"2024-05-09T09:17:30.203272+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203358+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:17:30.203358+0000","last_peered":"2024-05-09T09:17:30.203358+0000","last_clean":"2024-05-09T09:17:30.203358+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:17:30.203358+0000","last_undegraded":"2024-05-09T09:17:30.203358+0000","last_fullsized":"2024-05-09T09:17:30.203358+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.130432+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:17:59.130432+0000","last_peered":"2024-05-09T09:17:59.130432+0000","last_clean":"2024-05-09T09:17:59.130432+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:17:59.130432+0000","last_undegraded":"2024-05-09T09:17:59.130432+0000","last_fullsized":"2024-05-09T09:17:59.130432+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203451+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:17:30.203451+0000","last_peered":"2024-05-09T09:17:30.203451+0000","last_clean":"2024-05-09T09:17:30.203451+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:17:30.203451+0000","last_undegraded":"2024-05-09T09:17:30.203451+0000","last_fullsized":"2024-05-09T09:17:30.203451+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200099+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:17:30.200099+0000","last_peered":"2024-05-09T09:17:30.200099+0000","last_clean":"2024-05-09T09:17:30.200099+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:17:30.200099+0000","last_undegraded":"2024-05-09T09:17:30.200099+0000","last_fullsized":"2024-05-09T09:17:30.200099+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":15,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.208425+0000","last_change":"2024-05-09T09:17:30.208053+0000","last_active":"2024-05-09T09:17:30.208425+0000","last_peered":"2024-05-09T09:17:30.208425+0000","last_clean":"2024-05-09T09:17:30.208425+0000","last_became_active":"2024-05-09T09:17:30.203628+0000","last_became_peered":"2024-05-09T09:17:30.203628+0000","last_unstale":"2024-05-09T09:17:30.208425+0000","last_undegraded":"2024-05-09T09:17:30.208425+0000","last_fullsized":"2024-05-09T09:17:30.208425+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:37.871338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.127252+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:17:59.127252+0000","last_peered":"2024-05-09T09:17:59.127252+0000","last_clean":"2024-05-09T09:17:59.127252+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:17:59.127252+0000","last_undegraded":"2024-05-09T09:17:59.127252+0000","last_fullsized":"2024-05-09T09:17:59.127252+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.463898+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:17:30.463898+0000","last_peered":"2024-05-09T09:17:30.463898+0000","last_clean":"2024-05-09T09:17:30.463898+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:17:30.463898+0000","last_undegraded":"2024-05-09T09:17:30.463898+0000","last_fullsized":"2024-05-09T09:17:30.463898+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"91'9","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197199+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:17:30.197199+0000","last_peered":"2024-05-09T09:17:30.197199+0000","last_clean":"2024-05-09T09:17:30.197199+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:17:30.197199+0000","last_undegraded":"2024-05-09T09:17:30.197199+0000","last_fullsized":"2024-05-09T09:17:30.197199+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139503+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:17:59.139503+0000","last_peered":"2024-05-09T09:17:59.139503+0000","last_clean":"2024-05-09T09:17:59.139503+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:17:59.139503+0000","last_undegraded":"2024-05-09T09:17:59.139503+0000","last_fullsized":"2024-05-09T09:17:59.139503+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.198637+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:17:30.198637+0000","last_peered":"2024-05-09T09:17:30.198637+0000","last_clean":"2024-05-09T09:17:30.198637+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:17:30.198637+0000","last_undegraded":"2024-05-09T09:17:30.198637+0000","last_fullsized":"2024-05-09T09:17:30.198637+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276292+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:17:30.276292+0000","last_peered":"2024-05-09T09:17:30.276292+0000","last_clean":"2024-05-09T09:17:30.276292+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:17:30.276292+0000","last_undegraded":"2024-05-09T09:17:30.276292+0000","last_fullsized":"2024-05-09T09:17:30.276292+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139913+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:17:59.139913+0000","last_peered":"2024-05-09T09:17:59.139913+0000","last_clean":"2024-05-09T09:17:59.139913+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:17:59.139913+0000","last_undegraded":"2024-05-09T09:17:59.139913+0000","last_fullsized":"2024-05-09T09:17:59.139913+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275865+0000","last_change":"2024-05-09T09:16:36.967460+0000","last_active":"2024-05-09T09:17:30.275865+0000","last_peered":"2024-05-09T09:17:30.275865+0000","last_clean":"2024-05-09T09:17:30.275865+0000","last_became_active":"2024-05-09T09:16:36.966966+0000","last_became_peered":"2024-05-09T09:16:36.966966+0000","last_unstale":"2024-05-09T09:17:30.275865+0000","last_undegraded":"2024-05-09T09:17:30.275865+0000","last_fullsized":"2024-05-09T09:17:30.275865+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:32:14.954608+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200436+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:17:30.200436+0000","last_peered":"2024-05-09T09:17:30.200436+0000","last_clean":"2024-05-09T09:17:30.200436+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:17:30.200436+0000","last_undegraded":"2024-05-09T09:17:30.200436+0000","last_fullsized":"2024-05-09T09:17:30.200436+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.107450+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:17:59.107450+0000","last_peered":"2024-05-09T09:17:59.107450+0000","last_clean":"2024-05-09T09:17:59.107450+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:17:59.107450+0000","last_undegraded":"2024-05-09T09:17:59.107450+0000","last_fullsized":"2024-05-09T09:17:59.107450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197028+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:17:30.197028+0000","last_peered":"2024-05-09T09:17:30.197028+0000","last_clean":"2024-05-09T09:17:30.197028+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:17:30.197028+0000","last_undegraded":"2024-05-09T09:17:30.197028+0000","last_fullsized":"2024-05-09T09:17:30.197028+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.464027+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:17:30.464027+0000","last_peered":"2024-05-09T09:17:30.464027+0000","last_clean":"2024-05-09T09:17:30.464027+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:17:30.464027+0000","last_undegraded":"2024-05-09T09:17:30.464027+0000","last_fullsized":"2024-05-09T09:17:30.464027+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133860+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:17:59.133860+0000","last_peered":"2024-05-09T09:17:59.133860+0000","last_clean":"2024-05-09T09:17:59.133860+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:17:59.133860+0000","last_undegraded":"2024-05-09T09:17:59.133860+0000","last_fullsized":"2024-05-09T09:17:59.133860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.276003+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:17:30.276003+0000","last_peered":"2024-05-09T09:17:30.276003+0000","last_clean":"2024-05-09T09:17:30.276003+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:17:30.276003+0000","last_undegraded":"2024-05-09T09:17:30.276003+0000","last_fullsized":"2024-05-09T09:17:30.276003+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.197934+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:17:30.197934+0000","last_peered":"2024-05-09T09:17:30.197934+0000","last_clean":"2024-05-09T09:17:30.197934+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:17:30.197934+0000","last_undegraded":"2024-05-09T09:17:30.197934+0000","last_fullsized":"2024-05-09T09:17:30.197934+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.135332+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:17:59.135332+0000","last_peered":"2024-05-09T09:17:59.135332+0000","last_clean":"2024-05-09T09:17:59.135332+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:17:59.135332+0000","last_undegraded":"2024-05-09T09:17:59.135332+0000","last_fullsized":"2024-05-09T09:17:59.135332+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.275860+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:17:30.275860+0000","last_peered":"2024-05-09T09:17:30.275860+0000","last_clean":"2024-05-09T09:17:30.275860+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:17:30.275860+0000","last_undegraded":"2024-05-09T09:17:30.275860+0000","last_fullsized":"2024-05-09T09:17:30.275860+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200902+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:17:30.200902+0000","last_peered":"2024-05-09T09:17:30.200902+0000","last_clean":"2024-05-09T09:17:30.200902+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:17:30.200902+0000","last_undegraded":"2024-05-09T09:17:30.200902+0000","last_fullsized":"2024-05-09T09:17:30.200902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.146688+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:17:59.146688+0000","last_peered":"2024-05-09T09:17:59.146688+0000","last_clean":"2024-05-09T09:17:59.146688+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:17:59.146688+0000","last_undegraded":"2024-05-09T09:17:59.146688+0000","last_fullsized":"2024-05-09T09:17:59.146688+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200394+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:17:30.200394+0000","last_peered":"2024-05-09T09:17:30.200394+0000","last_clean":"2024-05-09T09:17:30.200394+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:17:30.200394+0000","last_undegraded":"2024-05-09T09:17:30.200394+0000","last_fullsized":"2024-05-09T09:17:30.200394+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.199138+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:17:30.199138+0000","last_peered":"2024-05-09T09:17:30.199138+0000","last_clean":"2024-05-09T09:17:30.199138+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:17:30.199138+0000","last_undegraded":"2024-05-09T09:17:30.199138+0000","last_fullsized":"2024-05-09T09:17:30.199138+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.146971+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:17:59.146971+0000","last_peered":"2024-05-09T09:17:59.146971+0000","last_clean":"2024-05-09T09:17:59.146971+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:17:59.146971+0000","last_undegraded":"2024-05-09T09:17:59.146971+0000","last_fullsized":"2024-05-09T09:17:59.146971+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.202235+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:17:30.202235+0000","last_peered":"2024-05-09T09:17:30.202235+0000","last_clean":"2024-05-09T09:17:30.202235+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:17:30.202235+0000","last_undegraded":"2024-05-09T09:17:30.202235+0000","last_fullsized":"2024-05-09T09:17:30.202235+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.203366+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:17:30.203366+0000","last_peered":"2024-05-09T09:17:30.203366+0000","last_clean":"2024-05-09T09:17:30.203366+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:17:30.203366+0000","last_undegraded":"2024-05-09T09:17:30.203366+0000","last_fullsized":"2024-05-09T09:17:30.203366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.140250+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:17:59.140250+0000","last_peered":"2024-05-09T09:17:59.140250+0000","last_clean":"2024-05-09T09:17:59.140250+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:17:59.140250+0000","last_undegraded":"2024-05-09T09:17:59.140250+0000","last_fullsized":"2024-05-09T09:17:59.140250+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.200378+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:17:30.200378+0000","last_peered":"2024-05-09T09:17:30.200378+0000","last_clean":"2024-05-09T09:17:30.200378+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:17:30.200378+0000","last_undegraded":"2024-05-09T09:17:30.200378+0000","last_fullsized":"2024-05-09T09:17:30.200378+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:30.272924+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:17:30.272924+0000","last_peered":"2024-05-09T09:17:30.272924+0000","last_clean":"2024-05-09T09:17:30.272924+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:17:30.272924+0000","last_undegraded":"2024-05-09T09:17:30.272924+0000","last_fullsized":"2024-05-09T09:17:30.272924+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.136500+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:17:59.136500+0000","last_peered":"2024-05-09T09:17:59.136500+0000","last_clean":"2024-05-09T09:17:59.136500+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:17:59.136500+0000","last_undegraded":"2024-05-09T09:17:59.136500+0000","last_fullsized":"2024-05-09T09:17:59.136500+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134194+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:17:59.134194+0000","last_peered":"2024-05-09T09:17:59.134194+0000","last_clean":"2024-05-09T09:17:59.134194+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:17:59.134194+0000","last_undegraded":"2024-05-09T09:17:59.134194+0000","last_fullsized":"2024-05-09T09:17:59.134194+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":36,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.130318+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:17:59.130318+0000","last_peered":"2024-05-09T09:17:59.130318+0000","last_clean":"2024-05-09T09:17:59.130318+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:17:59.130318+0000","last_undegraded":"2024-05-09T09:17:59.130318+0000","last_fullsized":"2024-05-09T09:17:59.130318+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133198+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:17:59.133198+0000","last_peered":"2024-05-09T09:17:59.133198+0000","last_clean":"2024-05-09T09:17:59.133198+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:17:59.133198+0000","last_undegraded":"2024-05-09T09:17:59.133198+0000","last_fullsized":"2024-05-09T09:17:59.133198+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.134304+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:17:59.134304+0000","last_peered":"2024-05-09T09:17:59.134304+0000","last_clean":"2024-05-09T09:17:59.134304+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:17:59.134304+0000","last_undegraded":"2024-05-09T09:17:59.134304+0000","last_fullsized":"2024-05-09T09:17:59.134304+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.132948+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:17:59.132948+0000","last_peered":"2024-05-09T09:17:59.132948+0000","last_clean":"2024-05-09T09:17:59.132948+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:17:59.132948+0000","last_undegraded":"2024-05-09T09:17:59.132948+0000","last_fullsized":"2024-05-09T09:17:59.132948+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123579+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:17:59.123579+0000","last_peered":"2024-05-09T09:17:59.123579+0000","last_clean":"2024-05-09T09:17:59.123579+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:17:59.123579+0000","last_undegraded":"2024-05-09T09:17:59.123579+0000","last_fullsized":"2024-05-09T09:17:59.123579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.145520+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:17:59.145520+0000","last_peered":"2024-05-09T09:17:59.145520+0000","last_clean":"2024-05-09T09:17:59.145520+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:17:59.145520+0000","last_undegraded":"2024-05-09T09:17:59.145520+0000","last_fullsized":"2024-05-09T09:17:59.145520+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.126935+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:17:59.126935+0000","last_peered":"2024-05-09T09:17:59.126935+0000","last_clean":"2024-05-09T09:17:59.126935+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:17:59.126935+0000","last_undegraded":"2024-05-09T09:17:59.126935+0000","last_fullsized":"2024-05-09T09:17:59.126935+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.123443+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:17:59.123443+0000","last_peered":"2024-05-09T09:17:59.123443+0000","last_clean":"2024-05-09T09:17:59.123443+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:17:59.123443+0000","last_undegraded":"2024-05-09T09:17:59.123443+0000","last_fullsized":"2024-05-09T09:17:59.123443+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.139102+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:17:59.139102+0000","last_peered":"2024-05-09T09:17:59.139102+0000","last_clean":"2024-05-09T09:17:59.139102+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:17:59.139102+0000","last_undegraded":"2024-05-09T09:17:59.139102+0000","last_fullsized":"2024-05-09T09:17:59.139102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.144660+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:17:59.144660+0000","last_peered":"2024-05-09T09:17:59.144660+0000","last_clean":"2024-05-09T09:17:59.144660+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:17:59.144660+0000","last_undegraded":"2024-05-09T09:17:59.144660+0000","last_fullsized":"2024-05-09T09:17:59.144660+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.133007+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:17:59.133007+0000","last_peered":"2024-05-09T09:17:59.133007+0000","last_clean":"2024-05-09T09:17:59.133007+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:17:59.133007+0000","last_undegraded":"2024-05-09T09:17:59.133007+0000","last_fullsized":"2024-05-09T09:17:59.133007+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-05-09T09:17:59.127122+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:17:59.127122+0000","last_peered":"2024-05-09T09:17:59.127122+0000","last_clean":"2024-05-09T09:17:59.127122+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:17:59.127122+0000","last_undegraded":"2024-05-09T09:17:59.127122+0000","last_fullsized":"2024-05-09T09:17:59.127122+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":67747,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":438,"num_read_kb":292,"num_write":167,"num_write_kb":120,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1035,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":417792,"data_stored":203241,"data_compressed":4887,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":145,"ondisk_log_size":145,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383712,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32152,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713000,"statfs":{"total":95995035648,"available":95962112000,"internally_reserved":0,"allocated":999424,"data_stored":500766,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645349,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1151617,"data_compressed":57955,"data_compressed_allocated":77824,"data_compressed_original":655360,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808874,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":505076,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070512,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28000,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717152,"statfs":{"total":95995035648,"available":95966363648,"internally_reserved":0,"allocated":1007616,"data_stored":500878,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332149,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32212,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712940,"statfs":{"total":95995035648,"available":95962050560,"internally_reserved":0,"allocated":1060864,"data_stored":1096950,"data_compressed":56326,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495674,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":496156,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574930,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28048,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717104,"statfs":{"total":95995035648,"available":95966314496,"internally_reserved":0,"allocated":1056768,"data_stored":559671,"data_compressed":22240,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738456,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28008,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717144,"statfs":{"total":95995035648,"available":95966355456,"internally_reserved":0,"allocated":1015808,"data_stored":497544,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411406,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28064,"kb_used_data":1048,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717088,"statfs":{"total":95995035648,"available":95966298112,"internally_reserved":0,"allocated":1073152,"data_stored":560399,"data_compressed":22240,"data_compressed_allocated":32768,"data_compressed_original":65536,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182472,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":496350,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920836,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28004,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717148,"statfs":{"total":95995035648,"available":95966359552,"internally_reserved":0,"allocated":1011712,"data_stored":501960,"data_compressed":20611,"data_compressed_allocated":28672,"data_compressed_original":57344,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757310,"num_pgs":45,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32220,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712932,"statfs":{"total":95995035648,"available":95962042368,"internally_reserved":0,"allocated":1069056,"data_stored":1092538,"data_compressed":56326,"data_compressed_allocated":73728,"data_compressed_original":647168,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63605,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64333,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2008,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65183,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:18:07.831 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-05-09T09:18:07.831 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-05-09T09:18:07.831 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-05-09T09:18:07.832 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph health --format=json 2024-05-09T09:18:08.074 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:08.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:08 smithi069 ceph-mon[30123]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:08.446 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:08 smithi069 ceph-mon[30123]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:08.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:08 smithi163 ceph-mon[34354]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:08 smithi163 ceph-mon[34354]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:08 smithi103 ceph-mon[34421]: from='client.15117 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:08 smithi103 ceph-mon[34421]: pgmap v340: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:09.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 -- 172.21.15.69:0/3861254755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 msgr2=0x7f208806b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 --2- 172.21.15.69:0/3861254755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208806b430 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f2078009990 tx=0x7f2078009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 -- 172.21.15.69:0/3861254755 shutdown_connections 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 --2- 172.21.15.69:0/3861254755 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088104d50 0x7f2088105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 --2- 172.21.15.69:0/3861254755 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088107240 0x7f208806aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 --2- 172.21.15.69:0/3861254755 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208806b430 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 -- 172.21.15.69:0/3861254755 >> 172.21.15.69:0/3861254755 conn(0x7f20880781f0 msgr2=0x7f2088078600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 -- 172.21.15.69:0/3861254755 shutdown_connections 2024-05-09T09:18:09.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.693+0000 7f208d2f0700 1 -- 172.21.15.69:0/3861254755 wait complete. 2024-05-09T09:18:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 Processor -- start 2024-05-09T09:18:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 -- start start 2024-05-09T09:18:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088104d50 0x7f208819f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:09.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 0x7f20881a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2088118ca0 con 0x7f208806afb0 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2088118b20 con 0x7f2088104d50 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f208d2f0700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2088118e20 con 0x7f2088107240 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.694+0000 7f2086ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57288/0 (socket says 172.21.15.69:57288) 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 -- 172.21.15.69:0/2446400348 learned_addr learned my addr 172.21.15.69:0/2446400348 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f20867fc700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088104d50 0x7f208819f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f20877fe700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 0x7f20881a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:09.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 -- 172.21.15.69:0/2446400348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 msgr2=0x7f20881a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 0x7f20881a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 -- 172.21.15.69:0/2446400348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088104d50 msgr2=0x7f208819f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088104d50 0x7f208819f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 -- 172.21.15.69:0/2446400348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20881a4160 con 0x7f208806afb0 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f20877fe700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 0x7f20881a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f2086ffd700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f207c00d950 tx=0x7f207c00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c016070 con 0x7f208806afb0 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20881a4450 con 0x7f208806afb0 2024-05-09T09:18:09.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f207c009e20 con 0x7f208806afb0 2024-05-09T09:18:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f207c00f620 con 0x7f208806afb0 2024-05-09T09:18:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.695+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20881abcd0 con 0x7f208806afb0 2024-05-09T09:18:09.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.696+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2088069010 con 0x7f208806afb0 2024-05-09T09:18:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.698+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f207c019070 con 0x7f208806afb0 2024-05-09T09:18:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.699+0000 7f206ffff700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 0x7f20700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.699+0000 7f20867fc700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 0x7f20700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:09.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.699+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f207c0cf2e0 con 0x7f208806afb0 2024-05-09T09:18:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.699+0000 7f20867fc700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 0x7f20700a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f20780098e0 tx=0x7f2078005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:09.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.702+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f207c09bf00 con 0x7f208806afb0 2024-05-09T09:18:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.943+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) -- 0x7f20881abf00 con 0x7f208806afb0 2024-05-09T09:18:09.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.943+0000 7f206ffff700 1 -- 172.21.15.69:0/2446400348 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) ==== 72+0+46 (secure 0 0 0) 0x7f207c09b800 con 0x7f208806afb0 2024-05-09T09:18:09.943 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:09.944 INFO:teuthology.orchestra.run.smithi069.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 msgr2=0x7f20700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 0x7f20700a50e0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f20780098e0 tx=0x7f2078005960 comp rx=0 tx=0).stop 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 msgr2=0x7f208819f150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f207c00d950 tx=0x7f207c00dc60 comp rx=0 tx=0).stop 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 shutdown_connections 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2088107240 0x7f20881a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f208806afb0 0x7f208819f150 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20700a2c20 0x7f20700a50e0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 --2- 172.21.15.69:0/2446400348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2088104d50 0x7f208819f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.945+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 >> 172.21.15.69:0/2446400348 conn(0x7f20880781f0 msgr2=0x7f2088103d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.946+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 shutdown_connections 2024-05-09T09:18:09.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:09.946+0000 7f208d2f0700 1 -- 172.21.15.69:0/2446400348 wait complete. 2024-05-09T09:18:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:10 smithi069 ceph-mon[30123]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:10 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2446400348' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T09:18:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:10 smithi163 ceph-mon[34354]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:10 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2446400348' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T09:18:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:10 smithi103 ceph-mon[34421]: pgmap v341: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:10 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2446400348' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-05-09T09:18:10.691 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-05-09T09:18:10.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:10.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 -- 172.21.15.69:0/3910417788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c101880 msgr2=0x7f054c101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:10.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 --2- 172.21.15.69:0/3910417788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c101880 0x7f054c101ca0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f0544009ca0 tx=0x7f054402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 -- 172.21.15.69:0/3910417788 shutdown_connections 2024-05-09T09:18:10.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 --2- 172.21.15.69:0/3910417788 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c103430 0x7f054c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 --2- 172.21.15.69:0/3910417788 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c102a70 0x7f054c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 --2- 172.21.15.69:0/3910417788 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c101880 0x7f054c101ca0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 -- 172.21.15.69:0/3910417788 >> 172.21.15.69:0/3910417788 conn(0x7f054c0fce10 msgr2=0x7f054c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 -- 172.21.15.69:0/3910417788 shutdown_connections 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.796+0000 7f0553ee1640 1 -- 172.21.15.69:0/3910417788 wait complete. 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 Processor -- start 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 -- start start 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 0x7f054c197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 0x7f054c19f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f054c10d4f0 con 0x7f054c102a70 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f054c10d370 con 0x7f054c101880 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0553ee1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f054c10d670 con 0x7f054c103430 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f055147c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:10.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f055147c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57300/0 (socket says 172.21.15.69:57300) 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f055147c640 1 -- 172.21.15.69:0/1232841508 learned_addr learned my addr 172.21.15.69:0/1232841508 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f055247e640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 0x7f054c19f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.797+0000 7f0551c7d640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 0x7f054c197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 -- 172.21.15.69:0/1232841508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 msgr2=0x7f054c19f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 0x7f054c19f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 -- 172.21.15.69:0/1232841508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 msgr2=0x7f054c197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 0x7f054c197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 -- 172.21.15.69:0/1232841508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f054c19f7c0 con 0x7f054c102a70 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055247e640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 0x7f054c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f0551c7d640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 0x7f054c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f055147c640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f054000e9f0 tx=0x7f054000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f054000cd80 con 0x7f054c102a70 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0540005500 con 0x7f054c102a70 2024-05-09T09:18:10.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f054c19fab0 con 0x7f054c102a70 2024-05-09T09:18:10.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0540010690 con 0x7f054c102a70 2024-05-09T09:18:10.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.798+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f054c1a0000 con 0x7f054c102a70 2024-05-09T09:18:10.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.799+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f054c108bc0 con 0x7f054c102a70 2024-05-09T09:18:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.800+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f05400046c0 con 0x7f054c102a70 2024-05-09T09:18:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.801+0000 7f054affd640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 0x7f05340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.801+0000 7f0551c7d640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 0x7f05340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.801+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f05400cf870 con 0x7f054c102a70 2024-05-09T09:18:10.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.801+0000 7f0551c7d640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 0x7f05340a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0544006010 tx=0x7f0544031bb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:10.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.805+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f054009c490 con 0x7f054c102a70 2024-05-09T09:18:10.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.982+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f054c062430 con 0x7f054c102a70 2024-05-09T09:18:10.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.984+0000 7f054affd640 1 -- 172.21.15.69:0/1232841508 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24614 (secure 0 0 0) 0x7f054009bd90 con 0x7f054c102a70 2024-05-09T09:18:10.984 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:10.984 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:17:30.193538+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:10.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 msgr2=0x7f05340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:10.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 0x7f05340a50d0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f0544006010 tx=0x7f0544031bb0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 msgr2=0x7f054c1980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:10.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f054000e9f0 tx=0x7f054000edf0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 shutdown_connections 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f05340a2c10 0x7f05340a50d0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f054c103430 0x7f054c19f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f054c102a70 0x7f054c1980e0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 --2- 172.21.15.69:0/1232841508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f054c101880 0x7f054c197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 >> 172.21.15.69:0/1232841508 conn(0x7f054c0fce10 msgr2=0x7f054c105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 shutdown_connections 2024-05-09T09:18:10.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:10.986+0000 7f0553ee1640 1 -- 172.21.15.69:0/1232841508 wait complete. 2024-05-09T09:18:10.998 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:10.998 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 -- 172.21.15.69:0/3894560364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c102b10 msgr2=0x7f420c102f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 --2- 172.21.15.69:0/3894560364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c102b10 0x7f420c102f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f4200009930 tx=0x7f420002a420 comp rx=0 tx=0).stop 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 -- 172.21.15.69:0/3894560364 shutdown_connections 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 --2- 172.21.15.69:0/3894560364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 --2- 172.21.15.69:0/3894560364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c102b10 0x7f420c102f90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.103 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 --2- 172.21.15.69:0/3894560364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c0ffb40 0x7f420c0fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 -- 172.21.15.69:0/3894560364 >> 172.21.15.69:0/3894560364 conn(0x7f420c0fb470 msgr2=0x7f420c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 -- 172.21.15.69:0/3894560364 shutdown_connections 2024-05-09T09:18:11.104 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.103+0000 7f4210e3f640 1 -- 172.21.15.69:0/3894560364 wait complete. 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 Processor -- start 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 -- start start 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 0x7f420c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10d570 con 0x7f420c0ffb40 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10d3f0 con 0x7f420c102b10 2024-05-09T09:18:11.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4210e3f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f420c10d6f0 con 0x7f420c1034d0 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57324/0 (socket says 172.21.15.69:57324) 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 -- 172.21.15.69:0/722772872 learned_addr learned my addr 172.21.15.69:0/722772872 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4209d9b640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 0x7f420c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420ad9d640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 -- 172.21.15.69:0/722772872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 msgr2=0x7f420c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 -- 172.21.15.69:0/722772872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 msgr2=0x7f420c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 0x7f420c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 -- 172.21.15.69:0/722772872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f420c19f840 con 0x7f420c0ffb40 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f4209d9b640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 0x7f420c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420ad9d640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.104+0000 7f420a59c640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00d770 tx=0x7f41fc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.105+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc017070 con 0x7f420c0ffb40 2024-05-09T09:18:11.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.105+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f41fc009c30 con 0x7f420c0ffb40 2024-05-09T09:18:11.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.105+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f41fc003dd0 con 0x7f420c0ffb40 2024-05-09T09:18:11.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.105+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f420c19fb30 con 0x7f420c0ffb40 2024-05-09T09:18:11.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.105+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f420c1a0080 con 0x7f420c0ffb40 2024-05-09T09:18:11.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.106+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f420c108cc0 con 0x7f420c0ffb40 2024-05-09T09:18:11.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.107+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f41fc003fb0 con 0x7f420c0ffb40 2024-05-09T09:18:11.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.107+0000 7f41fb7fe640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 0x7f41f40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.107+0000 7f4209d9b640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 0x7f41f40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.107+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f41fc0cee30 con 0x7f420c0ffb40 2024-05-09T09:18:11.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.108+0000 7f4209d9b640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 0x7f41f40a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4200006010 tx=0x7f4200032d90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.112+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f41fc009de0 con 0x7f420c0ffb40 2024-05-09T09:18:11.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:11 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1232841508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.291+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f420c062530 con 0x7f420c0ffb40 2024-05-09T09:18:11.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.291+0000 7f41fb7fe640 1 -- 172.21.15.69:0/722772872 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f41fc09ba50 con 0x7f420c0ffb40 2024-05-09T09:18:11.291 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:11.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.293+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 msgr2=0x7f41f40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.293+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 0x7f41f40a50d0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f4200006010 tx=0x7f4200032d90 comp rx=0 tx=0).stop 2024-05-09T09:18:11.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 msgr2=0x7f420c197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7f41fc00d770 tx=0x7f41fc00dc30 comp rx=0 tx=0).stop 2024-05-09T09:18:11.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 shutdown_connections 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41f40a2c10 0x7f41f40a50d0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f420c1034d0 0x7f420c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f420c102b10 0x7f420c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 --2- 172.21.15.69:0/722772872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f420c0ffb40 0x7f420c197c30 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 >> 172.21.15.69:0/722772872 conn(0x7f420c0fb470 msgr2=0x7f420c105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 shutdown_connections 2024-05-09T09:18:11.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.294+0000 7f4210e3f640 1 -- 172.21.15.69:0/722772872 wait complete. 2024-05-09T09:18:11.305 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:11.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 -- 172.21.15.69:0/2862932551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 msgr2=0x7f7f0c109e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 --2- 172.21.15.69:0/2862932551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c109e20 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f7f04009cd0 tx=0x7f7f0402a8e0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 -- 172.21.15.69:0/2862932551 shutdown_connections 2024-05-09T09:18:11.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 --2- 172.21.15.69:0/2862932551 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c109e20 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 --2- 172.21.15.69:0/2862932551 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c103050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 --2- 172.21.15.69:0/2862932551 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c0ffb60 0x7f7f0c0fff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 -- 172.21.15.69:0/2862932551 >> 172.21.15.69:0/2862932551 conn(0x7f7f0c0fb470 msgr2=0x7f7f0c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 -- 172.21.15.69:0/2862932551 shutdown_connections 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.410+0000 7f7f11876640 1 -- 172.21.15.69:0/2862932551 wait complete. 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 Processor -- start 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 -- start start 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c0ffb60 0x7f7f0c197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c10d500 con 0x7f7f0c103590 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c10d380 con 0x7f7f0c0ffb60 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f11876640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f0c10d680 con 0x7f7f0c102bd0 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f0b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f0b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57342/0 (socket says 172.21.15.69:57342) 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f0b7fe640 1 -- 172.21.15.69:0/219334768 learned_addr learned my addr 172.21.15.69:0/219334768 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.411+0000 7f7f0affd640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c0ffb60 0x7f7f0c197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0a7fc640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 -- 172.21.15.69:0/219334768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 msgr2=0x7f7f0c1980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c1980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 -- 172.21.15.69:0/219334768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c0ffb60 msgr2=0x7f7f0c197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c0ffb60 0x7f7f0c197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 -- 172.21.15.69:0/219334768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f0c19f790 con 0x7f7f0c103590 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0a7fc640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f0b7fe640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f040317f0 tx=0x7f7f04003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f04004240 con 0x7f7f0c103590 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f040305a0 con 0x7f7f0c103590 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f0c19fa20 con 0x7f7f0c103590 2024-05-09T09:18:11.414 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f0403a660 con 0x7f7f0c103590 2024-05-09T09:18:11.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.412+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f0c19ff70 con 0x7f7f0c103590 2024-05-09T09:18:11.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.414+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7f0c108c30 con 0x7f7f0c103590 2024-05-09T09:18:11.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.414+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f04030750 con 0x7f7f0c103590 2024-05-09T09:18:11.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.415+0000 7f7f10874640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 0x7f7ef40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.415+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f7f0402e030 con 0x7f7f0c103590 2024-05-09T09:18:11.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.415+0000 7f7f0affd640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 0x7f7ef40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.416+0000 7f7f0affd640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 0x7f7ef40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7efc005fd0 tx=0x7f7efc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.420+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7f040bd5e0 con 0x7f7f0c103590 2024-05-09T09:18:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:11 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1232841508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:11.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.611+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f7f0c062430 con 0x7f7f0c103590 2024-05-09T09:18:11.611 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.611+0000 7f7f10874640 1 -- 172.21.15.69:0/219334768 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7f7f040bcee0 con 0x7f7f0c103590 2024-05-09T09:18:11.611 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.613+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 msgr2=0x7f7ef40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.613+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 0x7f7ef40a5050 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7efc005fd0 tx=0x7f7efc00b390 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.613+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 msgr2=0x7f7f0c19f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.613+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7f040317f0 tx=0x7f7f04003ba0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 shutdown_connections 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7ef40a2b90 0x7f7ef40a5050 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f0c103590 0x7f7f0c19f120 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f0c102bd0 0x7f7f0c1980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 --2- 172.21.15.69:0/219334768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f0c0ffb60 0x7f7f0c197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 >> 172.21.15.69:0/219334768 conn(0x7f7f0c0fb470 msgr2=0x7f7f0c105e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 shutdown_connections 2024-05-09T09:18:11.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.614+0000 7f7f11876640 1 -- 172.21.15.69:0/219334768 wait complete. 2024-05-09T09:18:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:11 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1232841508' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:11.625 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 -- 172.21.15.69:0/4001032894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48100930 msgr2=0x7fba48100db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 --2- 172.21.15.69:0/4001032894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48100930 0x7fba48100db0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009900 tx=0x7fba3c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 -- 172.21.15.69:0/4001032894 shutdown_connections 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 --2- 172.21.15.69:0/4001032894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba481012f0 0x7fba48108b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 --2- 172.21.15.69:0/4001032894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba48100930 0x7fba48100db0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 --2- 172.21.15.69:0/4001032894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 0x7fba48102020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 -- 172.21.15.69:0/4001032894 >> 172.21.15.69:0/4001032894 conn(0x7fba480fb360 msgr2=0x7fba480fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 -- 172.21.15.69:0/4001032894 shutdown_connections 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.730+0000 7fba4d39e640 1 -- 172.21.15.69:0/4001032894 wait complete. 2024-05-09T09:18:11.731 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 Processor -- start 2024-05-09T09:18:11.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 -- start start 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba48100930 0x7fba48100390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 0x7fba4819ede0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810c300 con 0x7fba481012f0 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810c180 con 0x7fba48101c00 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba4d39e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fba4810c480 con 0x7fba48100930 2024-05-09T09:18:11.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba477fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba477fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57360/0 (socket says 172.21.15.69:57360) 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.731+0000 7fba477fe640 1 -- 172.21.15.69:0/1920088916 learned_addr learned my addr 172.21.15.69:0/1920088916 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba47fff640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba48100930 0x7fba48100390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba4cb9d640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 0x7fba4819ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 -- 172.21.15.69:0/1920088916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba48100930 msgr2=0x7fba48100390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba48100930 0x7fba48100390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 -- 172.21.15.69:0/1920088916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 msgr2=0x7fba4819ede0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 0x7fba4819ede0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 -- 172.21.15.69:0/1920088916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fba4819f6f0 con 0x7fba481012f0 2024-05-09T09:18:11.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba477fe640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009230 tx=0x7fba3c0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c03f030 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fba4819f920 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fba3c003ce0 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fba3c035070 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.732+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fba4819fe10 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.733+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fba48107a70 con 0x7fba481012f0 2024-05-09T09:18:11.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.734+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fba3c042030 con 0x7fba481012f0 2024-05-09T09:18:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.735+0000 7fba457fa640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 0x7fba300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.735+0000 7fba47fff640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 0x7fba300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.735+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fba3c0ef940 con 0x7fba481012f0 2024-05-09T09:18:11.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.735+0000 7fba47fff640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 0x7fba300a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fba48069b50 tx=0x7fba3800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:11.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.740+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fba3c0bc560 con 0x7fba481012f0 2024-05-09T09:18:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.916+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7fba48062430 con 0x7fba481012f0 2024-05-09T09:18:11.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.916+0000 7fba457fa640 1 -- 172.21.15.69:0/1920088916 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7fba3c0bbe60 con 0x7fba481012f0 2024-05-09T09:18:11.917 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.918+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 msgr2=0x7fba300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.918+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 0x7fba300a5050 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fba48069b50 tx=0x7fba3800a2d0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.918+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 msgr2=0x7fba4819c7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.918+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fba3c009230 tx=0x7fba3c0042f0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 shutdown_connections 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fba300a2b90 0x7fba300a5050 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fba48101c00 0x7fba4819ede0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fba481012f0 0x7fba4819c7c0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 --2- 172.21.15.69:0/1920088916 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fba48100930 0x7fba48100390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:11.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 >> 172.21.15.69:0/1920088916 conn(0x7fba480fb360 msgr2=0x7fba48104c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:11.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 shutdown_connections 2024-05-09T09:18:11.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:11.919+0000 7fba4d39e640 1 -- 172.21.15.69:0/1920088916 wait complete. 2024-05-09T09:18:11.930 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 -- 172.21.15.69:0/2366543305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc102330 msgr2=0x7f47fc108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 --2- 172.21.15.69:0/2366543305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc102330 0x7f47fc108bc0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f47f4009ca0 tx=0x7f47f402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 -- 172.21.15.69:0/2366543305 shutdown_connections 2024-05-09T09:18:12.034 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 --2- 172.21.15.69:0/2366543305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc102330 0x7f47fc108bc0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 --2- 172.21.15.69:0/2366543305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc101970 0x7f47fc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 --2- 172.21.15.69:0/2366543305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc100780 0x7f47fc100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 -- 172.21.15.69:0/2366543305 >> 172.21.15.69:0/2366543305 conn(0x7f47fc0fbcf0 msgr2=0x7f47fc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 -- 172.21.15.69:0/2366543305 shutdown_connections 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 -- 172.21.15.69:0/2366543305 wait complete. 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.034+0000 7f480191d640 1 Processor -- start 2024-05-09T09:18:12.035 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 -- start start 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc100780 0x7f47fc06a940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc102330 0x7f47fc19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc10c460 con 0x7f47fc101970 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc10c2e0 con 0x7f47fc102330 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f480191d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f47fc10c5e0 con 0x7f47fc100780 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57378/0 (socket says 172.21.15.69:57378) 2024-05-09T09:18:12.037 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 -- 172.21.15.69:0/1947704462 learned_addr learned my addr 172.21.15.69:0/1947704462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fb7fe640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc102330 0x7f47fc19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47faffd640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc100780 0x7f47fc06a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 -- 172.21.15.69:0/1947704462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc100780 msgr2=0x7f47fc06a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc100780 0x7f47fc06a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 -- 172.21.15.69:0/1947704462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc102330 msgr2=0x7f47fc19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc102330 0x7f47fc19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 -- 172.21.15.69:0/1947704462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f47fc19f850 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.035+0000 7f47fa7fc640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f47e400b4a0 tx=0x7f47e400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.036+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4014070 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.036+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47e4010af0 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.036+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47e4017d10 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.036+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f47fc19fae0 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.036+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f47fc1a0060 con 0x7f47fc101970 2024-05-09T09:18:12.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.037+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f47fc107aa0 con 0x7f47fc101970 2024-05-09T09:18:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.038+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f47e4017070 con 0x7f47fc101970 2024-05-09T09:18:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.038+0000 7f480091b640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 0x7f47e80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.038+0000 7f47faffd640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 0x7f47e80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.039+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f47e40ceaf0 con 0x7f47fc101970 2024-05-09T09:18:12.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.039+0000 7f47faffd640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 0x7f47e80a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005db0 tx=0x7f47ec005d40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.042+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f47e409b710 con 0x7f47fc101970 2024-05-09T09:18:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.217+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f47fc02d350 con 0x7f47fc101970 2024-05-09T09:18:12.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.217+0000 7f480091b640 1 -- 172.21.15.69:0/1947704462 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f47e409b010 con 0x7f47fc101970 2024-05-09T09:18:12.217 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:12.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 msgr2=0x7f47e80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 0x7f47e80a5050 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f47ec005db0 tx=0x7f47ec005d40 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 msgr2=0x7f47fc19c920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f47e400b4a0 tx=0x7f47e400b960 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 shutdown_connections 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47e80a2b90 0x7f47e80a5050 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f47fc102330 0x7f47fc19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f47fc101970 0x7f47fc19c920 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 --2- 172.21.15.69:0/1947704462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f47fc100780 0x7f47fc06a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 >> 172.21.15.69:0/1947704462 conn(0x7f47fc0fbcf0 msgr2=0x7f47fc104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 shutdown_connections 2024-05-09T09:18:12.220 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.219+0000 7f480191d640 1 -- 172.21.15.69:0/1947704462 wait complete. 2024-05-09T09:18:12.231 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:12.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 -- 172.21.15.69:0/3335471673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7180fff10 msgr2=0x7fe718100330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 --2- 172.21.15.69:0/3335471673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7180fff10 0x7fe718100330 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7fe714009ca0 tx=0x7fe71402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 -- 172.21.15.69:0/3335471673 shutdown_connections 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 --2- 172.21.15.69:0/3335471673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101a20 0x7fe718108340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 --2- 172.21.15.69:0/3335471673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe718101060 0x7fe7181014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 --2- 172.21.15.69:0/3335471673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7180fff10 0x7fe718100330 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 -- 172.21.15.69:0/3335471673 >> 172.21.15.69:0/3335471673 conn(0x7fe7180fb4a0 msgr2=0x7fe7180fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 -- 172.21.15.69:0/3335471673 shutdown_connections 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.332+0000 7fe71f996640 1 -- 172.21.15.69:0/3335471673 wait complete. 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 Processor -- start 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 -- start start 2024-05-09T09:18:12.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7180fff10 0x7fe7181098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101060 0x7fe71819c950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe71810c490 con 0x7fe718101a20 2024-05-09T09:18:12.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe71810c310 con 0x7fe7180fff10 2024-05-09T09:18:12.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71f996640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe71810c610 con 0x7fe718101060 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71df33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71df33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57408/0 (socket says 172.21.15.69:57408) 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71df33640 1 -- 172.21.15.69:0/2324237973 learned_addr learned my addr 172.21.15.69:0/2324237973 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.333+0000 7fe71d732640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7180fff10 0x7fe7181098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71cf31640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101060 0x7fe71819c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 -- 172.21.15.69:0/2324237973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101060 msgr2=0x7fe71819c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101060 0x7fe71819c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 -- 172.21.15.69:0/2324237973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7180fff10 msgr2=0x7fe7181098a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7180fff10 0x7fe7181098a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 -- 172.21.15.69:0/2324237973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe71819f880 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71df33640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe71000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe710011830 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe710016070 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe71819fb10 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe71000f2e0 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.334+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7181a0060 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.335+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe7181072a0 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.336+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe7100119d0 con 0x7fe718101a20 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.336+0000 7fe70e7fc640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 0x7fe7040a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.337+0000 7fe71d732640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 0x7fe7040a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.337+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fe7100cea10 con 0x7fe718101a20 2024-05-09T09:18:12.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.337+0000 7fe71d732640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 0x7fe7040a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe714000f80 tx=0x7fe714009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.341+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe71009b630 con 0x7fe718101a20 2024-05-09T09:18:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:12 smithi069 ceph-mon[30123]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/722772872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/219334768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1920088916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1947704462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.552+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fe718062460 con 0x7fe718101a20 2024-05-09T09:18:12.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.553+0000 7fe70e7fc640 1 -- 172.21.15.69:0/2324237973 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7fe71009af30 con 0x7fe718101a20 2024-05-09T09:18:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:12.555 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 msgr2=0x7fe7040a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 0x7fe7040a50d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fe714000f80 tx=0x7fe714009360 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 msgr2=0x7fe71819ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7fe710009fd0 tx=0x7fe71000b8e0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 shutdown_connections 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7040a2c10 0x7fe7040a50d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe718101a20 0x7fe71819ef70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe718101060 0x7fe71819c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 --2- 172.21.15.69:0/2324237973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7180fff10 0x7fe7181098a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 >> 172.21.15.69:0/2324237973 conn(0x7fe7180fb4a0 msgr2=0x7fe718104470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 shutdown_connections 2024-05-09T09:18:12.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.555+0000 7fe71f996640 1 -- 172.21.15.69:0/2324237973 wait complete. 2024-05-09T09:18:12.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:12 smithi163 ceph-mon[34354]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:12.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/722772872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/219334768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1920088916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1947704462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.568 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:12 smithi103 ceph-mon[34421]: pgmap v342: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/722772872' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/219334768' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1920088916' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1947704462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 -- 172.21.15.69:0/859591847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441007f0 msgr2=0x7fbf44100c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 --2- 172.21.15.69:0/859591847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441007f0 0x7fbf44100c10 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7fbf34009900 tx=0x7fbf3402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 -- 172.21.15.69:0/859591847 shutdown_connections 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 --2- 172.21.15.69:0/859591847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf441023a0 0x7fbf44108c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 --2- 172.21.15.69:0/859591847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441019e0 0x7fbf44101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.672 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 --2- 172.21.15.69:0/859591847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441007f0 0x7fbf44100c10 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 -- 172.21.15.69:0/859591847 >> 172.21.15.69:0/859591847 conn(0x7fbf440fbd80 msgr2=0x7fbf440fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 -- 172.21.15.69:0/859591847 shutdown_connections 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 -- 172.21.15.69:0/859591847 wait complete. 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.670+0000 7fbf4943f640 1 Processor -- start 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- start start 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441007f0 0x7fbf4406aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf441023a0 0x7fbf4419ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410c450 con 0x7fbf441019e0 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410c2d0 con 0x7fbf441007f0 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbf4410c5d0 con 0x7fbf441023a0 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57416/0 (socket says 172.21.15.69:57416) 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 -- 172.21.15.69:0/4077735118 learned_addr learned my addr 172.21.15.69:0/4077735118 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:12.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf42ffd640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441007f0 0x7fbf4406aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 -- 172.21.15.69:0/4077735118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf441023a0 msgr2=0x7fbf4419ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf441023a0 0x7fbf4419ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 -- 172.21.15.69:0/4077735118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441007f0 msgr2=0x7fbf4406aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441007f0 0x7fbf4406aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 -- 172.21.15.69:0/4077735118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf4419f870 con 0x7fbf441019e0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf3a7fc640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00d950 tx=0x7fbf2c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c017070 con 0x7fbf441019e0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf4419fb00 con 0x7fbf441019e0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf2c009c30 con 0x7fbf441019e0 2024-05-09T09:18:12.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf2c00f6b0 con 0x7fbf441019e0 2024-05-09T09:18:12.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.671+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf441a0120 con 0x7fbf441019e0 2024-05-09T09:18:12.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.672+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbf44107b10 con 0x7fbf441019e0 2024-05-09T09:18:12.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.674+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbf2c0109d0 con 0x7fbf441019e0 2024-05-09T09:18:12.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.674+0000 7fbf40ff9640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 0x7fbf300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.674+0000 7fbf42ffd640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 0x7fbf300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.674+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fbf2c0cf050 con 0x7fbf441019e0 2024-05-09T09:18:12.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.675+0000 7fbf42ffd640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 0x7fbf300a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf34006010 tx=0x7fbf34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.677+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbf2c009de0 con 0x7fbf441019e0 2024-05-09T09:18:12.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.854+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fbf4402d3c0 con 0x7fbf441019e0 2024-05-09T09:18:12.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.855+0000 7fbf40ff9640 1 -- 172.21.15.69:0/4077735118 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24614 (secure 0 0 0) 0x7fbf2c0bb070 con 0x7fbf441019e0 2024-05-09T09:18:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:12.856 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:17:30.193538+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.857+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 msgr2=0x7fbf300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.857+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 0x7fbf300a5050 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fbf34006010 tx=0x7fbf34000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 msgr2=0x7fbf4419c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7fbf2c00d950 tx=0x7fbf2c00de10 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 shutdown_connections 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf300a2b90 0x7fbf300a5050 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf441023a0 0x7fbf4419ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf441019e0 0x7fbf4419c940 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 --2- 172.21.15.69:0/4077735118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf441007f0 0x7fbf4406aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 >> 172.21.15.69:0/4077735118 conn(0x7fbf440fbd80 msgr2=0x7fbf44104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 shutdown_connections 2024-05-09T09:18:12.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.858+0000 7fbf4943f640 1 -- 172.21.15.69:0/4077735118 wait complete. 2024-05-09T09:18:12.869 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:12.870 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:12.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.976+0000 7f9128cee640 1 -- 172.21.15.69:0/2402421715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100aa0 msgr2=0x7f9124100f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.976+0000 7f9128cee640 1 --2- 172.21.15.69:0/2402421715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100aa0 0x7f9124100f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f9118009ca0 tx=0x7f911802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 -- 172.21.15.69:0/2402421715 shutdown_connections 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 --2- 172.21.15.69:0/2402421715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9124101440 0x7f91241018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 --2- 172.21.15.69:0/2402421715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100aa0 0x7f9124100f00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 --2- 172.21.15.69:0/2402421715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100140 0x7f9124100560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 -- 172.21.15.69:0/2402421715 >> 172.21.15.69:0/2402421715 conn(0x7f91240fbce0 msgr2=0x7f91240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 -- 172.21.15.69:0/2402421715 shutdown_connections 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 -- 172.21.15.69:0/2402421715 wait complete. 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 Processor -- start 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.977+0000 7f9128cee640 1 -- start start 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100aa0 0x7f912419a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9124101440 0x7f91241a13c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f912410f7a0 con 0x7f9124100140 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f912410f620 con 0x7f9124101440 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f912410f920 con 0x7f9124100aa0 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57434/0 (socket says 172.21.15.69:57434) 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 -- 172.21.15.69:0/3649134667 learned_addr learned my addr 172.21.15.69:0/3649134667 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9121d9b640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100aa0 0x7f912419a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 -- 172.21.15.69:0/3649134667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100aa0 msgr2=0x7f912419a350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100aa0 0x7f912419a350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 -- 172.21.15.69:0/3649134667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9124101440 msgr2=0x7f91241a13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9124101440 0x7f91241a13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 -- 172.21.15.69:0/3649134667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f91241a1a30 con 0x7f9124100140 2024-05-09T09:18:12.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f912259c640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f911400c970 tx=0x7f911400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9114007bb0 con 0x7f9124100140 2024-05-09T09:18:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9114007d50 con 0x7f9124100140 2024-05-09T09:18:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f91241a1d20 con 0x7f9124100140 2024-05-09T09:18:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f911400f450 con 0x7f9124100140 2024-05-09T09:18:12.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.978+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f91241a2240 con 0x7f9124100140 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.979+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f912410b030 con 0x7f9124100140 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.980+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9114005020 con 0x7f9124100140 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.981+0000 7f91137fe640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 0x7f910c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.981+0000 7f9121d9b640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 0x7f910c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.981+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f91140cf190 con 0x7f9124100140 2024-05-09T09:18:12.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.981+0000 7f9121d9b640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 0x7f910c0a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f912419b320 tx=0x7f9118031bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:12.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:12.985+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9114014030 con 0x7f9124100140 2024-05-09T09:18:13.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.161+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9124062530 con 0x7f9124100140 2024-05-09T09:18:13.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.162+0000 7f91137fe640 1 -- 172.21.15.69:0/3649134667 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) ==== 93+0+10 (secure 0 0 0) 0x7f911409bdb0 con 0x7f9124100140 2024-05-09T09:18:13.162 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 msgr2=0x7f910c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 0x7f910c0a50d0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f912419b320 tx=0x7f9118031bd0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 msgr2=0x7f9124199e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f911400c970 tx=0x7f911400ce30 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 shutdown_connections 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f910c0a2c10 0x7f910c0a50d0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9124101440 0x7f91241a13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9124100aa0 0x7f912419a350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 --2- 172.21.15.69:0/3649134667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9124100140 0x7f9124199e10 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.164 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 >> 172.21.15.69:0/3649134667 conn(0x7f91240fbce0 msgr2=0x7f91240fd410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 shutdown_connections 2024-05-09T09:18:13.165 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.164+0000 7f9128cee640 1 -- 172.21.15.69:0/3649134667 wait complete. 2024-05-09T09:18:13.175 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 -- 172.21.15.69:0/2554318221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1801013c0 msgr2=0x7ff180108c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2554318221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1801013c0 0x7ff180108c60 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7ff178009930 tx=0x7ff17802a420 comp rx=0 tx=0).stop 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 -- 172.21.15.69:0/2554318221 shutdown_connections 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2554318221 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff1801013c0 0x7ff180108c60 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2554318221 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff180100a00 0x7ff180100e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2554318221 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff180101d60 0x7ff180102180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 -- 172.21.15.69:0/2554318221 >> 172.21.15.69:0/2554318221 conn(0x7ff1800fb470 msgr2=0x7ff1800fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 -- 172.21.15.69:0/2554318221 shutdown_connections 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.284+0000 7ff187bb7640 1 -- 172.21.15.69:0/2554318221 wait complete. 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 Processor -- start 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 -- start start 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 0x7ff18019c860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff180101d60 0x7ff18019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010c3f0 con 0x7ff180100a00 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010c270 con 0x7ff1801013c0 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff187bb7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff18010c570 con 0x7ff180101d60 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57456/0 (socket says 172.21.15.69:57456) 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 -- 172.21.15.69:0/2363376896 learned_addr learned my addr 172.21.15.69:0/2363376896 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:13.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185152640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 0x7ff18019c860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 -- 172.21.15.69:0/2363376896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff180101d60 msgr2=0x7ff18019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff180101d60 0x7ff18019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 -- 172.21.15.69:0/2363376896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 msgr2=0x7ff18019c860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 0x7ff18019c860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 -- 172.21.15.69:0/2363376896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff18019f760 con 0x7ff180100a00 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185152640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 0x7ff18019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff185953640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff17000ccb0 tx=0x7ff17000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.285+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff170003b70 con 0x7ff180100a00 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.286+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff18019fa50 con 0x7ff180100a00 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.286+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff17000ae50 con 0x7ff180100a00 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.286+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff17000f810 con 0x7ff180100a00 2024-05-09T09:18:13.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.286+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff18019ffa0 con 0x7ff180100a00 2024-05-09T09:18:13.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.287+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ff180107b40 con 0x7ff180100a00 2024-05-09T09:18:13.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.288+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff17000f9b0 con 0x7ff180100a00 2024-05-09T09:18:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.288+0000 7ff17effd640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 0x7ff1680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.288+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7ff1700cf570 con 0x7ff180100a00 2024-05-09T09:18:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.288+0000 7ff185152640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 0x7ff1680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.289+0000 7ff185152640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 0x7ff1680a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff174005d00 tx=0x7ff174005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.293+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ff170014070 con 0x7ff180100a00 2024-05-09T09:18:13.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.467+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7ff1800624d0 con 0x7ff180100a00 2024-05-09T09:18:13.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.467+0000 7ff17effd640 1 -- 172.21.15.69:0/2363376896 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) ==== 104+0+11 (secure 0 0 0) 0x7ff17009c190 con 0x7ff180100a00 2024-05-09T09:18:13.467 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:13.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 msgr2=0x7ff1680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 0x7ff1680a5050 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7ff174005d00 tx=0x7ff174005ed0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 msgr2=0x7ff180100480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7ff17000ccb0 tx=0x7ff17000a620 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 shutdown_connections 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff1680a2b90 0x7ff1680a5050 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff180101d60 0x7ff18019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff1801013c0 0x7ff18019c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 --2- 172.21.15.69:0/2363376896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff180100a00 0x7ff180100480 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.469+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 >> 172.21.15.69:0/2363376896 conn(0x7ff1800fb470 msgr2=0x7ff180104d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.470+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 shutdown_connections 2024-05-09T09:18:13.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.470+0000 7ff187bb7640 1 -- 172.21.15.69:0/2363376896 wait complete. 2024-05-09T09:18:13.479 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:13 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2324237973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:13 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4077735118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:13 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3649134667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:13.481 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:13 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2324237973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:13 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4077735118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:13 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3649134667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.585+0000 7f6d12038640 1 -- 172.21.15.69:0/3675374024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 msgr2=0x7f6d0c107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.585+0000 7f6d12038640 1 --2- 172.21.15.69:0/3675374024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c107070 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4009900 tx=0x7f6cf402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- 172.21.15.69:0/3675374024 shutdown_connections 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- 172.21.15.69:0/3675374024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c1099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- 172.21.15.69:0/3675374024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c107070 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- 172.21.15.69:0/3675374024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 0x7f6d0c104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- 172.21.15.69:0/3675374024 >> 172.21.15.69:0/3675374024 conn(0x7f6d0c0fbce0 msgr2=0x7f6d0c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- 172.21.15.69:0/3675374024 shutdown_connections 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- 172.21.15.69:0/3675374024 wait complete. 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 Processor -- start 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- start start 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 0x7f6d0c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c19f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10d4e0 con 0x7f6d0c104c80 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10d360 con 0x7f6d0c1075b0 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.586+0000 7f6d12038640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6d0c10d660 con 0x7f6d0c102350 2024-05-09T09:18:13.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57478/0 (socket says 172.21.15.69:57478) 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 -- 172.21.15.69:0/2749030844 learned_addr learned my addr 172.21.15.69:0/2749030844 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d0b7fe640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 0x7f6d0c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 -- 172.21.15.69:0/2749030844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 msgr2=0x7f6d0c197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 0x7f6d0c197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d0bfff640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c19f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 -- 172.21.15.69:0/2749030844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 msgr2=0x7f6d0c19f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c19f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 -- 172.21.15.69:0/2749030844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6d0c19f800 con 0x7f6d0c104c80 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d0bfff640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c19f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d02ffd640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4031610 tx=0x7f6cf40043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf403f030 con 0x7f6d0c104c80 2024-05-09T09:18:13.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6d0c19fa90 con 0x7f6d0c104c80 2024-05-09T09:18:13.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6cf4003ce0 con 0x7f6d0c104c80 2024-05-09T09:18:13.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6cf4035070 con 0x7f6d0c104c80 2024-05-09T09:18:13.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.587+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6d0c19ffd0 con 0x7f6d0c104c80 2024-05-09T09:18:13.591 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.588+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6d0c0fd7d0 con 0x7f6d0c104c80 2024-05-09T09:18:13.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.589+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6cf4042030 con 0x7f6d0c104c80 2024-05-09T09:18:13.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.590+0000 7f6d097fa640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 0x7f6cf80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.590+0000 7f6d0b7fe640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 0x7f6cf80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.590+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f6cf40ef990 con 0x7f6d0c104c80 2024-05-09T09:18:13.592 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.591+0000 7f6d0b7fe640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 0x7f6cf80a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005fd0 tx=0x7f6cfc005e20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.594 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.594+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6cf40bc5b0 con 0x7f6d0c104c80 2024-05-09T09:18:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:13 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2324237973' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:13 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4077735118' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:13 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3649134667' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:13.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.771+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f6d0c062430 con 0x7f6d0c104c80 2024-05-09T09:18:13.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.771+0000 7f6d097fa640 1 -- 172.21.15.69:0/2749030844 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) ==== 100+0+11 (secure 0 0 0) 0x7f6cf40bbeb0 con 0x7f6d0c104c80 2024-05-09T09:18:13.771 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:13.773 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.773+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 msgr2=0x7f6cf80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.773+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 0x7f6cf80a5050 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f6cfc005fd0 tx=0x7f6cfc005e20 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.773+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 msgr2=0x7f6d0c198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.773+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f6cf4031610 tx=0x7f6cf40043b0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 shutdown_connections 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6cf80a2b90 0x7f6cf80a5050 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6d0c1075b0 0x7f6d0c19f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6d0c104c80 0x7f6d0c198120 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 --2- 172.21.15.69:0/2749030844 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6d0c102350 0x7f6d0c197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 >> 172.21.15.69:0/2749030844 conn(0x7f6d0c0fbce0 msgr2=0x7f6d0c102e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 shutdown_connections 2024-05-09T09:18:13.774 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.774+0000 7f6d12038640 1 -- 172.21.15.69:0/2749030844 wait complete. 2024-05-09T09:18:13.785 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:13.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 -- 172.21.15.69:0/3165939197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc102320 msgr2=0x7f08dc108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/3165939197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc102320 0x7f08dc108bb0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f08d4009ca0 tx=0x7f08d402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 -- 172.21.15.69:0/3165939197 shutdown_connections 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/3165939197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc102320 0x7f08dc108bb0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/3165939197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08dc101960 0x7f08dc101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/3165939197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc100770 0x7f08dc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 -- 172.21.15.69:0/3165939197 >> 172.21.15.69:0/3165939197 conn(0x7f08dc0fbce0 msgr2=0x7f08dc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 -- 172.21.15.69:0/3165939197 shutdown_connections 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.889+0000 7f08e0d5a640 1 -- 172.21.15.69:0/3165939197 wait complete. 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 Processor -- start 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 -- start start 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08dc100770 0x7f08dc199f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 0x7f08dc1a14f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d5f0 con 0x7f08dc101960 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d470 con 0x7f08dc102320 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08e0d5a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f08dc10d770 con 0x7f08dc100770 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57496/0 (socket says 172.21.15.69:57496) 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 -- 172.21.15.69:0/903221449 learned_addr learned my addr 172.21.15.69:0/903221449 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08dad9d640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 0x7f08dc1a14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 -- 172.21.15.69:0/903221449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08dc100770 msgr2=0x7f08dc199f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08dc100770 0x7f08dc199f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 -- 172.21.15.69:0/903221449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 msgr2=0x7f08dc1a14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 0x7f08dc1a14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 -- 172.21.15.69:0/903221449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f08dc1a1b60 con 0x7f08dc101960 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08dad9d640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 0x7f08dc1a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:13.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08d9d9b640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f08d000b4a0 tx=0x7f08d000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.890+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0014070 con 0x7f08dc101960 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.891+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f08d0010af0 con 0x7f08dc101960 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.891+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f08dc1a1e50 con 0x7f08dc101960 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.891+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f08d0017d10 con 0x7f08dc101960 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.891+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f08dc1a2340 con 0x7f08dc101960 2024-05-09T09:18:13.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.892+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f08dc1a25d0 con 0x7f08dc101960 2024-05-09T09:18:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.893+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f08d0017070 con 0x7f08dc101960 2024-05-09T09:18:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.894+0000 7f08cb7fe640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 0x7f08c40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.894+0000 7f08da59c640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 0x7f08c40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.894+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f08d00ce800 con 0x7f08dc101960 2024-05-09T09:18:13.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.894+0000 7f08da59c640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 0x7f08c40a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08dc1017c0 tx=0x7f08cc006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:13.898 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:13.898+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f08d009b420 con 0x7f08dc101960 2024-05-09T09:18:14.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.073+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f08dc1a27a0 con 0x7f08dc101960 2024-05-09T09:18:14.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.074+0000 7f08cb7fe640 1 -- 172.21.15.69:0/903221449 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) ==== 103+0+11 (secure 0 0 0) 0x7f08d009ad20 con 0x7f08dc101960 2024-05-09T09:18:14.074 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 msgr2=0x7f08c40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 0x7f08c40a5050 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f08dc1017c0 tx=0x7f08cc006c90 comp rx=0 tx=0).stop 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 msgr2=0x7f08dc19a480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f08d000b4a0 tx=0x7f08d000b960 comp rx=0 tx=0).stop 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 shutdown_connections 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08c40a2b90 0x7f08c40a5050 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f08dc102320 0x7f08dc1a14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f08dc101960 0x7f08dc19a480 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.076 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 --2- 172.21.15.69:0/903221449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f08dc100770 0x7f08dc199f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 >> 172.21.15.69:0/903221449 conn(0x7f08dc0fbce0 msgr2=0x7f08dc109570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 shutdown_connections 2024-05-09T09:18:14.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.076+0000 7f08e0d5a640 1 -- 172.21.15.69:0/903221449 wait complete. 2024-05-09T09:18:14.087 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:14.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.188+0000 7f8d21184640 1 -- 172.21.15.69:0/2485794725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 msgr2=0x7f8d1c100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.188+0000 7f8d21184640 1 --2- 172.21.15.69:0/2485794725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c100ba0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c009900 tx=0x7f8d0c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:14.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- 172.21.15.69:0/2485794725 shutdown_connections 2024-05-09T09:18:14.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- 172.21.15.69:0/2485794725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c102330 0x7f8d1c108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- 172.21.15.69:0/2485794725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c101970 0x7f8d1c101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- 172.21.15.69:0/2485794725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c100ba0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- 172.21.15.69:0/2485794725 >> 172.21.15.69:0/2485794725 conn(0x7f8d1c0fbd30 msgr2=0x7f8d1c0fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- 172.21.15.69:0/2485794725 shutdown_connections 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- 172.21.15.69:0/2485794725 wait complete. 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 Processor -- start 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- start start 2024-05-09T09:18:14.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c101970 0x7f8d1c198270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 0x7f8d1c19f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10d5e0 con 0x7f8d1c100780 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.189+0000 7f8d21184640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10d460 con 0x7f8d1c102330 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d21184640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d1c10d760 con 0x7f8d1c101970 2024-05-09T09:18:14.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57512/0 (socket says 172.21.15.69:57512) 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 -- 172.21.15.69:0/1747388532 learned_addr learned my addr 172.21.15.69:0/1747388532 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d13fff640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c101970 0x7f8d1c198270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1b59e640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 0x7f8d1c19f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 -- 172.21.15.69:0/1747388532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c101970 msgr2=0x7f8d1c198270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c101970 0x7f8d1c198270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 -- 172.21.15.69:0/1747388532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 msgr2=0x7f8d1c19f2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 0x7f8d1c19f2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 -- 172.21.15.69:0/1747388532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d1c19f940 con 0x7f8d1c100780 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1b59e640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 0x7f8d1c19f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d1ad9d640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c031610 tx=0x7f8d0c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c03f030 con 0x7f8d1c100780 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d0c003ce0 con 0x7f8d1c100780 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d1c19fbd0 con 0x7f8d1c100780 2024-05-09T09:18:14.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d0c035070 con 0x7f8d1c100780 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.190+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d1c1a00c0 con 0x7f8d1c100780 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.191+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d1c107b10 con 0x7f8d1c100780 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.192+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8d0c042030 con 0x7f8d1c100780 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.193+0000 7f8d18d99640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 0x7f8d080a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.193+0000 7f8d13fff640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 0x7f8d080a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.193+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f8d0c0efa90 con 0x7f8d1c100780 2024-05-09T09:18:14.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.194+0000 7f8d13fff640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 0x7f8d080a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009990 tx=0x7f8d04009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.197+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d0c0bc6b0 con 0x7f8d1c100780 2024-05-09T09:18:14.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.412+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8d1c02d3e0 con 0x7f8d1c100780 2024-05-09T09:18:14.412 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.413+0000 7f8d18d99640 1 -- 172.21.15.69:0/1747388532 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7f8d0c0bbfb0 con 0x7f8d1c100780 2024-05-09T09:18:14.413 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:14.413 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:18:14.413 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 msgr2=0x7f8d080a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 0x7f8d080a5050 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f8d04009990 tx=0x7f8d04009110 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 msgr2=0x7f8d1c197d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f8d0c031610 tx=0x7f8d0c0043b0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 shutdown_connections 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d080a2b90 0x7f8d080a5050 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d1c102330 0x7f8d1c19f2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d1c101970 0x7f8d1c198270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 --2- 172.21.15.69:0/1747388532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d1c100780 0x7f8d1c197d30 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 >> 172.21.15.69:0/1747388532 conn(0x7f8d1c0fbd30 msgr2=0x7f8d1c104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 shutdown_connections 2024-05-09T09:18:14.415 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.415+0000 7f8d21184640 1 -- 172.21.15.69:0/1747388532 wait complete. 2024-05-09T09:18:14.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:14 smithi069 ceph-mon[30123]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:14.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:14 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2363376896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:14 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2749030844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.425 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:14 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/903221449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.426 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:14.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3700375090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 msgr2=0x7f84f81099b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3700375090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f81099b0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f84f0009ca0 tx=0x7f84f002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3700375090 shutdown_connections 2024-05-09T09:18:14.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3700375090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f81099b0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3700375090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3700375090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 0x7f84f81046f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3700375090 >> 172.21.15.69:0/3700375090 conn(0x7f84f80fbd10 msgr2=0x7f84f80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3700375090 shutdown_connections 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3700375090 wait complete. 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.527+0000 7f84fdf1d640 1 Processor -- start 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 -- start start 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 0x7f84f8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f81980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810d520 con 0x7f84f8107560 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810d3a0 con 0x7f84f8104c30 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84fdf1d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f84f810d6a0 con 0x7f84f8102300 2024-05-09T09:18:14.530 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57518/0 (socket says 172.21.15.69:57518) 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 -- 172.21.15.69:0/3887517784 learned_addr learned my addr 172.21.15.69:0/3887517784 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f77fe640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 0x7f84f8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f6ffd640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f81980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 -- 172.21.15.69:0/3887517784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 msgr2=0x7f84f8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 0x7f84f8197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 -- 172.21.15.69:0/3887517784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 msgr2=0x7f84f81980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f81980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 -- 172.21.15.69:0/3887517784 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84f819f7b0 con 0x7f84f8107560 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f6ffd640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.528+0000 7f84f7fff640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84f0005e30 tx=0x7f84f0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.529+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f0003c30 con 0x7f84f8107560 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.529+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84f819fa40 con 0x7f84f8107560 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.529+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84f0003dd0 con 0x7f84f8107560 2024-05-09T09:18:14.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.529+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84f002fde0 con 0x7f84f8107560 2024-05-09T09:18:14.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.529+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84f819ff90 con 0x7f84f8107560 2024-05-09T09:18:14.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.530+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f84f80fd800 con 0x7f84f8107560 2024-05-09T09:18:14.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.531+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f84f0004050 con 0x7f84f8107560 2024-05-09T09:18:14.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.531+0000 7f84f4ff9640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.531+0000 7f84f77fe640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.531+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f84f00f0940 con 0x7f84f8107560 2024-05-09T09:18:14.533 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.532+0000 7f84f77fe640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 0x7f84e00a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84e8005fd0 tx=0x7f84e8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.535 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.536+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f84f00bd680 con 0x7f84f8107560 2024-05-09T09:18:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:14 smithi163 ceph-mon[34354]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:14 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2363376896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:14 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2749030844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:14 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/903221449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:14 smithi103 ceph-mon[34421]: pgmap v343: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:14 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2363376896' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:14 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2749030844' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:14 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/903221449' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:14.715 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.715+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f84f8062430 con 0x7f84f8107560 2024-05-09T09:18:14.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.716+0000 7f84f4ff9640 1 -- 172.21.15.69:0/3887517784 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) ==== 74+0+24614 (secure 0 0 0) 0x7f84f00bcf80 con 0x7f84f8107560 2024-05-09T09:18:14.716 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:14.717 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":91,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:17:30.193538+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 msgr2=0x7f84e00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 0x7f84e00a5050 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f84e8005fd0 tx=0x7f84e8005850 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 msgr2=0x7f84f819f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f84f0005e30 tx=0x7f84f0031ee0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 shutdown_connections 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84e00a2b90 0x7f84e00a5050 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84f8107560 0x7f84f819f140 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84f8104c30 0x7f84f81980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 --2- 172.21.15.69:0/3887517784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84f8102300 0x7f84f8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 >> 172.21.15.69:0/3887517784 conn(0x7f84f80fbd10 msgr2=0x7f84f8102c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 shutdown_connections 2024-05-09T09:18:14.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.719+0000 7f84fdf1d640 1 -- 172.21.15.69:0/3887517784 wait complete. 2024-05-09T09:18:14.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 -- 172.21.15.69:0/4203012772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0101970 msgr2=0x7fd1b0101df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4203012772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0101970 0x7fd1b0101df0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd1a4009900 tx=0x7fd1a402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 -- 172.21.15.69:0/4203012772 shutdown_connections 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4203012772 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0102330 0x7fd1b0108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4203012772 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0101970 0x7fd1b0101df0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4203012772 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0100780 0x7fd1b0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 -- 172.21.15.69:0/4203012772 >> 172.21.15.69:0/4203012772 conn(0x7fd1b00fbd30 msgr2=0x7fd1b00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 -- 172.21.15.69:0/4203012772 shutdown_connections 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.832+0000 7fd1b664c640 1 -- 172.21.15.69:0/4203012772 wait complete. 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 Processor -- start 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 -- start start 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 0x7fd1b0195970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101970 0x7fd1b0195eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010b310 con 0x7fd1b0102330 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010b190 con 0x7fd1b0101970 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b664c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd1b010b490 con 0x7fd1b0100780 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57542/0 (socket says 172.21.15.69:57542) 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 -- 172.21.15.69:0/4194672539 learned_addr learned my addr 172.21.15.69:0/4194672539 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1af7fe640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101970 0x7fd1b0195eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1affff640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 0x7fd1b0195970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 -- 172.21.15.69:0/4194672539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 msgr2=0x7fd1b0195970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 0x7fd1b0195970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 -- 172.21.15.69:0/4194672539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101970 msgr2=0x7fd1b0195eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101970 0x7fd1b0195eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 -- 172.21.15.69:0/4194672539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd1b019d590 con 0x7fd1b0102330 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1affff640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 0x7fd1b0195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.833+0000 7fd1b4be9640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800bfb0 tx=0x7fd1a80114a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.834+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8011dc0 con 0x7fd1b0102330 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.834+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1a8005020 con 0x7fd1b0102330 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.834+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1a8017650 con 0x7fd1b0102330 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.834+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1b019d880 con 0x7fd1b0102330 2024-05-09T09:18:14.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.834+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1b019dd50 con 0x7fd1b0102330 2024-05-09T09:18:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.835+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd1b0107aa0 con 0x7fd1b0102330 2024-05-09T09:18:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.836+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd1a801e020 con 0x7fd1b0102330 2024-05-09T09:18:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.837+0000 7fd1ad7fa640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.837+0000 7fd1affff640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:14.837 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.837+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fd1a80d3430 con 0x7fd1b0102330 2024-05-09T09:18:14.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.837+0000 7fd1affff640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 0x7fd1980a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd1b01017d0 tx=0x7fd1a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:14.840 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:14.841+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd1a8014070 con 0x7fd1b0102330 2024-05-09T09:18:15.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.054+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fd1b002d380 con 0x7fd1b0102330 2024-05-09T09:18:15.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.055+0000 7fd1ad7fa640 1 -- 172.21.15.69:0/4194672539 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7fd1b002d380 con 0x7fd1b0102330 2024-05-09T09:18:15.055 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:15.055 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:18:15.055 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 msgr2=0x7fd1980a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 0x7fd1980a5050 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fd1b01017d0 tx=0x7fd1a0005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 msgr2=0x7fd1b019cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd1a800bfb0 tx=0x7fd1a80114a0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 shutdown_connections 2024-05-09T09:18:15.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1980a2b90 0x7fd1980a5050 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1b0102330 0x7fd1b019cf20 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd1b0101970 0x7fd1b0195eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 --2- 172.21.15.69:0/4194672539 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1b0100780 0x7fd1b0195970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 >> 172.21.15.69:0/4194672539 conn(0x7fd1b00fbd30 msgr2=0x7fd1b0104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 shutdown_connections 2024-05-09T09:18:15.058 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.057+0000 7fd1b664c640 1 -- 172.21.15.69:0/4194672539 wait complete. 2024-05-09T09:18:15.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T09:16:32.211035+0000', 'modified': '2024-05-09T09:16:53.206774+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24511, 'mds_2': 24491}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/3790151668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 3790151668}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 3790151668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6835/989258568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 989258568}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 989258568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24490': {'gid': 24490, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6837/3224689113', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6836', 'nonce': 3224689113}, {'type': 'v1', 'addr': '172.21.15.163:6837', 'nonce': 3224689113}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.103:6837/1084447179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 1084447179}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1084447179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.163:6839/282166335', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 282166335}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 282166335}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/189860875', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 189860875}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 189860875}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24491, 24511, 24527]} 2024-05-09T09:18:15.069 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-09T09:18:15.069 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T09:18:15.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 -- 172.21.15.69:0/380405784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 msgr2=0x7f2068100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 --2- 172.21.15.69:0/380405784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f2068100b90 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2058009900 tx=0x7f205802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 -- 172.21.15.69:0/380405784 shutdown_connections 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 --2- 172.21.15.69:0/380405784 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068102320 0x7f2068108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 --2- 172.21.15.69:0/380405784 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068101960 0x7f2068101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 --2- 172.21.15.69:0/380405784 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f2068100b90 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 -- 172.21.15.69:0/380405784 >> 172.21.15.69:0/380405784 conn(0x7f20680fbd20 msgr2=0x7f20680fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 -- 172.21.15.69:0/380405784 shutdown_connections 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.172+0000 7f206e6d4640 1 -- 172.21.15.69:0/380405784 wait complete. 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 Processor -- start 2024-05-09T09:18:15.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 -- start start 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068101960 0x7f2068109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068102320 0x7f2068109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f206810d510 con 0x7f2068100770 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f206810d390 con 0x7f2068101960 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206e6d4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f206810d690 con 0x7f2068102320 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57558/0 (socket says 172.21.15.69:57558) 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 -- 172.21.15.69:0/490847966 learned_addr learned my addr 172.21.15.69:0/490847966 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f20677fe640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068101960 0x7f2068109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f206cc71640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068102320 0x7f2068109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 -- 172.21.15.69:0/490847966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068102320 msgr2=0x7f2068109c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068102320 0x7f2068109c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 -- 172.21.15.69:0/490847966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068101960 msgr2=0x7f2068109730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068101960 0x7f2068109730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.173+0000 7f2067fff640 1 -- 172.21.15.69:0/490847966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f206810a500 con 0x7f2068100770 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f2067fff640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2058031610 tx=0x7f2058004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f20657fa640 1 -- 172.21.15.69:0/490847966 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f205803f030 con 0x7f2068100770 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f20657fa640 1 -- 172.21.15.69:0/490847966 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2058003ad0 con 0x7f2068100770 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20681a1c80 con 0x7f2068100770 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f20657fa640 1 -- 172.21.15.69:0/490847966 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058035070 con 0x7f2068100770 2024-05-09T09:18:15.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.174+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20681a21d0 con 0x7f2068100770 2024-05-09T09:18:15.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.176+0000 7f20657fa640 1 -- 172.21.15.69:0/490847966 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2058042030 con 0x7f2068100770 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.176+0000 7f20713e7740 1 Processor -- start 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.176+0000 7f20713e7740 1 -- start start 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ee8c0 0x5580ec9eece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 0x5580ec9f68b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5580ec9524d0 con 0x5580ec9ef2b0 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580ec8fa3e0 con 0x5580ec9ee8c0 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20713e7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5580ec94e490 con 0x5580ec9f4420 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20657fa640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 0x7f20500a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f206cc71640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 0x7f20500a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57570/0 (socket says 172.21.15.69:57570) 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 -- 172.21.15.69:0/1908100491 learned_addr learned my addr 172.21.15.69:0/1908100491 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20677fe640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 0x5580ec9f68b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20657fa640 1 -- 172.21.15.69:0/490847966 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f20580efa40 con 0x7f2068100770 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f206cc71640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ee8c0 0x5580ec9eece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 -- 172.21.15.69:0/1908100491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 msgr2=0x5580ec9f68b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 0x5580ec9f68b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 -- 172.21.15.69:0/1908100491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ee8c0 msgr2=0x5580ec9eece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ee8c0 0x5580ec9eece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 -- 172.21.15.69:0/1908100491 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ec6662d0 con 0x5580ec9ef2b0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f20677fe640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 0x5580ec9f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f206cc71640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 0x7f20500a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2060009270 tx=0x7f2060012c10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f2067fff640 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2058087050 tx=0x7f205808f020 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f84ced9f23d91f70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f204e7fc640 1 -- 172.21.15.69:0/1908100491 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2058049070 con 0x5580ec9ef2b0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f204e7fc640 1 -- 172.21.15.69:0/1908100491 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20580bc660 con 0x5580ec9ef2b0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.177+0000 7f204e7fc640 1 -- 172.21.15.69:0/1908100491 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20580035f0 con 0x5580ec9ef2b0 2024-05-09T09:18:15.186 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- 172.21.15.69:0/1908100491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 msgr2=0x5580ec9f3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2058087050 tx=0x7f205808f020 comp rx=0 tx=0).stop 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- 172.21.15.69:0/1908100491 shutdown_connections 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9f4420 0x5580ec9f68b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f3db0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 --2- 172.21.15.69:0/1908100491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ee8c0 0x5580ec9eece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- 172.21.15.69:0/1908100491 >> 172.21.15.69:0/1908100491 conn(0x5580ec9e9f30 msgr2=0x5580ec9ea330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- 172.21.15.69:0/1908100491 shutdown_connections 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- 172.21.15.69:0/1908100491 wait complete. 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.178+0000 7f20713e7740 1 -- start start 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 Processor -- start 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f8940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f8e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.187 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5580ec9f7190 con 0x5580ec9f4420 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580ec90d8e0 con 0x5580ec9ef2b0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5580ec9f7310 con 0x5580ec9ee8c0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20713e7740 10 client.0 _ll_register_callbacks cb 0x5580ec8da9c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57578/0 (socket says 172.21.15.69:57578) 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 -- 172.21.15.69:0/1131618181 learned_addr learned my addr 172.21.15.69:0/1131618181 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20677fe640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f8e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f2067fff640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f8940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 -- 172.21.15.69:0/1131618181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 msgr2=0x5580ec9f8940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f8940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 -- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 msgr2=0x5580ec9f8e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f8e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 -- 172.21.15.69:0/1131618181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ec7306b0 con 0x5580ec9f4420 2024-05-09T09:18:15.188 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20677fe640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f206cc71640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2060010b80 tx=0x7f2060009500 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b86847fcab1317b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.179+0000 7f20477fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060012e60 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f2060027400 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20580bc660 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2060027600 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 10 client.15219 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5580eca02f10 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20580bc660 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f206001e070 con 0x5580ec9f4420 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 10 client.15219 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 10 client.15219 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 20 client.15219 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca055f0 0x5580eca06a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.189 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 4 client.15219 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.180+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5580ec95fc00 con 0x5580eca055f0 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.181+0000 7f2067fff640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca055f0 0x5580eca06a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.181+0000 7f20477fe640 10 client.15219 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.181+0000 7f2067fff640 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca055f0 0x5580eca06a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=25714c614cd660be server_cookie=1d0dfde00ef5a3da in_seq=0 out_seq=0 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20477fe640 1 -- 172.21.15.69:0/1131618181 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5580ec95fc00 con 0x5580eca055f0 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20477fe640 10 client.15219 handle_command_reply: tid=0 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 client.15219 shutdown 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 msgr2=0x5580ec9ffec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2060010b80 tx=0x7f2060009500 comp rx=0 tx=0).stop 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 shutdown_connections 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca055f0 0x5580eca06a60 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9ffec0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20477fe640 0 client.15219 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:15.190 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9f8e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 --2- 172.21.15.69:0/1131618181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f8940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 >> 172.21.15.69:0/1131618181 conn(0x5580ec9e9f30 msgr2=0x5580ec9e9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 shutdown_connections 2024-05-09T09:18:15.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 1 -- 172.21.15.69:0/1131618181 wait complete. 2024-05-09T09:18:15.191 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.182+0000 7f20713e7740 20 client.15219 trim_cache size 0 max 16384 2024-05-09T09:18:15.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 Processor -- start 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 -- start start 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9ffe80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5580eca04da0 con 0x5580ec9f4420 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580eca05470 con 0x5580ec9ef2b0 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20713e7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5580eca05b40 con 0x5580ec9ee8c0 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57582/0 (socket says 172.21.15.69:57582) 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f206cc71640 1 -- 172.21.15.69:0/1933058824 learned_addr learned my addr 172.21.15.69:0/1933058824 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f20677fe640 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9ffe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.200+0000 7f2067fff640 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 -- 172.21.15.69:0/1933058824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 msgr2=0x5580ec9f6810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f6810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 -- 172.21.15.69:0/1933058824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 msgr2=0x5580ec9ffe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9ffe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 -- 172.21.15.69:0/1933058824 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580ec9eaca0 con 0x5580ec9f4420 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f206cc71640 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f20600107c0 tx=0x7f2060009960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=485c39b61c32216e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f204dffb640 1 -- 172.21.15.69:0/1933058824 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060039070 con 0x5580ec9f4420 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f204dffb640 1 -- 172.21.15.69:0/1933058824 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20600046d0 con 0x5580ec9f4420 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f204dffb640 1 -- 172.21.15.69:0/1933058824 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060012e60 con 0x5580ec9f4420 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f20713e7740 1 -- 172.21.15.69:0/1933058824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 msgr2=0x5580ec9f7c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.201+0000 7f20713e7740 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f20600107c0 tx=0x7f2060009960 comp rx=0 tx=0).stop 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- 172.21.15.69:0/1933058824 shutdown_connections 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9f7c10 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9ffe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- 172.21.15.69:0/1933058824 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9f6810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- 172.21.15.69:0/1933058824 >> 172.21.15.69:0/1933058824 conn(0x5580ec9e9f30 msgr2=0x5580ec9f05d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- 172.21.15.69:0/1933058824 shutdown_connections 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- 172.21.15.69:0/1933058824 wait complete. 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- start start 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 Processor -- start 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9fb6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9fbc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5580eca068e0 con 0x5580ec9f4420 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5580eca06210 con 0x5580ec9ef2b0 2024-05-09T09:18:15.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5580eca06fb0 con 0x5580ec9ee8c0 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20713e7740 10 client.0 _ll_register_callbacks cb 0x7f2060003d90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f206cc71640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57598/0 (socket says 172.21.15.69:57598) 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f206cc71640 1 -- 172.21.15.69:0/2092548464 learned_addr learned my addr 172.21.15.69:0/2092548464 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f20677fe640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9fbc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.202+0000 7f2067fff640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9fb6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 -- 172.21.15.69:0/2092548464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 msgr2=0x5580ec9fb6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9fb6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 -- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 msgr2=0x5580ec9fbc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9fbc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 -- 172.21.15.69:0/2092548464 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5580ec9ff240 con 0x5580ec9f4420 2024-05-09T09:18:15.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20677fe640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f206cc71640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2060037040 tx=0x7f2060003cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec402df62215ef94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2060039070 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f20600279b0 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20600046d0 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f2060012e60 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 10 client.15225 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5580eca6b6d0 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20600046d0 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f206001e050 con 0x5580ec9f4420 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 10 client.15225 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 10 client.15225 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 20 client.15225 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca6feb0 0x5580eca6ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 4 client.15225 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.203+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "status"}) -- 0x5580ec959f50 con 0x5580eca6feb0 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f2067fff640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca6feb0 0x5580eca6ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20477fe640 10 client.15225 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f2067fff640 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca6feb0 0x5580eca6ebc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a80cd0e4229bc779 server_cookie=793a8f5b824b078b in_seq=0 out_seq=0 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20477fe640 1 -- 172.21.15.69:0/2092548464 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+327 (crc 0 0 0) 0x5580ec959f50 con 0x5580eca6feb0 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20477fe640 10 client.15225 handle_command_reply: tid=0 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 client.15225 shutdown 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 msgr2=0x5580ec9fec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f2060037040 tx=0x7f2060003cc0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 shutdown_connections 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5580eca6feb0 0x5580eca6ebc0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.211 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5580ec9f4420 0x5580ec9fec50 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5580ec9ef2b0 0x5580ec9fbc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 --2- 172.21.15.69:0/2092548464 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5580ec9ee8c0 0x5580ec9fb6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 >> 172.21.15.69:0/2092548464 conn(0x5580ec9e9f30 msgr2=0x5580ec9f7c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.204+0000 7f20477fe640 0 client.15225 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 shutdown_connections 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f20713e7740 1 -- 172.21.15.69:0/2092548464 wait complete. 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f20713e7740 20 client.15225 trim_cache size 0 max 16384 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 2, 2024-05-09T09:18:15.212 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24491, 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 83.010725784000002, 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 91, 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 119.995510709 2024-05-09T09:18:15.213 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 msgr2=0x7f20500a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 0x7f20500a50d0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f2060009270 tx=0x7f2060012c10 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 msgr2=0x7f206810b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2058031610 tx=0x7f2058004270 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 shutdown_connections 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20500a2c10 0x7f20500a50d0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2068102320 0x7f2068109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2068101960 0x7f2068109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 --2- 172.21.15.69:0/490847966 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2068100770 0x7f206810b0f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 >> 172.21.15.69:0/490847966 conn(0x7f20680fbd20 msgr2=0x7f2068104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 shutdown_connections 2024-05-09T09:18:15.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.205+0000 7f206e6d4640 1 -- 172.21.15.69:0/490847966 wait complete. 2024-05-09T09:18:15.216 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 -- 172.21.15.69:0/1062812382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 msgr2=0x7f15b8102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 --2- 172.21.15.69:0/1062812382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8102ef0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009900 tx=0x7f15a402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 -- 172.21.15.69:0/1062812382 shutdown_connections 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 --2- 172.21.15.69:0/1062812382 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b8103430 0x7f15b8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 --2- 172.21.15.69:0/1062812382 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8102ef0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 --2- 172.21.15.69:0/1062812382 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8101880 0x7f15b8101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 -- 172.21.15.69:0/1062812382 >> 172.21.15.69:0/1062812382 conn(0x7f15b80fce10 msgr2=0x7f15b80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.321 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 -- 172.21.15.69:0/1062812382 shutdown_connections 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.319+0000 7f15bec38640 1 -- 172.21.15.69:0/1062812382 wait complete. 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 Processor -- start 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 -- start start 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b8101880 0x7f15b8197d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 0x7f15b819f300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810d5f0 con 0x7f15b8102a70 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810d470 con 0x7f15b8101880 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15b810d770 con 0x7f15b8103430 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57614/0 (socket says 172.21.15.69:57614) 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 -- 172.21.15.69:0/3914368973 learned_addr learned my addr 172.21.15.69:0/3914368973 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 0x7f15b819f300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.322 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 -- 172.21.15.69:0/3914368973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 msgr2=0x7f15b819f300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 0x7f15b819f300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 -- 172.21.15.69:0/3914368973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b8101880 msgr2=0x7f15b8197d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b8101880 0x7f15b8197d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 -- 172.21.15.69:0/3914368973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b819f970 con 0x7f15b8102a70 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 0x7f15b819f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009230 tx=0x7f15a4004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b5ffb640 1 -- 172.21.15.69:0/3914368973 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a403f030 con 0x7f15b8102a70 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b819fc00 con 0x7f15b8102a70 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b5ffb640 1 -- 172.21.15.69:0/3914368973 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15a4003ce0 con 0x7f15b8102a70 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.320+0000 7f15b5ffb640 1 -- 172.21.15.69:0/3914368973 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15a4035070 con 0x7f15b8102a70 2024-05-09T09:18:15.323 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.321+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b81a0140 con 0x7f15b8102a70 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15b5ffb640 1 -- 172.21.15.69:0/3914368973 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f15a4042030 con 0x7f15b8102a70 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15c1a25740 1 Processor -- start 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15c1a25740 1 -- start start 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 0x5563e0046ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.323+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e004e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15c1a25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563dffaa4d0 con 0x5563e00472b0 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15c1a25740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5563dff523e0 con 0x5563e004c420 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15c1a25740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563dffa6490 con 0x5563e00468c0 2024-05-09T09:18:15.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57622/0 (socket says 172.21.15.69:57622) 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 -- 172.21.15.69:0/418480356 learned_addr learned my addr 172.21.15.69:0/418480356 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 0x5563e0046ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15b7fff640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e004e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15b5ffb640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 0x7f15a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 -- 172.21.15.69:0/418480356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 msgr2=0x5563e0046ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 0x5563e0046ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 -- 172.21.15.69:0/418480356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 msgr2=0x5563e004e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e004e8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 -- 172.21.15.69:0/418480356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563dfcbe2d0 con 0x5563e00472b0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 0x5563e0046ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0097f0 tx=0x7f15ac009cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b0f19ed6ca35abe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15b5ffb640 1 -- 172.21.15.69:0/3914368973 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f15a40efba0 con 0x7f15b8102a70 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 0x7f15a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.331 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15a2ffd640 1 -- 172.21.15.69:0/418480356 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac015020 con 0x5563e00472b0 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15a2ffd640 1 -- 172.21.15.69:0/418480356 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ac011420 con 0x5563e00472b0 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.324+0000 7f15a2ffd640 1 -- 172.21.15.69:0/418480356 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac0105e0 con 0x5563e00472b0 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- 172.21.15.69:0/418480356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 msgr2=0x5563e004bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 0x7f15a80a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5563e0047b10 tx=0x7f15b0004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7f15ac0097f0 tx=0x7f15ac009cb0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- 172.21.15.69:0/418480356 shutdown_connections 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e004e8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00472b0 0x5563e004bdb0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 --2- 172.21.15.69:0/418480356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00468c0 0x5563e0046ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- 172.21.15.69:0/418480356 >> 172.21.15.69:0/418480356 conn(0x5563e0041f30 msgr2=0x5563e0042330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- 172.21.15.69:0/418480356 shutdown_connections 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- 172.21.15.69:0/418480356 wait complete. 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 -- start start 2024-05-09T09:18:15.332 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.325+0000 7f15c1a25740 1 Processor -- start 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00511e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e0058210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563e004f3d0 con 0x5563e00468c0 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5563dff658e0 con 0x5563e004c420 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563e004f550 con 0x5563e00472b0 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 10 client.0 _ll_register_callbacks cb 0x5563dff329c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57634/0 (socket says 172.21.15.69:57634) 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 -- 172.21.15.69:0/3161239564 learned_addr learned my addr 172.21.15.69:0/3161239564 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00511e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 -- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 msgr2=0x5563e00511e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00511e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 -- 172.21.15.69:0/3161239564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 msgr2=0x5563e0058210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.333 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e0058210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 -- 172.21.15.69:0/3161239564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563dfd886b0 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00b4b0 tx=0x7f15ac006bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dee88c3b6f24233 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f1597fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00b8d0 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f15ac00ba70 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.326+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15ac011420 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f15ac02f050 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 10 client.15237 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5563e005b1f0 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ac011420 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f15ac01a430 con 0x5563e00468c0 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 10 client.15237 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 10 client.15237 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T09:18:15.334 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 20 client.15237 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e005d6e0 0x5563e005eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 4 client.15237 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5563dffb7c00 con 0x5563e005d6e0 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.327+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e005d6e0 0x5563e005eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.328+0000 7f1597fff640 10 client.15237 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.328+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e005d6e0 0x5563e005eb50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=f84e1ec4c05784c8 server_cookie=1a1c2003f083a997 in_seq=0 out_seq=0 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f1597fff640 1 -- 172.21.15.69:0/3161239564 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5563dffb7c00 con 0x5563e005d6e0 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f1597fff640 10 client.15237 handle_command_reply: tid=0 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 client.15237 shutdown 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 msgr2=0x5563e0050ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f15ac00b4b0 tx=0x7f15ac006bf0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 shutdown_connections 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e005d6e0 0x5563e005eb50 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e004c420 0x5563e0058210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f1597fff640 0 client.15237 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00511e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3161239564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e0050ca0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 >> 172.21.15.69:0/3161239564 conn(0x5563e0041f30 msgr2=0x5563e0041000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 shutdown_connections 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 1 -- 172.21.15.69:0/3161239564 wait complete. 2024-05-09T09:18:15.336 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.329+0000 7f15c1a25740 20 client.15237 trim_cache size 0 max 16384 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 Processor -- start 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 -- start start 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 0x5563e0058130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e004c420 0x5563e005c650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563e005ce90 con 0x5563e00468c0 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5563e005d560 con 0x5563e00472b0 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15c1a25740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563e005dc30 con 0x5563e004c420 2024-05-09T09:18:15.352 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57646/0 (socket says 172.21.15.69:57646) 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 -- 172.21.15.69:0/53881219 learned_addr learned my addr 172.21.15.69:0/53881219 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 -- 172.21.15.69:0/53881219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e004c420 msgr2=0x5563e005c650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15b7fff640 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 0x5563e0058130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e004c420 0x5563e005c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 -- 172.21.15.69:0/53881219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 msgr2=0x5563e0058130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 0x5563e0058130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 -- 172.21.15.69:0/53881219 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5563e0042ca0 con 0x5563e00468c0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15b7fff640 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 0x5563e0058130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.347+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009f80 tx=0x7f15ac009120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0271665cbc621e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15a27fc640 1 -- 172.21.15.69:0/53881219 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac015040 con 0x5563e00468c0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15a27fc640 1 -- 172.21.15.69:0/53881219 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ac01a430 con 0x5563e00468c0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15a27fc640 1 -- 172.21.15.69:0/53881219 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15ac00bb10 con 0x5563e00468c0 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- 172.21.15.69:0/53881219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 msgr2=0x5563e004bc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.353 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009f80 tx=0x7f15ac009120 comp rx=0 tx=0).stop 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- 172.21.15.69:0/53881219 shutdown_connections 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e004c420 0x5563e005c650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00472b0 0x5563e0058130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 --2- 172.21.15.69:0/53881219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e00468c0 0x5563e004bc60 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- 172.21.15.69:0/53881219 >> 172.21.15.69:0/53881219 conn(0x5563e0041f30 msgr2=0x5563e0042310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- 172.21.15.69:0/53881219 shutdown_connections 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- 172.21.15.69:0/53881219 wait complete. 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.348+0000 7f15c1a25740 1 -- start start 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 Processor -- start 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00468c0 0x5563e0056120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00bf670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5563e005e9d0 con 0x5563e004c420 2024-05-09T09:18:15.354 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5563e005e300 con 0x5563e00468c0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5563e005f0a0 con 0x5563e00472b0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15c1a25740 10 client.0 _ll_register_callbacks cb 0x7f15ac009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57654/0 (socket says 172.21.15.69:57654) 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 -- 172.21.15.69:0/3288213867 learned_addr learned my addr 172.21.15.69:0/3288213867 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00468c0 0x5563e0056120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00bf670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 -- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 msgr2=0x5563e00bf670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00bf670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 -- 172.21.15.69:0/3288213867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00468c0 msgr2=0x5563e0056120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00468c0 0x5563e0056120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15bd1d5640 1 -- 172.21.15.69:0/3288213867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5563e00c2260 con 0x5563e004c420 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.349+0000 7f15b7fff640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15bd1d5640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f15b0011a80 tx=0x7f15b0011e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42b723fd805c6acf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.355 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15b0022020 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f15b0010cd0 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15ac01a430 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f15b001e450 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 10 client.15243 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x5563e00c3540 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15ac01a430 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f15b001d780 con 0x5563e004c420 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 10 client.15243 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 10 client.15243 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 20 client.15243 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e00c81d0 0x5563e00c6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 4 client.15243 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.350+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "status"}) -- 0x5563dffb1f50 con 0x5563e00c81d0 2024-05-09T09:18:15.356 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e00c81d0 0x5563e00c6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f1597fff640 10 client.15243 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15bc9d4640 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e00c81d0 0x5563e00c6ee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d3372f6ecb3cec3b server_cookie=2e4c394a635a61d3 in_seq=0 out_seq=0 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f1597fff640 1 -- 172.21.15.69:0/3288213867 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x5563dffb1f50 con 0x5563e00c81d0 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f1597fff640 10 client.15243 handle_command_reply: tid=0 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 client.15243 shutdown 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 msgr2=0x5563e00c1bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f15b0011a80 tx=0x7f15b0011e80 comp rx=0 tx=0).stop 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 shutdown_connections 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5563e00c81d0 0x5563e00c6ee0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5563e004c420 0x5563e00c1bc0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5563e00472b0 0x5563e00bf670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f1597fff640 0 client.15243 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 --2- 172.21.15.69:0/3288213867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5563e00468c0 0x5563e0056120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 >> 172.21.15.69:0/3288213867 conn(0x5563e0041f30 msgr2=0x5563e004bc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.351+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 shutdown_connections 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15c1a25740 1 -- 172.21.15.69:0/3288213867 wait complete. 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15c1a25740 20 client.15243 trim_cache size 0 max 16384 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 1, 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24511, 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 84.162659923000007, 2024-05-09T09:18:15.358 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:15.359 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 91, 2024-05-09T09:18:15.359 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:15.359 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 117.33449091200001 2024-05-09T09:18:15.359 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 msgr2=0x7f15a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 0x7f15a80a5050 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x5563e0047b10 tx=0x7f15b0004b50 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 msgr2=0x7f15b8198290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f15a4009230 tx=0x7f15a4004430 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 shutdown_connections 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a80a2b90 0x7f15a80a5050 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b8103430 0x7f15b819f300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b8102a70 0x7f15b8198290 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 --2- 172.21.15.69:0/3914368973 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b8101880 0x7f15b8197d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 >> 172.21.15.69:0/3914368973 conn(0x7f15b80fce10 msgr2=0x7f15b8105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 shutdown_connections 2024-05-09T09:18:15.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.352+0000 7f15bec38640 1 -- 172.21.15.69:0/3914368973 wait complete. 2024-05-09T09:18:15.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:15 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1747388532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:15 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3887517784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:15 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4194672539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.362 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:15 smithi069 ceph-mon[30123]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:15.363 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T09:18:15.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 -- 172.21.15.69:0/4274799378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe4102b20 msgr2=0x7effe4102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 --2- 172.21.15.69:0/4274799378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe4102b20 0x7effe4102fa0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7effd4009930 tx=0x7effd402a420 comp rx=0 tx=0).stop 2024-05-09T09:18:15.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 -- 172.21.15.69:0/4274799378 shutdown_connections 2024-05-09T09:18:15.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 --2- 172.21.15.69:0/4274799378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effe41034e0 0x7effe4109d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 --2- 172.21.15.69:0/4274799378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe4102b20 0x7effe4102fa0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 --2- 172.21.15.69:0/4274799378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe40ffb50 0x7effe40fff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 -- 172.21.15.69:0/4274799378 >> 172.21.15.69:0/4274799378 conn(0x7effe40fb460 msgr2=0x7effe40fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 -- 172.21.15.69:0/4274799378 shutdown_connections 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.465+0000 7effec5b0640 1 -- 172.21.15.69:0/4274799378 wait complete. 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 Processor -- start 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 -- start start 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe4102b20 0x7effe4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effe41034e0 0x7effe419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7effe410d530 con 0x7effe40ffb50 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7effe410d3b0 con 0x7effe4102b20 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effec5b0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7effe410d6b0 con 0x7effe41034e0 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57672/0 (socket says 172.21.15.69:57672) 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 -- 172.21.15.69:0/93588156 learned_addr learned my addr 172.21.15.69:0/93588156 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effe9b4b640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe4102b20 0x7effe4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 -- 172.21.15.69:0/93588156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effe41034e0 msgr2=0x7effe419f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effe41034e0 0x7effe419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 -- 172.21.15.69:0/93588156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe4102b20 msgr2=0x7effe4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe4102b20 0x7effe4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 -- 172.21.15.69:0/93588156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7effe419f740 con 0x7effe40ffb50 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.466+0000 7effea34c640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7effe000c970 tx=0x7effe000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.467+0000 7effdb7fe640 1 -- 172.21.15.69:0/93588156 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0007c10 con 0x7effe40ffb50 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.467+0000 7effdb7fe640 1 -- 172.21.15.69:0/93588156 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe0007db0 con 0x7effe40ffb50 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.467+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7effe419fa30 con 0x7effe40ffb50 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.467+0000 7effdb7fe640 1 -- 172.21.15.69:0/93588156 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe000f940 con 0x7effe40ffb50 2024-05-09T09:18:15.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.467+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7effe419ff50 con 0x7effe40ffb50 2024-05-09T09:18:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.469+0000 7effdb7fe640 1 -- 172.21.15.69:0/93588156 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7effe000fae0 con 0x7effe40ffb50 2024-05-09T09:18:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.469+0000 7effef39e740 1 Processor -- start 2024-05-09T09:18:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.469+0000 7effef39e740 1 -- start start 2024-05-09T09:18:15.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cbc0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75cc3cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560f75c1b4d0 con 0x560f75cbc6c0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f75bc33e0 con 0x560f75cc1830 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effef39e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f75c17490 con 0x560f75cbbcd0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57674/0 (socket says 172.21.15.69:57674) 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 -- 172.21.15.69:0/354673149 learned_addr learned my addr 172.21.15.69:0/354673149 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effdb7fe640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 0x7effd00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effeab4d640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cbc0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effeab4d640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 0x7effd00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effe9b4b640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75cc3cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 -- 172.21.15.69:0/354673149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 msgr2=0x560f75cbc0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cbc0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 -- 172.21.15.69:0/354673149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 msgr2=0x560f75cc3cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75cc3cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 -- 172.21.15.69:0/354673149 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f7592f2d0 con 0x560f75cbc6c0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effe9b4b640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75cc3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effdb7fe640 1 -- 172.21.15.69:0/93588156 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7effe00cf890 con 0x7effe40ffb50 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effea34c640 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effe009bd70 tx=0x7effe0020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=899a4669b8a2afaf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.470+0000 7effeab4d640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 0x7effd00a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effdc009270 tx=0x7effdc005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effd8bfe640 1 -- 172.21.15.69:0/354673149 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0016c30 con 0x560f75cbc6c0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effd8bfe640 1 -- 172.21.15.69:0/354673149 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe009c4e0 con 0x560f75cbc6c0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effd8bfe640 1 -- 172.21.15.69:0/354673149 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0012e10 con 0x560f75cbc6c0 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 -- 172.21.15.69:0/354673149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 msgr2=0x560f75cc11c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7effe009bd70 tx=0x7effe0020000 comp rx=0 tx=0).stop 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 -- 172.21.15.69:0/354673149 shutdown_connections 2024-05-09T09:18:15.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75cc3cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbc6c0 0x560f75cc11c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 --2- 172.21.15.69:0/354673149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cbc0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 -- 172.21.15.69:0/354673149 >> 172.21.15.69:0/354673149 conn(0x560f75cb2f30 msgr2=0x560f75cb3330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 -- 172.21.15.69:0/354673149 shutdown_connections 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.471+0000 7effef39e740 1 -- 172.21.15.69:0/354673149 wait complete. 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 -- start start 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 Processor -- start 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cc6070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75cc65b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560f75cc4780 con 0x560f75cc1830 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f75bd68e0 con 0x560f75cbc6c0 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f75cc4900 con 0x560f75cbbcd0 2024-05-09T09:18:15.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effef39e740 10 client.0 _ll_register_callbacks cb 0x560f75ba39c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57682/0 (socket says 172.21.15.69:57682) 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 -- 172.21.15.69:0/1101361437 learned_addr learned my addr 172.21.15.69:0/1101361437 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effea34c640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cc6070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effe9b4b640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75cc65b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 -- 172.21.15.69:0/1101361437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 msgr2=0x560f75cc6070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cc6070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 -- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 msgr2=0x560f75cc65b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75cc65b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effeab4d640 1 -- 172.21.15.69:0/1101361437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f759f96b0 con 0x560f75cc1830 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.472+0000 7effea34c640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cc6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effeab4d640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effdc009550 tx=0x7effdc004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=185aa609ed2b7786 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effdc0109a0 con 0x560f75cc1830 2024-05-09T09:18:15.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7effdc010b40 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effe009c4e0 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7effdc02e050 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 10 client.15255 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560f75cd0610 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe009c4e0 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7effdc017030 con 0x560f75cc1830 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 10 client.15255 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 10 client.15255 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 20 client.15255 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75cd2cf0 0x560f75cd4160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 4 client.15255 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.473+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560f75c28c00 con 0x560f75cd2cf0 2024-05-09T09:18:15.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.474+0000 7effea34c640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75cd2cf0 0x560f75cd4160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.474+0000 7effcdffb640 10 client.15255 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.474+0000 7effea34c640 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75cd2cf0 0x560f75cd4160 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20dc16189eed9196 server_cookie=fb884454705ed212 in_seq=0 out_seq=0 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effcdffb640 1 -- 172.21.15.69:0/1101361437 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560f75c28c00 con 0x560f75cd2cf0 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effcdffb640 10 client.15255 handle_command_reply: tid=0 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 client.15255 shutdown 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 msgr2=0x560f75ccd5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7effdc009550 tx=0x7effdc004a40 comp rx=0 tx=0).stop 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 shutdown_connections 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cc1830 0x560f75ccd5f0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75cc65b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75cd2cf0 0x560f75cd4160 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 --2- 172.21.15.69:0/1101361437 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbbcd0 0x560f75cc6070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effcdffb640 0 client.15255 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:15.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 >> 172.21.15.69:0/1101361437 conn(0x560f75cb2f30 msgr2=0x560f75cb2000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 shutdown_connections 2024-05-09T09:18:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 1 -- 172.21.15.69:0/1101361437 wait complete. 2024-05-09T09:18:15.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.475+0000 7effef39e740 20 client.15255 trim_cache size 0 max 16384 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 Processor -- start 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 -- start start 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75ccd5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cc1830 0x560f75cbda20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd24a0 con 0x560f75cbbcd0 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd2b70 con 0x560f75cbc6c0 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effef39e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd3240 con 0x560f75cc1830 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57696/0 (socket says 172.21.15.69:57696) 2024-05-09T09:18:15.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 -- 172.21.15.69:0/1114438077 learned_addr learned my addr 172.21.15.69:0/1114438077 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 -- 172.21.15.69:0/1114438077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cc1830 msgr2=0x560f75cbda20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cc1830 0x560f75cbda20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 -- 172.21.15.69:0/1114438077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 msgr2=0x560f75ccd5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75ccd5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 -- 172.21.15.69:0/1114438077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560f75cbcaa0 con 0x560f75cbbcd0 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effea34c640 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effe00987b0 tx=0x7effe00cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c8de2397bc888b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effcffff640 1 -- 172.21.15.69:0/1114438077 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe0099e80 con 0x560f75cbbcd0 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effcffff640 1 -- 172.21.15.69:0/1114438077 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe009f070 con 0x560f75cbbcd0 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.493+0000 7effcffff640 1 -- 172.21.15.69:0/1114438077 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe009c040 con 0x560f75cbbcd0 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- 172.21.15.69:0/1114438077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 msgr2=0x560f75cd1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7effe00987b0 tx=0x7effe00cf040 comp rx=0 tx=0).stop 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- 172.21.15.69:0/1114438077 shutdown_connections 2024-05-09T09:18:15.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cc1830 0x560f75cbda20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cbc6c0 0x560f75ccd5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 --2- 172.21.15.69:0/1114438077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cd1a80 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- 172.21.15.69:0/1114438077 >> 172.21.15.69:0/1114438077 conn(0x560f75cb2f30 msgr2=0x560f75cb3310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- 172.21.15.69:0/1114438077 shutdown_connections 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- 172.21.15.69:0/1114438077 wait complete. 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 -- start start 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.494+0000 7effef39e740 1 Processor -- start 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbc6c0 0x560f75cc8840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75d36c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd3fe0 con 0x560f75cbbcd0 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd3910 con 0x560f75cc1830 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560f75cd46b0 con 0x560f75cbc6c0 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 10 client.0 _ll_register_callbacks cb 0x7effdc005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:15.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57704/0 (socket says 172.21.15.69:57704) 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 -- 172.21.15.69:0/759386511 learned_addr learned my addr 172.21.15.69:0/759386511 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 -- 172.21.15.69:0/759386511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbc6c0 msgr2=0x560f75cc8840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbc6c0 0x560f75cc8840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 -- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 msgr2=0x560f75d36c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75d36c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 -- 172.21.15.69:0/759386511 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560f75cc8e80 con 0x560f75cbbcd0 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effea34c640 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effe00a4490 tx=0x7effe0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16acb08e6c444d9a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:15.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effcdffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7effe009c4e0 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7effe009a0e0 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7effe009f070 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7effe0018b80 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 10 client.15261 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.495+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x560f75d38260 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7effe009f070 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7effe00d1ce0 con 0x560f75cbbcd0 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 10 client.15261 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 10 client.15261 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 20 client.15261 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75d3c800 0x560f75d3b510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 4 client.15261 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "status"}) -- 0x560f75c22f50 con 0x560f75d3c800 2024-05-09T09:18:15.503 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effe9b4b640 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75d3c800 0x560f75d3b510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effe9b4b640 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75d3c800 0x560f75d3b510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5190b5fca0f62ac0 server_cookie=5ed38c0729cfb889 in_seq=0 out_seq=0 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.496+0000 7effcdffb640 10 client.15261 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effcdffb640 1 -- 172.21.15.69:0/759386511 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x560f75c22f50 con 0x560f75d3c800 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effcdffb640 10 client.15261 handle_command_reply: tid=0 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 client.15261 shutdown 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 msgr2=0x560f75cc8300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7effe00a4490 tx=0x7effe0000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 shutdown_connections 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560f75cc1830 0x560f75d36c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560f75d3c800 0x560f75d3b510 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560f75cbc6c0 0x560f75cc8840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effcdffb640 0 client.15261 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 --2- 172.21.15.69:0/759386511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560f75cbbcd0 0x560f75cc8300 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 >> 172.21.15.69:0/759386511 conn(0x560f75cb2f30 msgr2=0x560f75cbda20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 shutdown_connections 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 1 -- 172.21.15.69:0/759386511 wait complete. 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effef39e740 20 client.15261 trim_cache size 0 max 16384 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:15.505 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24527, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 102.639650977, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 91, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 108.757059688 2024-05-09T09:18:15.506 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 msgr2=0x7effd00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 0x7effd00a5050 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7effdc009270 tx=0x7effdc005bd0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 msgr2=0x7effe4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7effe000c970 tx=0x7effe000ce30 comp rx=0 tx=0).stop 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 shutdown_connections 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7effd00a2b90 0x7effd00a5050 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7effe41034e0 0x7effe419f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7effe4102b20 0x7effe4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 --2- 172.21.15.69:0/93588156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7effe40ffb50 0x7effe4197b20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:15.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 >> 172.21.15.69:0/93588156 conn(0x7effe40fb460 msgr2=0x7effe4105ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:15.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.497+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 shutdown_connections 2024-05-09T09:18:15.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:15.498+0000 7effec5b0640 1 -- 172.21.15.69:0/93588156 wait complete. 2024-05-09T09:18:15.509 INFO:tasks.cephadm:Setup complete, yielding 2024-05-09T09:18:15.509 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T09:18:15.524 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:18:15.524 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph orch status' 2024-05-09T09:18:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:15 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1747388532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:15 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3887517784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:15 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4194672539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:15 smithi163 ceph-mon[34354]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:15 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1747388532' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:15 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3887517784' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:15 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4194672539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:15 smithi103 ceph-mon[34421]: pgmap v344: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:15.762 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:17.541 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 -- 172.21.15.69:0/718635054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 msgr2=0x7f8be81096f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/718635054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be81096f0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f8be000b0d0 tx=0x7f8be000b3e0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 -- 172.21.15.69:0/718635054 shutdown_connections 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/718635054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be81096f0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/718635054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81012a0 0x7f8be81016c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/718635054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81024a0 0x7f8be8102900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 -- 172.21.15.69:0/718635054 >> 172.21.15.69:0/718635054 conn(0x7f8be80fc880 msgr2=0x7f8be80feca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 -- 172.21.15.69:0/718635054 shutdown_connections 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.531+0000 7f8bee9e8700 1 -- 172.21.15.69:0/718635054 wait complete. 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 Processor -- start 2024-05-09T09:18:17.542 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 -- start start 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 0x7f8be8197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81024a0 0x7f8be8197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8be810cf80 con 0x7f8be8102e40 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8be810ce00 con 0x7f8be81012a0 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8bee9e8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8be810d100 con 0x7f8be81024a0 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8becf85700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8becf85700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50598/0 (socket says 172.21.15.69:50598) 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.532+0000 7f8becf85700 1 -- 172.21.15.69:0/15256720 learned_addr learned my addr 172.21.15.69:0/15256720 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:17.543 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be7fff700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 0x7f8be8197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be77fe700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81024a0 0x7f8be8197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 -- 172.21.15.69:0/15256720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81024a0 msgr2=0x7f8be8197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81024a0 0x7f8be8197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 -- 172.21.15.69:0/15256720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 msgr2=0x7f8be8197560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 0x7f8be8197560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 -- 172.21.15.69:0/15256720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8be819f1c0 con 0x7f8be8102e40 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be7fff700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 0x7f8be8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8becf85700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006070 tx=0x7f8be0007e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0003c00 con 0x7f8be8102e40 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8be819f450 con 0x7f8be8102e40 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8be0003da0 con 0x7f8be8102e40 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8be0005570 con 0x7f8be8102e40 2024-05-09T09:18:17.544 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.533+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8be819fa00 con 0x7f8be8102e40 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.534+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8be8069010 con 0x7f8be8102e40 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.535+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8be0005710 con 0x7f8be8102e40 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.536+0000 7f8be57fa700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 0x7f8bd00a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.536+0000 7f8be7fff700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 0x7f8bd00a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.536+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f8be000f020 con 0x7f8be8102e40 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.536+0000 7f8be7fff700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 0x7f8bd00a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8be8102300 tx=0x7f8bdc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:17.545 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.540+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8be002b020 con 0x7f8be8102e40 2024-05-09T09:18:17.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.754+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) -- 0x7f8be8062760 con 0x7f8bd00a2c20 2024-05-09T09:18:17.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.755+0000 7f8be57fa700 1 -- 172.21.15.69:0/15256720 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+43 (secure 0 0 0) 0x7f8be8062760 con 0x7f8bd00a2c20 2024-05-09T09:18:17.755 INFO:teuthology.orchestra.run.smithi069.stdout:Backend: cephadm 2024-05-09T09:18:17.755 INFO:teuthology.orchestra.run.smithi069.stdout:Available: Yes 2024-05-09T09:18:17.756 INFO:teuthology.orchestra.run.smithi069.stdout:Paused: No 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 msgr2=0x7f8bd00a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 0x7f8bd00a50e0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f8be8102300 tx=0x7f8bdc008040 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 msgr2=0x7f8be819eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f8be0006070 tx=0x7f8be0007e40 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 shutdown_connections 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8be8102e40 0x7f8be819eb20 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8bd00a2c20 0x7f8bd00a50e0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8be81012a0 0x7f8be8197560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 --2- 172.21.15.69:0/15256720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8be81024a0 0x7f8be8197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.757+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 >> 172.21.15.69:0/15256720 conn(0x7f8be80fc880 msgr2=0x7f8be81057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.758+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 shutdown_connections 2024-05-09T09:18:17.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:17.758+0000 7f8bee9e8700 1 -- 172.21.15.69:0/15256720 wait complete. 2024-05-09T09:18:18.450 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:18 smithi069 ceph-mon[30123]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:18.478 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ps' 2024-05-09T09:18:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:18 smithi163 ceph-mon[34354]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:18 smithi103 ceph-mon[34421]: pgmap v345: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:18.719 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:19 smithi069 ceph-mon[30123]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:19 smithi163 ceph-mon[34354]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:19 smithi103 ceph-mon[34421]: from='client.15267 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:20.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- 172.21.15.69:0/2202143061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc0103700 msgr2=0x7fdfc0103b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:20.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/2202143061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc0103700 0x7fdfc0103b60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fdfb0009990 tx=0x7fdfb0009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- 172.21.15.69:0/2202143061 shutdown_connections 2024-05-09T09:18:20.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/2202143061 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc01040a0 0x7fdfc010a950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/2202143061 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 0x7fdfc0100b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/2202143061 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc0103700 0x7fdfc0103b60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- 172.21.15.69:0/2202143061 >> 172.21.15.69:0/2202143061 conn(0x7fdfc00fc080 msgr2=0x7fdfc00fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- 172.21.15.69:0/2202143061 shutdown_connections 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- 172.21.15.69:0/2202143061 wait complete. 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 Processor -- start 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 -- start start 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 0x7fdfc0103060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.389+0000 7fdfc7d59700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc0103700 0x7fdfc01016b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc010e040 con 0x7fdfc01040a0 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc010dec0 con 0x7fdfc0100720 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdfc010e1c0 con 0x7fdfc0103700 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:20.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50616/0 (socket says 172.21.15.69:50616) 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 -- 172.21.15.69:0/1273092254 learned_addr learned my addr 172.21.15.69:0/1273092254 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc52f4700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc0103700 0x7fdfc01016b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc5af5700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 0x7fdfc0103060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 -- 172.21.15.69:0/1273092254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc0103700 msgr2=0x7fdfc01016b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc0103700 0x7fdfc01016b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 -- 172.21.15.69:0/1273092254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 msgr2=0x7fdfc0103060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 0x7fdfc0103060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 -- 172.21.15.69:0/1273092254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdfc01024a0 con 0x7fdfc01040a0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc62f6700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800ada0 tx=0x7fdfb800c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8017020 con 0x7fdfc01040a0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdfc01a2800 con 0x7fdfc01040a0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdfb800a0d0 con 0x7fdfc01040a0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdfb8005480 con 0x7fdfc01040a0 2024-05-09T09:18:20.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.390+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdfc01a2d10 con 0x7fdfc01040a0 2024-05-09T09:18:20.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.391+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdfc0069010 con 0x7fdfc01040a0 2024-05-09T09:18:20.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.392+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdfb8005620 con 0x7fdfc01040a0 2024-05-09T09:18:20.393 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.393+0000 7fdfb6ffd700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 0x7fdfac0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:20.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.393+0000 7fdfc5af5700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 0x7fdfac0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:20.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.393+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7fdfb80cf0c0 con 0x7fdfc01040a0 2024-05-09T09:18:20.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.393+0000 7fdfc5af5700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 0x7fdfac0a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005fd0 tx=0x7fdfbc005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:20.396 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.396+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdfb8021020 con 0x7fdfc01040a0 2024-05-09T09:18:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:20 smithi069 ceph-mon[30123]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:20 smithi163 ceph-mon[34354]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:20.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.605+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) -- 0x7fdfc0062760 con 0x7fdfac0a2c20 2024-05-09T09:18:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:20 smithi103 ceph-mon[34421]: pgmap v346: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.618+0000 7fdfb6ffd700 1 -- 172.21.15.69:0/1273092254 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+4185 (secure 0 0 0) 0x7fdfc0062760 con 0x7fdfac0a2c20 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.a smithi069 running (117s) 25s ago 116s 12.8M - 19.0.0-2515-g96805dcf d7acac7d5657 faa5f4b26d2e 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.b smithi103 running (114s) 47s ago 113s 28.4M - 19.0.0-2515-g96805dcf d7acac7d5657 6dee82707971 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.c smithi163 running (2m) 47s ago 2m 18.6M - 19.0.0-2515-g96805dcf d7acac7d5657 0519309381ca 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.d smithi069 running (2m) 25s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 eaa752f966d3 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.e smithi103 running (2m) 47s ago 2m 17.5M - 19.0.0-2515-g96805dcf d7acac7d5657 1605d788409f 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.f smithi163 running (2m) 47s ago 2m 15.1M - 19.0.0-2515-g96805dcf d7acac7d5657 aad47eda99fe 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.g smithi069 running (2m) 25s ago 119s 16.1M - 19.0.0-2515-g96805dcf d7acac7d5657 6ddbe82800b8 2024-05-09T09:18:20.618 INFO:teuthology.orchestra.run.smithi069.stdout:mds.h smithi103 running (2m) 47s ago 2m 17.1M - 19.0.0-2515-g96805dcf d7acac7d5657 7143ffd289bd 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mds.i smithi163 running (2m) 47s ago 2m 16.7M - 19.0.0-2515-g96805dcf d7acac7d5657 87c090455bbc 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mds.j smithi069 running (2m) 25s ago 2m 16.2M - 19.0.0-2515-g96805dcf d7acac7d5657 bef9aa007455 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mds.k smithi103 running (2m) 47s ago 2m 16.4M - 19.0.0-2515-g96805dcf d7acac7d5657 e77657395d34 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mds.l smithi163 running (2m) 47s ago 2m 13.1M - 19.0.0-2515-g96805dcf d7acac7d5657 6f32cfbe6813 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.x smithi069 *:9283,8765 running (9m) 25s ago 9m 514M - 19.0.0-2515-g96805dcf d7acac7d5657 15f5ef413d46 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.y smithi103 *:8443,8765 running (7m) 47s ago 7m 454M - 19.0.0-2515-g96805dcf d7acac7d5657 1d980e30dd6a 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mgr.z smithi163 *:8443,8765 running (7m) 47s ago 7m 456M - 19.0.0-2515-g96805dcf d7acac7d5657 ac5e3177a566 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mon.a smithi069 running (9m) 25s ago 9m 49.7M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 6a7dcd9185a2 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mon.b smithi103 running (7m) 47s ago 7m 39.7M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 65ad0f4b3714 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:mon.c smithi163 running (8m) 47s ago 8m 39.7M 2048M 19.0.0-2515-g96805dcf d7acac7d5657 39dbdc4607cd 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 smithi069 running (7m) 25s ago 7m 45.8M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 48a0ae1ad5aa 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 smithi103 running (7m) 47s ago 7m 46.3M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 477b99ff28b9 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 smithi163 running (6m) 47s ago 6m 49.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 97e859dd79bc 2024-05-09T09:18:20.619 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 smithi069 running (6m) 25s ago 6m 46.9M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 1d3cd7b10f71 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 smithi103 running (5m) 47s ago 5m 45.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 1cc7535215d7 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 smithi163 running (5m) 47s ago 5m 55.8M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 4ac1380d2892 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 smithi069 running (4m) 25s ago 4m 53.2M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 f03f5af84cdd 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 smithi103 running (4m) 47s ago 4m 55.1M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 52d2a990c247 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.8 smithi163 running (4m) 47s ago 4m 45.5M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 e33eafbf32b4 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.9 smithi069 running (3m) 25s ago 3m 41.7M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 753948ff2fd7 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.10 smithi103 running (3m) 47s ago 3m 41.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 ca1f32f32ea3 2024-05-09T09:18:20.620 INFO:teuthology.orchestra.run.smithi069.stdout:osd.11 smithi163 running (2m) 47s ago 2m 52.6M 4272M 19.0.0-2515-g96805dcf d7acac7d5657 cd7114a9398d 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 msgr2=0x7fdfac0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 0x7fdfac0a50e0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdfbc005fd0 tx=0x7fdfbc005dd0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 msgr2=0x7fdfc0101bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800ada0 tx=0x7fdfb800c330 comp rx=0 tx=0).stop 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 shutdown_connections 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdfc01040a0 0x7fdfc0101bf0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdfac0a2c20 0x7fdfac0a50e0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdfc0100720 0x7fdfc0103060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 --2- 172.21.15.69:0/1273092254 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdfc0103700 0x7fdfc01016b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:20.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 >> 172.21.15.69:0/1273092254 conn(0x7fdfc00fc080 msgr2=0x7fdfc01069a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:20.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 shutdown_connections 2024-05-09T09:18:20.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:20.620+0000 7fdfc7d59700 1 -- 172.21.15.69:0/1273092254 wait complete. 2024-05-09T09:18:21.361 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph orch ls' 2024-05-09T09:18:21.611 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:22 smithi069 ceph-mon[30123]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:22 smithi069 ceph-mon[30123]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:22 smithi163 ceph-mon[34354]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:22 smithi163 ceph-mon[34354]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:22 smithi103 ceph-mon[34421]: from='client.15273 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:22 smithi103 ceph-mon[34421]: pgmap v347: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:23.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 -- 172.21.15.69:0/2887355091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80105740 msgr2=0x7efe80107b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:24.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 --2- 172.21.15.69:0/2887355091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80105740 0x7efe80107b30 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7efe74009990 tx=0x7efe74009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 -- 172.21.15.69:0/2887355091 shutdown_connections 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 --2- 172.21.15.69:0/2887355091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80108070 0x7efe8010a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 --2- 172.21.15.69:0/2887355091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 0x7efe80105200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 --2- 172.21.15.69:0/2887355091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80105740 0x7efe80107b30 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 -- 172.21.15.69:0/2887355091 >> 172.21.15.69:0/2887355091 conn(0x7efe800fc820 msgr2=0x7efe800fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 -- 172.21.15.69:0/2887355091 shutdown_connections 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.208+0000 7efe865a1700 1 -- 172.21.15.69:0/2887355091 wait complete. 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 Processor -- start 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 -- start start 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 0x7efe80102770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80105740 0x7efe80100dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:24.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efe8010dfd0 con 0x7efe80108070 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efe8010de50 con 0x7efe80102e10 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe865a1700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efe8010e150 con 0x7efe80105740 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50620/0 (socket says 172.21.15.69:50620) 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 -- 172.21.15.69:0/3005602775 learned_addr learned my addr 172.21.15.69:0/3005602775 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe7ffff700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 0x7efe80102770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe7f7fe700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80105740 0x7efe80100dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 -- 172.21.15.69:0/3005602775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80105740 msgr2=0x7efe80100dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80105740 0x7efe80100dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 -- 172.21.15.69:0/3005602775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 msgr2=0x7efe80102770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.209+0000 7efe84b3e700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 0x7efe80102770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe84b3e700 1 -- 172.21.15.69:0/3005602775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efe80101bb0 con 0x7efe80108070 2024-05-09T09:18:24.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe84b3e700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe7800b9e0 tx=0x7efe7800bcf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78019070 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efe801a2740 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efe7800ce10 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efe78012340 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.210+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efe801a2d00 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.211+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efe80069010 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.212+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efe7801c070 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.213+0000 7efe7d7fa700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 0x7efe680a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.213+0000 7efe7ffff700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 0x7efe680a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.213+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7efe780ce6e0 con 0x7efe80108070 2024-05-09T09:18:24.017 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.213+0000 7efe7ffff700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 0x7efe680a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efe70005950 tx=0x7efe7000a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:24.018 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.218+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efe7809b2a0 con 0x7efe80108070 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.423+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) -- 0x7efe80062760 con 0x7efe680a2ca0 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.429+0000 7efe7d7fa700 1 -- 172.21.15.69:0/3005602775 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+980 (secure 0 0 0) 0x7efe80062760 con 0x7efe680a2ca0 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stdout:mds.all 12/12 50s ago 2m smithi069=a;smithi069=d;smithi069=g;smithi069=j;smithi103=b;smithi103=e;smithi103=h;smithi103=k;smithi163=c;smithi163=f;smithi163=i;smithi163=l;count:12 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stdout:mgr 3/3 50s ago 7m smithi069=x;smithi103=y;smithi163=z;count:3 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stdout:mon 3/3 50s ago 8m smithi069:172.21.15.69=a;smithi103:172.21.15.103=b;smithi163:172.21.15.163=c;count:3 2024-05-09T09:18:24.019 INFO:teuthology.orchestra.run.smithi069.stdout:osd 12 50s ago - 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 msgr2=0x7efe680a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 0x7efe680a5160 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7efe70005950 tx=0x7efe7000a300 comp rx=0 tx=0).stop 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 msgr2=0x7efe80101300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7efe7800b9e0 tx=0x7efe7800bcf0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 shutdown_connections 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efe80108070 0x7efe80101300 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.020 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efe680a2ca0 0x7efe680a5160 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efe80102e10 0x7efe80102770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 --2- 172.21.15.69:0/3005602775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efe80105740 0x7efe80100dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:24.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.431+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 >> 172.21.15.69:0/3005602775 conn(0x7efe800fc820 msgr2=0x7efe80103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:24.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.432+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 shutdown_connections 2024-05-09T09:18:24.021 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:23.432+0000 7efe865a1700 1 -- 172.21.15.69:0/3005602775 wait complete. 2024-05-09T09:18:24.155 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph orch host ls' 2024-05-09T09:18:24.401 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:24.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:24 smithi069 ceph-mon[30123]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:24 smithi163 ceph-mon[34354]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:24 smithi103 ceph-mon[34421]: pgmap v348: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:25.502 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:25 smithi069 ceph-mon[30123]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:25 smithi163 ceph-mon[34354]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:25 smithi103 ceph-mon[34421]: from='client.15279 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:25.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 -- 172.21.15.69:0/4134677520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 msgr2=0x7f7be410a500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 --2- 172.21.15.69:0/4134677520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be410a500 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00b0a0 tx=0x7f7bdc00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 -- 172.21.15.69:0/4134677520 shutdown_connections 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 --2- 172.21.15.69:0/4134677520 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be410a500 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 --2- 172.21.15.69:0/4134677520 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 0x7f7be41052a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 --2- 172.21.15.69:0/4134677520 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 0x7f7be4107bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 -- 172.21.15.69:0/4134677520 >> 172.21.15.69:0/4134677520 conn(0x7f7be40fc820 msgr2=0x7f7be40fec60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 -- 172.21.15.69:0/4134677520 shutdown_connections 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.984+0000 7f7be985a700 1 -- 172.21.15.69:0/4134677520 wait complete. 2024-05-09T09:18:25.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 Processor -- start 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 -- start start 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 0x7f7be4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 0x7f7be4198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7be410e030 con 0x7f7be4108110 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7be410deb0 con 0x7f7be41057e0 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be985a700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7be410e1b0 con 0x7f7be4102eb0 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.985+0000 7f7be37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:25.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50634/0 (socket says 172.21.15.69:50634) 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 -- 172.21.15.69:0/3494639576 learned_addr learned my addr 172.21.15.69:0/3494639576 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be27fc700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 0x7f7be4198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be2ffd700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 0x7f7be4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 -- 172.21.15.69:0/3494639576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 msgr2=0x7f7be4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 0x7f7be4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 -- 172.21.15.69:0/3494639576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 msgr2=0x7f7be4198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 0x7f7be4198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 -- 172.21.15.69:0/3494639576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7be41a0280 con 0x7f7be4108110 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be37fe700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00d010 tx=0x7f7bdc00be60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc021070 con 0x7f7be4108110 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7bdc007c80 con 0x7f7be4108110 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7be41a0510 con 0x7f7be4108110 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7bdc01d450 con 0x7f7be4108110 2024-05-09T09:18:25.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.986+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7be41a0a40 con 0x7f7be4108110 2024-05-09T09:18:25.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.987+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7be4069010 con 0x7f7be4108110 2024-05-09T09:18:25.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.988+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7bdc004190 con 0x7f7be4108110 2024-05-09T09:18:25.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.989+0000 7f7be8858700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 0x7f7bcc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:25.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.990+0000 7f7be2ffd700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 0x7f7bcc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:25.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.990+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f7bdc00f020 con 0x7f7be4108110 2024-05-09T09:18:25.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.990+0000 7f7be2ffd700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 0x7f7bcc0a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009a10 tx=0x7f7bd8009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:25.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:25.993+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7bdc02a020 con 0x7f7be4108110 2024-05-09T09:18:26.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.192+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) -- 0x7f7be4062760 con 0x7f7bcc0a2c20 2024-05-09T09:18:26.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.194+0000 7f7be8858700 1 -- 172.21.15.69:0/3494639576 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+191 (secure 0 0 0) 0x7f7be4062760 con 0x7f7bcc0a2c20 2024-05-09T09:18:26.193 INFO:teuthology.orchestra.run.smithi069.stdout:HOST ADDR LABELS STATUS 2024-05-09T09:18:26.194 INFO:teuthology.orchestra.run.smithi069.stdout:smithi069 172.21.15.69 2024-05-09T09:18:26.194 INFO:teuthology.orchestra.run.smithi069.stdout:smithi103 172.21.15.103 2024-05-09T09:18:26.194 INFO:teuthology.orchestra.run.smithi069.stdout:smithi163 172.21.15.163 2024-05-09T09:18:26.194 INFO:teuthology.orchestra.run.smithi069.stdout:3 hosts in cluster 2024-05-09T09:18:26.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 msgr2=0x7f7bcc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 0x7f7bcc0a50e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f7bd8009a10 tx=0x7f7bd8009190 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 msgr2=0x7f7be419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00d010 tx=0x7f7bdc00be60 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 shutdown_connections 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7be4108110 0x7f7be419fbe0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7be4102eb0 0x7f7be4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7bcc0a2c20 0x7f7bcc0a50e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 --2- 172.21.15.69:0/3494639576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7be41057e0 0x7f7be4198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 >> 172.21.15.69:0/3494639576 conn(0x7f7be40fc820 msgr2=0x7f7be4103750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 shutdown_connections 2024-05-09T09:18:26.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:26.196+0000 7f7be985a700 1 -- 172.21.15.69:0/3494639576 wait complete. 2024-05-09T09:18:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:26 smithi069 ceph-mon[30123]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:26 smithi163 ceph-mon[34354]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:26 smithi103 ceph-mon[34421]: pgmap v349: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:26.989 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph orch device ls' 2024-05-09T09:18:27.230 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:27.259 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:27 smithi069 ceph-mon[30123]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:27 smithi163 ceph-mon[34354]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:27 smithi103 ceph-mon[34421]: from='client.15285 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:28.429 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:28 smithi069 ceph-mon[30123]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:28 smithi163 ceph-mon[34354]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:28 smithi103 ceph-mon[34421]: pgmap v350: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-05-09T09:18:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.964+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1332216178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98103f20 msgr2=0x7f7a9810a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1332216178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98103f20 0x7f7a9810a7d0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f7a9000b0d0 tx=0x7f7a9000b3e0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1332216178 shutdown_connections 2024-05-09T09:18:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1332216178 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98103f20 0x7f7a9810a7d0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1332216178 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98102380 0x7f7a981027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1332216178 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a981039e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1332216178 >> 172.21.15.69:0/1332216178 conn(0x7f7a980fd920 msgr2=0x7f7a980ffd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:28.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1332216178 shutdown_connections 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1332216178 wait complete. 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 Processor -- start 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.965+0000 7f7a9c8c8700 1 -- start start 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a98198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98103f20 0x7f7a9819fb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:28.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810dfc0 con 0x7f7a98102380 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810de40 con 0x7f7a98103f20 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a9810e140 con 0x7f7a98103580 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60858/0 (socket says 172.21.15.69:60858) 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 -- 172.21.15.69:0/1788104898 learned_addr learned my addr 172.21.15.69:0/1788104898 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a95d9b700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a98198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a96d9d700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98103f20 0x7f7a9819fb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 -- 172.21.15.69:0/1788104898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 msgr2=0x7f7a98198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:28.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a98198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 -- 172.21.15.69:0/1788104898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98103f20 msgr2=0x7f7a9819fb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98103f20 0x7f7a9819fb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 -- 172.21.15.69:0/1788104898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a981a01e0 con 0x7f7a98102380 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a95d9b700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a98198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9659c700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c9a0 tx=0x7f7a8800cd60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a88004220 con 0x7f7a98102380 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a88007d40 con 0x7f7a98102380 2024-05-09T09:18:28.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a981a04d0 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.966+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a8800f450 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.967+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a981a0ae0 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.968+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7a98069010 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.969+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7a88007860 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.969+0000 7f7a877fe700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 0x7f7a800a5160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.970+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(91..91 src has 1..91) ==== 9071+0+0 (secure 0 0 0) 0x7f7a880cf270 con 0x7f7a98102380 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.970+0000 7f7a95d9b700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 0x7f7a800a5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.970+0000 7f7a95d9b700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 0x7f7a800a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0099d0 tx=0x7f7a8c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:28.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:28.974+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7a88014030 con 0x7f7a98102380 2024-05-09T09:18:29.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.181+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) -- 0x7f7a98062760 con 0x7f7a800a2ca0 2024-05-09T09:18:29.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.183+0000 7f7a877fe700 1 -- 172.21.15.69:0/1788104898 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 ) ==== 8+0+532 (secure 0 0 0) 0x7f7a98062760 con 0x7f7a800a2ca0 2024-05-09T09:18:29.181 INFO:teuthology.orchestra.run.smithi069.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-05-09T09:18:29.182 INFO:teuthology.orchestra.run.smithi069.stdout:smithi069 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400Y0400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T09:18:29.182 INFO:teuthology.orchestra.run.smithi069.stdout:smithi103 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT6204012Z400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-05-09T09:18:29.182 INFO:teuthology.orchestra.run.smithi069.stdout:smithi163 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT623300GU400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-05-09T09:18:29.184 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 msgr2=0x7f7a800a5160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 0x7f7a800a5160 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f7a8c0099d0 tx=0x7f7a8c008040 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 msgr2=0x7f7a98198580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f7a8800c9a0 tx=0x7f7a8800cd60 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 shutdown_connections 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a800a2ca0 0x7f7a800a5160 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a98103f20 0x7f7a9819fb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a98102380 0x7f7a98198580 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 --2- 172.21.15.69:0/1788104898 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a98103580 0x7f7a98198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 >> 172.21.15.69:0/1788104898 conn(0x7f7a980fd920 msgr2=0x7f7a98106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 shutdown_connections 2024-05-09T09:18:29.185 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:29.185+0000 7f7a9c8c8700 1 -- 172.21.15.69:0/1788104898 wait complete. 2024-05-09T09:18:29.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-05-09T09:18:29.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-05-09T09:18:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-05-09T09:18:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-05-09T09:18:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]: dispatch 2024-05-09T09:18:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]: dispatch 2024-05-09T09:18:29.791 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-05-09T09:18:29.944 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:18:29.944 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph fs dump' 2024-05-09T09:18:30.189 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:30 smithi069 ceph-mon[30123]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:30 smithi069 ceph-mon[30123]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-05-09T09:18:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-05-09T09:18:30.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:30 smithi069 ceph-mon[30123]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T09:18:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:30 smithi163 ceph-mon[34354]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:30 smithi163 ceph-mon[34354]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-05-09T09:18:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-05-09T09:18:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:30 smithi163 ceph-mon[34354]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T09:18:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:30 smithi103 ceph-mon[34421]: pgmap v351: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:30 smithi103 ceph-mon[34421]: from='client.15291 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-05-09T09:18:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.e", "id": [5, 8]}]': finished 2024-05-09T09:18:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.1c", "id": [9, 10]}]': finished 2024-05-09T09:18:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:30 smithi103 ceph-mon[34421]: osdmap e92: 12 total, 12 up, 12 in 2024-05-09T09:18:31.480 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:31 smithi069 ceph-mon[30123]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T09:18:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:31 smithi163 ceph-mon[34354]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T09:18:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:31 smithi103 ceph-mon[34421]: osdmap e93: 12 total, 12 up, 12 in 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.845+0000 7fb18b492700 1 -- 172.21.15.69:0/2532152538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184100670 msgr2=0x7fb184100a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.845+0000 7fb18b492700 1 --2- 172.21.15.69:0/2532152538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184100670 0x7fb184100a90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fb180009990 tx=0x7fb180009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 -- 172.21.15.69:0/2532152538 shutdown_connections 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 --2- 172.21.15.69:0/2532152538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 0x7fb18410a8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 --2- 172.21.15.69:0/2532152538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184100670 0x7fb184100a90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 --2- 172.21.15.69:0/2532152538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184103650 0x7fb184103ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 -- 172.21.15.69:0/2532152538 >> 172.21.15.69:0/2532152538 conn(0x7fb1840fbfd0 msgr2=0x7fb1840fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:31.845 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 -- 172.21.15.69:0/2532152538 shutdown_connections 2024-05-09T09:18:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 -- 172.21.15.69:0/2532152538 wait complete. 2024-05-09T09:18:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 Processor -- start 2024-05-09T09:18:31.846 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.846+0000 7fb18b492700 1 -- start start 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 0x7fb1841984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 0x7fb18419fa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410df80 con 0x7fb184103650 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410de00 con 0x7fb184100670 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb18410e100 con 0x7fb184103ff0 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60876/0 (socket says 172.21.15.69:60876) 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 -- 172.21.15.69:0/2266980709 learned_addr learned my addr 172.21.15.69:0/2266980709 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb189a2f700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 0x7fb18419fa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 -- 172.21.15.69:0/2266980709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 msgr2=0x7fb18419fa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 0x7fb18419fa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18922e700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 0x7fb1841984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 -- 172.21.15.69:0/2266980709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 msgr2=0x7fb1841984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 0x7fb1841984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 -- 172.21.15.69:0/2266980709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb1841a0100 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18922e700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 0x7fb1841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb188a2d700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c9e0 tx=0x7fb17800cda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb178007ac0 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb1841a03f0 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb178007c60 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb1780186f0 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.847+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb1841a0980 con 0x7fb184103650 2024-05-09T09:18:31.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.848+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb184069010 con 0x7fb184103650 2024-05-09T09:18:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.850+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb1780188b0 con 0x7fb184103650 2024-05-09T09:18:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.850+0000 7fb1767fc700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.851+0000 7fb18922e700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.851+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fb1780cf370 con 0x7fb184103650 2024-05-09T09:18:31.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.851+0000 7fb18922e700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 0x7fb1700a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb1800039c0 tx=0x7fb180005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:31.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:31.855+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb178014030 con 0x7fb184103650 2024-05-09T09:18:32.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.097+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump"} v 0) -- 0x7fb18402db80 con 0x7fb184103650 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.098+0000 7fb1767fc700 1 -- 172.21.15.69:0/2266980709 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 24 v24) ==== 76+0+3056 (secure 0 0 0) 0x7fb17809bff0 con 0x7fb184103650 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:e24 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:legacy client fscid: 1 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:Filesystem 'cephfs' (1) 2024-05-09T09:18:32.097 INFO:teuthology.orchestra.run.smithi069.stdout:fs_name cephfs 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 24 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-05-09T09:16:32.211035+0000 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:modified 2024-05-09T09:16:53.206774+0000 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:tableserver 0 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:root 0 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:session_timeout 300 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:session_autoclose 300 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:max_file_size 1099511627776 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:max_xattr_size 65536 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:required_client_features {} 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:last_failure 0 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:last_failure_osd_epoch 0 2024-05-09T09:18:32.098 INFO:teuthology.orchestra.run.smithi069.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments,12=quiesce subvolumes} 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:max_mds 3 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:in 0,1,2 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:up {0=24527,1=24511,2=24491} 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:failed 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:damaged 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:stopped 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:data_pools [3,4] 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:metadata_pool 2 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:inline_data disabled 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:balancer 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:bal_rank_mask -1 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:standby_count_wanted 0 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:qdb_cluster leader: 24527 members: 24491,24511,24527 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.b{0:24527} state up:active seq 3 addr [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.099 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.c{0:24490} state up:standby-replay seq 1 addr [v2:172.21.15.163:6836/3224689113,v1:172.21.15.163:6837/3224689113] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.i{1:24511} state up:active seq 10 addr [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.f{1:24478} state up:standby-replay seq 1 addr [v2:172.21.15.163:6834/989258568,v1:172.21.15.163:6835/989258568] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.h{2:24491} state up:active seq 11 addr [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.e{2:24458} state up:standby-replay seq 1 addr [v2:172.21.15.103:6834/3790151668,v1:172.21.15.103:6835/3790151668] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:Standby daemons: 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.j{-1:14541} state up:standby seq 1 addr [v2:172.21.15.69:6834/3886932870,v1:172.21.15.69:6835/3886932870] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.d{-1:14553} state up:standby seq 1 addr [v2:172.21.15.69:6836/4268769273,v1:172.21.15.69:6837/4268769273] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.g{-1:14616} state up:standby seq 1 addr [v2:172.21.15.69:6838/1794172552,v1:172.21.15.69:6839/1794172552] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.a{-1:14628} state up:standby seq 1 addr [v2:172.21.15.69:6840/4006889644,v1:172.21.15.69:6841/4006889644] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.l{-1:24430} state up:standby seq 1 addr [v2:172.21.15.163:6832/3615156387,v1:172.21.15.163:6833/3615156387] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.100 INFO:teuthology.orchestra.run.smithi069.stdout:[mds.k{-1:24431} state up:standby seq 1 addr [v2:172.21.15.103:6832/1248493104,v1:172.21.15.103:6833/1248493104] compat {c=[1],r=[1],i=[1fff]}] 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 msgr2=0x7fb1700a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 0x7fb1700a50e0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb1800039c0 tx=0x7fb180005fb0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 msgr2=0x7fb1841989e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fb17800c9e0 tx=0x7fb17800cda0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 shutdown_connections 2024-05-09T09:18:32.101 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb184103ff0 0x7fb18419fa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb1700a2c20 0x7fb1700a50e0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb184100670 0x7fb1841984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 --2- 172.21.15.69:0/2266980709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb184103650 0x7fb1841989e0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 >> 172.21.15.69:0/2266980709 conn(0x7fb1840fbfd0 msgr2=0x7fb184106880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 shutdown_connections 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:32.100+0000 7fb18b492700 1 -- 172.21.15.69:0/2266980709 wait complete. 2024-05-09T09:18:32.102 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:32.483 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:32 smithi069 ceph-mon[30123]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:32.483 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:32 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2266980709' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T09:18:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:32 smithi163 ceph-mon[34354]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:32 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2266980709' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T09:18:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:32 smithi103 ceph-mon[34421]: pgmap v354: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:32 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2266980709' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-05-09T09:18:32.691 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- bash -c 'ceph osd dump' 2024-05-09T09:18:32.945 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:34.443 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:34 smithi069 ceph-mon[30123]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:34 smithi163 ceph-mon[34354]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 -- 172.21.15.69:0/3276686343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 msgr2=0x7f01a4103db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 --2- 172.21.15.69:0/3276686343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a4103db0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0198009990 tx=0x7f0198009ca0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 -- 172.21.15.69:0/3276686343 shutdown_connections 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 --2- 172.21.15.69:0/3276686343 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 0x7f01a410a980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 --2- 172.21.15.69:0/3276686343 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a4103db0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.575 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 --2- 172.21.15.69:0/3276686343 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4100870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.546+0000 7f01ab166700 1 -- 172.21.15.69:0/3276686343 >> 172.21.15.69:0/3276686343 conn(0x7f01a40fbfd0 msgr2=0x7f01a40fe410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- 172.21.15.69:0/3276686343 shutdown_connections 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- 172.21.15.69:0/3276686343 wait complete. 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 Processor -- start 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- start start 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 0x7f01a4198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01a410dff0 con 0x7f01a4103990 2024-05-09T09:18:34.576 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01a410de70 con 0x7f01a4100410 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.547+0000 7f01ab166700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01a410e170 con 0x7f01a4100db0 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60908/0 (socket says 172.21.15.69:60908) 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 -- 172.21.15.69:0/648578185 learned_addr learned my addr 172.21.15.69:0/648578185 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a3fff700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 0x7f01a4198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a8f02700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 -- 172.21.15.69:0/648578185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 msgr2=0x7f01a4198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 0x7f01a4198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 -- 172.21.15.69:0/648578185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 msgr2=0x7f01a4198620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:34.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4198620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 -- 172.21.15.69:0/648578185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01a41a0280 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a8f02700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a9703700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f019c00b9a0 tx=0x7f019c00bcb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c019070 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f019c00ce10 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01a41a0570 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.548+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f019c0073d0 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.549+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01a41a0ae0 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.550+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f01a4069010 con 0x7f01a4103990 2024-05-09T09:18:34.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.551+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f019c01c070 con 0x7f01a4103990 2024-05-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.551+0000 7f01a1ffb700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 0x7f01940a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.551+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f019c0ce770 con 0x7f01a4103990 2024-05-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.551+0000 7f01a8f02700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 0x7f01940a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.552+0000 7f01a8f02700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 0x7f01940a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0198003780 tx=0x7f0198005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:34.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.556+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f019c09b350 con 0x7f01a4103990 2024-05-09T09:18:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:34 smithi103 ceph-mon[34421]: pgmap v355: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:34.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.752+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump"} v 0) -- 0x7f01a4062f30 con 0x7f01a4103990 2024-05-09T09:18:34.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.753+0000 7f01a1ffb700 1 -- 172.21.15.69:0/648578185 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) ==== 56+0+5628 (secure 0 0 0) 0x7f019c09ac50 con 0x7f01a4103990 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:epoch 93 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:created 2024-05-09T09:08:27.932250+0000 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:modified 2024-05-09T09:18:30.220267+0000 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:crush_version 27 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:full_ratio 0.95 2024-05-09T09:18:34.752 INFO:teuthology.orchestra.run.smithi069.stdout:backfillfull_ratio 0.9 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:nearfull_ratio 0.85 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:require_min_compat_client luminous 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:min_compat_client luminous 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:require_osd_release squid 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:stretch_mode_enabled false 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-05-09T09:18:34.753 INFO:teuthology.orchestra.run.smithi069.stdout:max_osd 12 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6802/1965346268,v1:172.21.15.69:6803/1965346268] [v2:172.21.15.69:6804/1965346268,v1:172.21.15.69:6805/1965346268] exists,up 54339b38-12b5-46a7-a293-1cea9feb97e6 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6800/4055201011,v1:172.21.15.103:6801/4055201011] [v2:172.21.15.103:6802/4055201011,v1:172.21.15.103:6803/4055201011] exists,up 9edd74d4-c926-46b9-b564-36ea82da5321 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.2 up in weight 1 up_from 18 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6800/250759774,v1:172.21.15.163:6801/250759774] [v2:172.21.15.163:6802/250759774,v1:172.21.15.163:6803/250759774] exists,up b8268e98-1c34-44a4-b53f-779dd3ea68ab 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6810/3308296525,v1:172.21.15.69:6811/3308296525] [v2:172.21.15.69:6812/3308296525,v1:172.21.15.69:6813/3308296525] exists,up ebea5b0f-0931-447f-848c-342af0a730a7 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.4 up in weight 1 up_from 33 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6808/613528366,v1:172.21.15.103:6809/613528366] [v2:172.21.15.103:6810/613528366,v1:172.21.15.103:6811/613528366] exists,up 7cf7c161-cb83-47cc-be48-1a7bcba78412 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6808/3212961219,v1:172.21.15.163:6809/3212961219] [v2:172.21.15.163:6810/3212961219,v1:172.21.15.163:6811/3212961219] exists,up 3e0e1bf3-cc14-4202-a96a-a557f5041ac0 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.6 up in weight 1 up_from 46 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6818/3160153338,v1:172.21.15.69:6819/3160153338] [v2:172.21.15.69:6820/3160153338,v1:172.21.15.69:6821/3160153338] exists,up ef0f7d90-b664-4519-a34a-27549cfe3406 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.7 up in weight 1 up_from 51 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6816/4108396337,v1:172.21.15.103:6817/4108396337] [v2:172.21.15.103:6818/4108396337,v1:172.21.15.103:6819/4108396337] exists,up 7236537c-3738-4ad0-bc46-1a96a7bd8b70 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6816/2914036945,v1:172.21.15.163:6817/2914036945] [v2:172.21.15.163:6818/2914036945,v1:172.21.15.163:6819/2914036945] exists,up e7cfa791-797c-4dab-84a1-d2ff938e96c4 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.69:6826/2042751079,v1:172.21.15.69:6827/2042751079] [v2:172.21.15.69:6828/2042751079,v1:172.21.15.69:6829/2042751079] exists,up 55ae1e0d-2d12-4aef-9449-e498e1cc719f 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.103:6824/955568662,v1:172.21.15.103:6825/955568662] [v2:172.21.15.103:6826/955568662,v1:172.21.15.103:6827/955568662] exists,up dbc63c27-da12-4420-b03a-0c4d5721bd9c 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:osd.11 up in weight 1 up_from 80 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.163:6824/208079664,v1:172.21.15.163:6825/208079664] [v2:172.21.15.163:6826/208079664,v1:172.21.15.163:6827/208079664] exists,up 9e951639-2f9f-4524-9e05-c9b222ac9a05 2024-05-09T09:18:34.754 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 2.5 [8,11] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 2.1b [2,4] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 2.1d [8,11,2,1] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 3.0 [5,9] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 3.3 [4,9] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 3.34 [4,9] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 3.3a [4,8] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.e [5,8] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.17 [9,8] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.1c [9,10] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:pg_upmap_items 4.1f [9,8] 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/4182744829 expires 2024-05-10T09:09:29.020087+0000 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6801/936285351 expires 2024-05-10T09:09:29.020087+0000 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1673751021 expires 2024-05-10T09:09:29.020087+0000 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1911852009 expires 2024-05-10T09:09:02.961220+0000 2024-05-09T09:18:34.755 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/1056628795 expires 2024-05-10T09:09:02.961220+0000 2024-05-09T09:18:34.756 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6800/936285351 expires 2024-05-10T09:09:29.020087+0000 2024-05-09T09:18:34.756 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6800/2869747387 expires 2024-05-10T09:09:02.961220+0000 2024-05-09T09:18:34.756 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/2520675955 expires 2024-05-10T09:09:02.961220+0000 2024-05-09T09:18:34.756 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:6801/2869747387 expires 2024-05-10T09:09:02.961220+0000 2024-05-09T09:18:34.756 INFO:teuthology.orchestra.run.smithi069.stdout:blocklist 172.21.15.69:0/3187678781 expires 2024-05-10T09:09:29.020087+0000 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 msgr2=0x7f01940a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 0x7f01940a50e0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0198003780 tx=0x7f0198005fb0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 msgr2=0x7f01a419fbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f019c00b9a0 tx=0x7f019c00bcb0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 shutdown_connections 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01a4103990 0x7f01a419fbe0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01940a2c20 0x7f01940a50e0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01a4100410 0x7f01a4198620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 --2- 172.21.15.69:0/648578185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01a4100db0 0x7f01a4198b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 >> 172.21.15.69:0/648578185 conn(0x7f01a40fbfd0 msgr2=0x7f01a41069d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 shutdown_connections 2024-05-09T09:18:34.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:34.755+0000 7f01ab166700 1 -- 172.21.15.69:0/648578185 wait complete. 2024-05-09T09:18:35.341 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:35 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/648578185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T09:18:35.342 INFO:teuthology.run_tasks:Running task fs.ready... 2024-05-09T09:18:35.353 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:35.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.461+0000 7f6602309640 1 -- 172.21.15.69:0/524788377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc101d20 msgr2=0x7f65fc102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.461+0000 7f6602309640 1 --2- 172.21.15.69:0/524788377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc101d20 0x7f65fc102140 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f65ec009900 tx=0x7f65ec02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:35.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- 172.21.15.69:0/524788377 shutdown_connections 2024-05-09T09:18:35.463 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- 172.21.15.69:0/524788377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101410 0x7f65fc108cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- 172.21.15.69:0/524788377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc100a50 0x7f65fc100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- 172.21.15.69:0/524788377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc101d20 0x7f65fc102140 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- 172.21.15.69:0/524788377 >> 172.21.15.69:0/524788377 conn(0x7f65fc0fb4a0 msgr2=0x7f65fc0fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- 172.21.15.69:0/524788377 shutdown_connections 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- 172.21.15.69:0/524788377 wait complete. 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 Processor -- start 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- start start 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc101410 0x7f65fc198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 0x7f65fc19f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc10c410 con 0x7f65fc100a50 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc10c290 con 0x7f65fc101d20 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f6602309640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f65fc10c590 con 0x7f65fc101410 2024-05-09T09:18:35.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f65fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60924/0 (socket says 172.21.15.69:60924) 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.462+0000 7f65fbfff640 1 -- 172.21.15.69:0/1856337223 learned_addr learned my addr 172.21.15.69:0/1856337223 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f66008a6640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 0x7f65fc19f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 -- 172.21.15.69:0/1856337223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc101410 msgr2=0x7f65fc198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65f3fff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc101410 0x7f65fc198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc101410 0x7f65fc198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 -- 172.21.15.69:0/1856337223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 msgr2=0x7f65fc19f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 0x7f65fc19f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 -- 172.21.15.69:0/1856337223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f65fc19f7e0 con 0x7f65fc100a50 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f66008a6640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 0x7f65fc19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:35.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65fbfff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f65ec005ab0 tx=0x7f65ec0042e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec03f030 con 0x7f65fc100a50 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f65ec0044d0 con 0x7f65fc100a50 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f65fc19fa70 con 0x7f65fc100a50 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f65ec035070 con 0x7f65fc100a50 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.463+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f65fc19ffb0 con 0x7f65fc100a50 2024-05-09T09:18:35.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.464+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f65fc1a01e0 con 0x7f65fc100a50 2024-05-09T09:18:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.465+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f65ec042030 con 0x7f65fc100a50 2024-05-09T09:18:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.466+0000 7f65f9ffb640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 0x7f65e80a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.466+0000 7f65f3fff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 0x7f65e80a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.466+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f65ec0ef920 con 0x7f65fc100a50 2024-05-09T09:18:35.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.466+0000 7f65f3fff640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 0x7f65e80a5150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65fc1990d0 tx=0x7f65e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:35.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.470+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f65ec0bc4a0 con 0x7f65fc100a50 2024-05-09T09:18:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:35 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/648578185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T09:18:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:35 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/648578185' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-05-09T09:18:35.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.647+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f65fc1a04d0 con 0x7f65fc100a50 2024-05-09T09:18:35.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.648+0000 7f65f9ffb640 1 -- 172.21.15.69:0/1856337223 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7f65ec0bbda0 con 0x7f65fc100a50 2024-05-09T09:18:35.647 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:35.648 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:35.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.650+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 msgr2=0x7f65e80a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.650+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 0x7f65e80a5150 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f65fc1990d0 tx=0x7f65e4005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.650+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 msgr2=0x7f65fc197bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.650+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f65ec005ab0 tx=0x7f65ec0042e0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 shutdown_connections 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f65e80a2c90 0x7f65e80a5150 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f65fc101d20 0x7f65fc19f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f65fc101410 0x7f65fc198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 --2- 172.21.15.69:0/1856337223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f65fc100a50 0x7f65fc197bc0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 >> 172.21.15.69:0/1856337223 conn(0x7f65fc0fb4a0 msgr2=0x7f65fc0ff940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 shutdown_connections 2024-05-09T09:18:35.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.651+0000 7f6602309640 1 -- 172.21.15.69:0/1856337223 wait complete. 2024-05-09T09:18:35.661 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:35.662 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- 172.21.15.69:0/2786719578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380ffeb0 msgr2=0x7f7c381002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2786719578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380ffeb0 0x7f7c381002d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f7c34009900 tx=0x7f7c3402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- 172.21.15.69:0/2786719578 shutdown_connections 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2786719578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c38101a50 0x7f7c38108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2786719578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c38101090 0x7f7c38101510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2786719578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c380ffeb0 0x7f7c381002d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- 172.21.15.69:0/2786719578 >> 172.21.15.69:0/2786719578 conn(0x7f7c380fb470 msgr2=0x7f7c380fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- 172.21.15.69:0/2786719578 shutdown_connections 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- 172.21.15.69:0/2786719578 wait complete. 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 Processor -- start 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.768+0000 7f7c402d4640 1 -- start start 2024-05-09T09:18:35.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c380ffeb0 0x7f7c38197ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c38101a50 0x7f7c3819f240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3810bc20 con 0x7f7c38101090 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3810baa0 con 0x7f7c38101a50 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c3810bda0 con 0x7f7c380ffeb0 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60944/0 (socket says 172.21.15.69:60944) 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 -- 172.21.15.69:0/2194636939 learned_addr learned my addr 172.21.15.69:0/2194636939 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 -- 172.21.15.69:0/2194636939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c380ffeb0 msgr2=0x7f7c38197ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3e070640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c380ffeb0 0x7f7c38197ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c380ffeb0 0x7f7c38197ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 -- 172.21.15.69:0/2194636939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c38101a50 msgr2=0x7f7c3819f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c38101a50 0x7f7c3819f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 -- 172.21.15.69:0/2194636939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c3819f8e0 con 0x7f7c38101090 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c3d86f640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c970 tx=0x7f7c2800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c28007bb0 con 0x7f7c38101090 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c3819fbd0 con 0x7f7c38101090 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c28007d50 con 0x7f7c38101090 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c2800f450 con 0x7f7c38101090 2024-05-09T09:18:35.771 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.769+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c381a0150 con 0x7f7c38101090 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.770+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f7c381a0380 con 0x7f7c38101090 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.772+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7c28004fe0 con 0x7f7c38101090 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.772+0000 7f7c2f7fe640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 0x7f7c240a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.772+0000 7f7c3e070640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 0x7f7c240a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.772+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f7c280cf5f0 con 0x7f7c38101090 2024-05-09T09:18:35.772 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.773+0000 7f7c3e070640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 0x7f7c240a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006010 tx=0x7f7c34000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:35.775 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.776+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f7c28014030 con 0x7f7c38101090 2024-05-09T09:18:35.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.950+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f7c381a0550 con 0x7f7c38101090 2024-05-09T09:18:35.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.950+0000 7f7c2f7fe640 1 -- 172.21.15.69:0/2194636939 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f7c2809c1d0 con 0x7f7c38101090 2024-05-09T09:18:35.949 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:35.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 msgr2=0x7f7c240a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 0x7f7c240a5050 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f7c34006010 tx=0x7f7c34000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 msgr2=0x7f7c381981e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f7c2800c970 tx=0x7f7c2800ce30 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 shutdown_connections 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c240a2b90 0x7f7c240a5050 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c38101a50 0x7f7c3819f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c38101090 0x7f7c381981e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 --2- 172.21.15.69:0/2194636939 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c380ffeb0 0x7f7c38197ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.952+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 >> 172.21.15.69:0/2194636939 conn(0x7f7c380fb470 msgr2=0x7f7c380fcf80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.953+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 shutdown_connections 2024-05-09T09:18:35.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:35.953+0000 7f7c402d4640 1 -- 172.21.15.69:0/2194636939 wait complete. 2024-05-09T09:18:35.964 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 -- 172.21.15.69:0/1541052841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac601075d0 msgr2=0x7fac601099c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 --2- 172.21.15.69:0/1541052841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac601075d0 0x7fac601099c0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fac58009ca0 tx=0x7fac5802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 -- 172.21.15.69:0/1541052841 shutdown_connections 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 --2- 172.21.15.69:0/1541052841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac601075d0 0x7fac601099c0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 --2- 172.21.15.69:0/1541052841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac60104ca0 0x7fac60107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 --2- 172.21.15.69:0/1541052841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60102370 0x7fac60104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 -- 172.21.15.69:0/1541052841 >> 172.21.15.69:0/1541052841 conn(0x7fac600fbd20 msgr2=0x7fac600fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 -- 172.21.15.69:0/1541052841 shutdown_connections 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.070+0000 7fac667a2640 1 -- 172.21.15.69:0/1541052841 wait complete. 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 Processor -- start 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 -- start start 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 0x7fac601980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac601075d0 0x7fac6019f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fac6010d530 con 0x7fac60102370 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fac6010d3b0 con 0x7fac60104ca0 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac667a2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fac6010d6b0 con 0x7fac601075d0 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60970/0 (socket says 172.21.15.69:60970) 2024-05-09T09:18:36.072 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 -- 172.21.15.69:0/3528161903 learned_addr learned my addr 172.21.15.69:0/3528161903 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac64d3f640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac601075d0 0x7fac6019f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac57fff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 0x7fac601980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 -- 172.21.15.69:0/3528161903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac601075d0 msgr2=0x7fac6019f130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac601075d0 0x7fac6019f130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 -- 172.21.15.69:0/3528161903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 msgr2=0x7fac601980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 0x7fac601980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 -- 172.21.15.69:0/3528161903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fac6019f7a0 con 0x7fac60102370 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac57fff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 0x7fac601980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5ffff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac5000d950 tx=0x7fac5000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.071+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac50017070 con 0x7fac60102370 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.072+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fac6019fa90 con 0x7fac60102370 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.072+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fac50009c30 con 0x7fac60102370 2024-05-09T09:18:36.073 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.072+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fac5000f6b0 con 0x7fac60102370 2024-05-09T09:18:36.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.072+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fac6019ffe0 con 0x7fac60102370 2024-05-09T09:18:36.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.073+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fac600fd810 con 0x7fac60102370 2024-05-09T09:18:36.074 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.074+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fac500109d0 con 0x7fac60102370 2024-05-09T09:18:36.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.074+0000 7fac5dffb640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 0x7fac4c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.074+0000 7fac57fff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 0x7fac4c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.074+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fac500cf0e0 con 0x7fac60102370 2024-05-09T09:18:36.075 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.075+0000 7fac57fff640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 0x7fac4c0a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fac60199090 tx=0x7fac48005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.077 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.079+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fac50009de0 con 0x7fac60102370 2024-05-09T09:18:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.255+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fac60062430 con 0x7fac60102370 2024-05-09T09:18:36.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.255+0000 7fac5dffb640 1 -- 172.21.15.69:0/3528161903 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fac5009bcc0 con 0x7fac60102370 2024-05-09T09:18:36.254 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 msgr2=0x7fac4c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 0x7fac4c0a5050 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fac60199090 tx=0x7fac48005e50 comp rx=0 tx=0).stop 2024-05-09T09:18:36.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 msgr2=0x7fac60197b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fac5000d950 tx=0x7fac5000de10 comp rx=0 tx=0).stop 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 shutdown_connections 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fac4c0a2b90 0x7fac4c0a5050 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fac601075d0 0x7fac6019f130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fac60104ca0 0x7fac601980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 --2- 172.21.15.69:0/3528161903 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fac60102370 0x7fac60197b80 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 >> 172.21.15.69:0/3528161903 conn(0x7fac600fbd20 msgr2=0x7fac60102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 shutdown_connections 2024-05-09T09:18:36.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.257+0000 7fac667a2640 1 -- 172.21.15.69:0/3528161903 wait complete. 2024-05-09T09:18:36.268 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:36.374 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 -- 172.21.15.69:0/4034880688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0101870 msgr2=0x7f2cf0101c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 --2- 172.21.15.69:0/4034880688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0101870 0x7f2cf0101c90 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009900 tx=0x7f2ce002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 -- 172.21.15.69:0/4034880688 shutdown_connections 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 --2- 172.21.15.69:0/4034880688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf0103420 0x7f2cf0109cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 --2- 172.21.15.69:0/4034880688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0102a60 0x7f2cf0102ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 --2- 172.21.15.69:0/4034880688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0101870 0x7f2cf0101c90 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 -- 172.21.15.69:0/4034880688 >> 172.21.15.69:0/4034880688 conn(0x7f2cf00fcde0 msgr2=0x7f2cf00ff220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 -- 172.21.15.69:0/4034880688 shutdown_connections 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.373+0000 7f2cf52da640 1 -- 172.21.15.69:0/4034880688 wait complete. 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 Processor -- start 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 -- start start 2024-05-09T09:18:36.375 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 0x7f2cf0199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf0103420 0x7f2cf01a1380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf010d4f0 con 0x7f2cf0102a60 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf010d370 con 0x7f2cf0101870 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cf52da640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cf010d670 con 0x7f2cf0103420 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60990/0 (socket says 172.21.15.69:60990) 2024-05-09T09:18:36.376 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 -- 172.21.15.69:0/2641221837 learned_addr learned my addr 172.21.15.69:0/2641221837 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ceeffd640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 0x7f2cf0199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 -- 172.21.15.69:0/2641221837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf0103420 msgr2=0x7f2cf01a1380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf0103420 0x7f2cf01a1380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 -- 172.21.15.69:0/2641221837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 msgr2=0x7f2cf0199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 0x7f2cf0199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 -- 172.21.15.69:0/2641221837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cf01a19f0 con 0x7f2cf0102a60 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ceeffd640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 0x7f2cf0199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2ce7fff640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800c890 tx=0x7f2cd800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.374+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8004220 con 0x7f2cf0102a60 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.375+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd80043c0 con 0x7f2cf0102a60 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.375+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd80049e0 con 0x7f2cf0102a60 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.375+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cf01a1ce0 con 0x7f2cf0102a60 2024-05-09T09:18:36.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.375+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cf01a2220 con 0x7f2cf0102a60 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.376+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2cf01a24b0 con 0x7f2cf0102a60 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.377+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2cd80079b0 con 0x7f2cf0102a60 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.378+0000 7f2cecff9640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 0x7f2cdc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.378+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2cd80cf430 con 0x7f2cf0102a60 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.378+0000 7f2ceeffd640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 0x7f2cdc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.378+0000 7f2ceeffd640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 0x7f2cdc0a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006010 tx=0x7f2ce0032dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.382+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2cd8014030 con 0x7f2cf0102a60 2024-05-09T09:18:36.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:36 smithi069 ceph-mon[30123]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:36 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1856337223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:36 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2194636939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:36 smithi163 ceph-mon[34354]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:36 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1856337223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:36 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2194636939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:36.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.558+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2cf01a2680 con 0x7f2cf0102a60 2024-05-09T09:18:36.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.558+0000 7f2cecff9640 1 -- 172.21.15.69:0/2641221837 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f2cd809c090 con 0x7f2cf0102a60 2024-05-09T09:18:36.557 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 msgr2=0x7f2cdc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 0x7f2cdc0a50d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0006010 tx=0x7f2ce0032dd0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 msgr2=0x7f2cf019a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800c890 tx=0x7f2cd800cd50 comp rx=0 tx=0).stop 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 shutdown_connections 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cdc0a2c10 0x7f2cdc0a50d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cf0103420 0x7f2cf01a1380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cf0102a60 0x7f2cf019a310 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 --2- 172.21.15.69:0/2641221837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cf0101870 0x7f2cf0199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 >> 172.21.15.69:0/2641221837 conn(0x7f2cf00fcde0 msgr2=0x7f2cf0073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 shutdown_connections 2024-05-09T09:18:36.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.560+0000 7f2cf52da640 1 -- 172.21.15.69:0/2641221837 wait complete. 2024-05-09T09:18:36.570 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:36 smithi103 ceph-mon[34421]: pgmap v356: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-05-09T09:18:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:36 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1856337223' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:36 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2194636939' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 -- 172.21.15.69:0/1322431004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124102320 msgr2=0x7f0124108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 --2- 172.21.15.69:0/1322431004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124102320 0x7f0124108bb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f011c009900 tx=0x7f011c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 -- 172.21.15.69:0/1322431004 shutdown_connections 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 --2- 172.21.15.69:0/1322431004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124102320 0x7f0124108bb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 --2- 172.21.15.69:0/1322431004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124101960 0x7f0124101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 --2- 172.21.15.69:0/1322431004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124100770 0x7f0124100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.675+0000 7f012b493640 1 -- 172.21.15.69:0/1322431004 >> 172.21.15.69:0/1322431004 conn(0x7f01240fbce0 msgr2=0x7f01240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- 172.21.15.69:0/1322431004 shutdown_connections 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- 172.21.15.69:0/1322431004 wait complete. 2024-05-09T09:18:36.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 Processor -- start 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- start start 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124100770 0x7f0124196a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124102320 0x7f012419e020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f012410c3f0 con 0x7f0124101960 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f012410c270 con 0x7f0124102320 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012b493640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f012410c570 con 0x7f0124100770 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f0128a2e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f0128a2e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32788/0 (socket says 172.21.15.69:32788) 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f0128a2e640 1 -- 172.21.15.69:0/1125370428 learned_addr learned my addr 172.21.15.69:0/1125370428 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.676+0000 7f012922f640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124100770 0x7f0124196a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0129a30640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124102320 0x7f012419e020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 -- 172.21.15.69:0/1125370428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124100770 msgr2=0x7f0124196a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124100770 0x7f0124196a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 -- 172.21.15.69:0/1125370428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124102320 msgr2=0x7f012419e020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124102320 0x7f012419e020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 -- 172.21.15.69:0/1125370428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f012419e690 con 0x7f0124101960 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f0128a2e640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f011800d770 tx=0x7f011800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118017070 con 0x7f0124101960 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0118005500 con 0x7f0124101960 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f012419e980 con 0x7f0124101960 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0118004e00 con 0x7f0124101960 2024-05-09T09:18:36.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.677+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f012419ee80 con 0x7f0124101960 2024-05-09T09:18:36.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.678+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f0124107b60 con 0x7f0124101960 2024-05-09T09:18:36.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.679+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0118004fa0 con 0x7f0124101960 2024-05-09T09:18:36.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.680+0000 7f01227fc640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 0x7f010c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.680+0000 7f012922f640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 0x7f010c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.680+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f01180cee40 con 0x7f0124101960 2024-05-09T09:18:36.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.680+0000 7f012922f640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 0x7f010c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f01241017c0 tx=0x7f0114009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.685+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f011809ba20 con 0x7f0124101960 2024-05-09T09:18:36.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.861+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f012402d1c0 con 0x7f0124101960 2024-05-09T09:18:36.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.861+0000 7f01227fc640 1 -- 172.21.15.69:0/1125370428 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f011809b320 con 0x7f0124101960 2024-05-09T09:18:36.860 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 msgr2=0x7f010c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 0x7f010c0a5050 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f01241017c0 tx=0x7f0114009670 comp rx=0 tx=0).stop 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 msgr2=0x7f0124196fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f011800d770 tx=0x7f011800dc30 comp rx=0 tx=0).stop 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 shutdown_connections 2024-05-09T09:18:36.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f010c0a2b90 0x7f010c0a5050 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0124102320 0x7f012419e020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0124101960 0x7f0124196fb0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 --2- 172.21.15.69:0/1125370428 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0124100770 0x7f0124196a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 >> 172.21.15.69:0/1125370428 conn(0x7f01240fbce0 msgr2=0x7f0124104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 shutdown_connections 2024-05-09T09:18:36.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.863+0000 7f012b493640 1 -- 172.21.15.69:0/1125370428 wait complete. 2024-05-09T09:18:36.874 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:36.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2571874874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec103430 msgr2=0x7fabec109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2571874874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec103430 0x7fabec109cc0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009ca0 tx=0x7fabe402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2571874874 shutdown_connections 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2571874874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec103430 0x7fabec109cc0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2571874874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec102a70 0x7fabec102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2571874874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec101880 0x7fabec101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2571874874 >> 172.21.15.69:0/2571874874 conn(0x7fabec0fce10 msgr2=0x7fabec0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:36.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2571874874 shutdown_connections 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2571874874 wait complete. 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 Processor -- start 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- start start 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec101880 0x7fabec197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec103430 0x7fabec19f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10d500 con 0x7fabec102a70 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10d380 con 0x7fabec103430 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.979+0000 7fabf2bb7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fabec10d680 con 0x7fabec101880 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32812/0 (socket says 172.21.15.69:32812) 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 -- 172.21.15.69:0/2701592985 learned_addr learned my addr 172.21.15.69:0/2701592985 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabf1154640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec103430 0x7fabec19f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabf0953640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec101880 0x7fabec197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 -- 172.21.15.69:0/2701592985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec101880 msgr2=0x7fabec197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec101880 0x7fabec197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 -- 172.21.15.69:0/2701592985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec103430 msgr2=0x7fabec19f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec103430 0x7fabec19f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 -- 172.21.15.69:0/2701592985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fabec19f790 con 0x7fabec102a70 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabebfff640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fabd800e9f0 tx=0x7fabd800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd800cd80 con 0x7fabec102a70 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fabd8005500 con 0x7fabec102a70 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fabec19fa80 con 0x7fabec102a70 2024-05-09T09:18:36.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fabd8010690 con 0x7fabec102a70 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.980+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fabec19ff80 con 0x7fabec102a70 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.981+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fabec108c70 con 0x7fabec102a70 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.982+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fabd80046c0 con 0x7fabec102a70 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.983+0000 7fabe9ffb640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 0x7fabdc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.983+0000 7fabf0953640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 0x7fabdc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.983+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fabd80cf850 con 0x7fabec102a70 2024-05-09T09:18:36.987 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.984+0000 7fabf0953640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 0x7fabdc0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fabe0005fd0 tx=0x7fabe0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:36.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:36.987+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fabd809c430 con 0x7fabec102a70 2024-05-09T09:18:37.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.200+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fabec062430 con 0x7fabec102a70 2024-05-09T09:18:37.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.201+0000 7fabe9ffb640 1 -- 172.21.15.69:0/2701592985 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7fabd809bd30 con 0x7fabec102a70 2024-05-09T09:18:37.200 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:37.201 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:18:37.201 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 msgr2=0x7fabdc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 0x7fabdc0a5050 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fabe0005fd0 tx=0x7fabe0005850 comp rx=0 tx=0).stop 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 msgr2=0x7fabec1980b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7fabd800e9f0 tx=0x7fabd800edf0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 shutdown_connections 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fabdc0a2b90 0x7fabdc0a5050 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fabec103430 0x7fabec19f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fabec102a70 0x7fabec1980b0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 --2- 172.21.15.69:0/2701592985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fabec101880 0x7fabec197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.203+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 >> 172.21.15.69:0/2701592985 conn(0x7fabec0fce10 msgr2=0x7fabec105e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.204+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 shutdown_connections 2024-05-09T09:18:37.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.204+0000 7fabf2bb7640 1 -- 172.21.15.69:0/2701592985 wait complete. 2024-05-09T09:18:37.215 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 -- 172.21.15.69:0/1135043157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0102330 msgr2=0x7f38b0108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 --2- 172.21.15.69:0/1135043157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0102330 0x7f38b0108bc0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f38a8009ca0 tx=0x7f38a802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 -- 172.21.15.69:0/1135043157 shutdown_connections 2024-05-09T09:18:37.324 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 --2- 172.21.15.69:0/1135043157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0102330 0x7f38b0108bc0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 --2- 172.21.15.69:0/1135043157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0101970 0x7f38b0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 --2- 172.21.15.69:0/1135043157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0100780 0x7f38b0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 -- 172.21.15.69:0/1135043157 >> 172.21.15.69:0/1135043157 conn(0x7f38b00fbd10 msgr2=0x7f38b00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 -- 172.21.15.69:0/1135043157 shutdown_connections 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.318+0000 7f38b662a640 1 -- 172.21.15.69:0/1135043157 wait complete. 2024-05-09T09:18:37.325 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 Processor -- start 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 -- start start 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 0x7f38b0197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0102330 0x7f38b019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d520 con 0x7f38b0101970 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d3a0 con 0x7f38b0102330 2024-05-09T09:18:37.326 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b662a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f38b010d6a0 con 0x7f38b0100780 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32820/0 (socket says 172.21.15.69:32820) 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 -- 172.21.15.69:0/3853014015 learned_addr learned my addr 172.21.15.69:0/3853014015 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38b4bc7640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0102330 0x7f38b019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38affff640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 0x7f38b0197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 -- 172.21.15.69:0/3853014015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 msgr2=0x7f38b0197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 0x7f38b0197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.327 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 -- 172.21.15.69:0/3853014015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0102330 msgr2=0x7f38b019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0102330 0x7f38b019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38af7fe640 1 -- 172.21.15.69:0/3853014015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38b019f7d0 con 0x7f38b0101970 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.319+0000 7f38affff640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 0x7f38b0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38af7fe640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f389800c970 tx=0x7f389800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3898007bb0 con 0x7f38b0101970 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3898007d50 con 0x7f38b0101970 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38b019fac0 con 0x7f38b0101970 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f389800f450 con 0x7f38b0101970 2024-05-09T09:18:37.328 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.320+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38b01a0010 con 0x7f38b0101970 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.321+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f38b0107a40 con 0x7f38b0101970 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.322+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3898005020 con 0x7f38b0101970 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.322+0000 7f38ad7fa640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 0x7f389c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.323+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f38980cf130 con 0x7f38b0101970 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.323+0000 7f38affff640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 0x7f389c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.323+0000 7f38affff640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 0x7f389c0a50d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f38b01017d0 tx=0x7f38a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.330 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.327+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3898014030 con 0x7f38b0101970 2024-05-09T09:18:37.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.503+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f38b0062430 con 0x7f38b0101970 2024-05-09T09:18:37.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.505+0000 7f38ad7fa640 1 -- 172.21.15.69:0/3853014015 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7f389809bd10 con 0x7f38b0101970 2024-05-09T09:18:37.504 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:37.504 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:37.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 msgr2=0x7f389c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 0x7f389c0a50d0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f38b01017d0 tx=0x7f38a0005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:37.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 msgr2=0x7f38b01980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7f389800c970 tx=0x7f389800ce30 comp rx=0 tx=0).stop 2024-05-09T09:18:37.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 shutdown_connections 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f389c0a2c10 0x7f389c0a50d0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38b0102330 0x7f38b019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38b0101970 0x7f38b01980f0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 --2- 172.21.15.69:0/3853014015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38b0100780 0x7f38b0197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 >> 172.21.15.69:0/3853014015 conn(0x7f38b00fbd10 msgr2=0x7f38b0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 shutdown_connections 2024-05-09T09:18:37.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.507+0000 7f38b662a640 1 -- 172.21.15.69:0/3853014015 wait complete. 2024-05-09T09:18:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3528161903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2641221837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1125370428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:37 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2701592985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:37.518 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:37.518 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3528161903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2641221837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1125370428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:37 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2701592985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3528161903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2641221837' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1125370428' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:37 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2701592985' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:37.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- 172.21.15.69:0/4122018961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c101090 msgr2=0x7faa4c101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- 172.21.15.69:0/4122018961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c101090 0x7faa4c101510 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7faa40009900 tx=0x7faa4002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- 172.21.15.69:0/4122018961 shutdown_connections 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- 172.21.15.69:0/4122018961 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101a50 0x7faa4c108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- 172.21.15.69:0/4122018961 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c101090 0x7faa4c101510 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- 172.21.15.69:0/4122018961 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- 172.21.15.69:0/4122018961 >> 172.21.15.69:0/4122018961 conn(0x7faa4c0fb470 msgr2=0x7faa4c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- 172.21.15.69:0/4122018961 shutdown_connections 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- 172.21.15.69:0/4122018961 wait complete. 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 Processor -- start 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- start start 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101090 0x7faa4c19c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c101a50 0x7faa4c19ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10c400 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10c280 con 0x7faa4c101a50 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa52aa5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa4c10c580 con 0x7faa4c101090 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.624+0000 7faa50841640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42980/0 (socket says 172.21.15.69:42980) 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 -- 172.21.15.69:0/1688697981 learned_addr learned my addr 172.21.15.69:0/1688697981 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa51042640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c101a50 0x7faa4c19ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa4bfff640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101090 0x7faa4c19c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 -- 172.21.15.69:0/1688697981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101090 msgr2=0x7faa4c19c8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101090 0x7faa4c19c8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 -- 172.21.15.69:0/1688697981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c101a50 msgr2=0x7faa4c19ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c101a50 0x7faa4c19ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 -- 172.21.15.69:0/1688697981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa4c19f820 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa50841640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d770 tx=0x7faa3c00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c017070 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa3c009c30 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa4c19fab0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa3c003dd0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.625+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa4c19ffd0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.626+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faa4c107250 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.627+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faa3c003fb0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.628+0000 7faa49ffb640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 0x7faa340a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.628+0000 7faa4bfff640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 0x7faa340a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.628+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7faa3c0cee00 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.628+0000 7faa4bfff640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 0x7faa340a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa40000c00 tx=0x7faa400315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.632+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faa3c09b9e0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.807+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7faa4c062430 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.808+0000 7faa49ffb640 1 -- 172.21.15.69:0/1688697981 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7faa3c009de0 con 0x7faa4c0ffeb0 2024-05-09T09:18:37.807 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 msgr2=0x7faa340a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 0x7faa340a5050 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7faa40000c00 tx=0x7faa400315d0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 msgr2=0x7faa4c1098a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7faa3c00d770 tx=0x7faa3c00dc30 comp rx=0 tx=0).stop 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 shutdown_connections 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faa340a2b90 0x7faa340a5050 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa4c101a50 0x7faa4c19ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa4c101090 0x7faa4c19c8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 --2- 172.21.15.69:0/1688697981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa4c0ffeb0 0x7faa4c1098a0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 >> 172.21.15.69:0/1688697981 conn(0x7faa4c0fb470 msgr2=0x7faa4c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 shutdown_connections 2024-05-09T09:18:37.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.810+0000 7faa52aa5640 1 -- 172.21.15.69:0/1688697981 wait complete. 2024-05-09T09:18:37.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- 172.21.15.69:0/3254171083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 msgr2=0x7fef80101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- 172.21.15.69:0/3254171083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80101de0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fef74009900 tx=0x7fef7402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- 172.21.15.69:0/3254171083 shutdown_connections 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- 172.21.15.69:0/3254171083 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef80108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- 172.21.15.69:0/3254171083 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80101de0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- 172.21.15.69:0/3254171083 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 0x7fef80100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- 172.21.15.69:0/3254171083 >> 172.21.15.69:0/3254171083 conn(0x7fef800fbce0 msgr2=0x7fef800fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:37.928 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- 172.21.15.69:0/3254171083 shutdown_connections 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- 172.21.15.69:0/3254171083 wait complete. 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 Processor -- start 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- start start 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 0x7fef80197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef8019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fef8010c3f0 con 0x7fef80101960 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fef8010c270 con 0x7fef80102320 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.926+0000 7fef8867d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fef8010c570 con 0x7fef80100770 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42998/0 (socket says 172.21.15.69:42998) 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 -- 172.21.15.69:0/245550328 learned_addr learned my addr 172.21.15.69:0/245550328 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef86c1a640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef8019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.929 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef86419640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 0x7fef80197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 -- 172.21.15.69:0/245550328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 msgr2=0x7fef80197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 0x7fef80197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 -- 172.21.15.69:0/245550328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 msgr2=0x7fef8019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef8019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 -- 172.21.15.69:0/245550328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fef8019f780 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef86c1a640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef8019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef85c18640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fef74009230 tx=0x7fef74004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef7403f030 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fef8019fa10 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fef74003ce0 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fef8019ff00 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.927+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fef74049060 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.928+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fef801a01f0 con 0x7fef80101960 2024-05-09T09:18:37.930 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.929+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fef74042030 con 0x7fef80101960 2024-05-09T09:18:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.930+0000 7fef7f7fe640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 0x7fef680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.930+0000 7fef86419640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 0x7fef680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.930+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fef740efe70 con 0x7fef80101960 2024-05-09T09:18:37.931 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.931+0000 7fef86419640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 0x7fef680a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef70005c40 tx=0x7fef70005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:37.932 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:37.934+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fef740bca50 con 0x7fef80101960 2024-05-09T09:18:38.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.109+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fef801a03c0 con 0x7fef80101960 2024-05-09T09:18:38.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.109+0000 7fef7f7fe640 1 -- 172.21.15.69:0/245550328 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fef740bc350 con 0x7fef80101960 2024-05-09T09:18:38.108 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 msgr2=0x7fef680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 0x7fef680a5050 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7fef70005c40 tx=0x7fef70005bd0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 msgr2=0x7fef80198110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fef74009230 tx=0x7fef74004360 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 shutdown_connections 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fef680a2b90 0x7fef680a5050 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fef80102320 0x7fef8019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fef80101960 0x7fef80198110 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 --2- 172.21.15.69:0/245550328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fef80100770 0x7fef80197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 >> 172.21.15.69:0/245550328 conn(0x7fef800fbce0 msgr2=0x7fef800fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.111+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 shutdown_connections 2024-05-09T09:18:38.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.112+0000 7fef8867d640 1 -- 172.21.15.69:0/245550328 wait complete. 2024-05-09T09:18:38.122 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:38.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.227+0000 7f9c96142640 1 -- 172.21.15.69:0/2141083760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90101960 msgr2=0x7f9c90101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.227+0000 7f9c96142640 1 --2- 172.21.15.69:0/2141083760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90101960 0x7f9c90101de0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f9c80009900 tx=0x7f9c8002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- 172.21.15.69:0/2141083760 shutdown_connections 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- 172.21.15.69:0/2141083760 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90102320 0x7f9c90108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- 172.21.15.69:0/2141083760 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90101960 0x7f9c90101de0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- 172.21.15.69:0/2141083760 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- 172.21.15.69:0/2141083760 >> 172.21.15.69:0/2141083760 conn(0x7f9c900fbce0 msgr2=0x7f9c900fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- 172.21.15.69:0/2141083760 shutdown_connections 2024-05-09T09:18:38.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- 172.21.15.69:0/2141083760 wait complete. 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 Processor -- start 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- start start 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90101960 0x7f9c90196f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9010c3d0 con 0x7f9c90102320 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9010c250 con 0x7f9c90101960 2024-05-09T09:18:38.233 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.228+0000 7f9c96142640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9c9010c550 con 0x7f9c90100770 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43018/0 (socket says 172.21.15.69:43018) 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 -- 172.21.15.69:0/68968462 learned_addr learned my addr 172.21.15.69:0/68968462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8effd640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90101960 0x7f9c90196f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8f7fe640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.234 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 -- 172.21.15.69:0/68968462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 msgr2=0x7f9c90196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 -- 172.21.15.69:0/68968462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90101960 msgr2=0x7f9c90196f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90101960 0x7f9c90196f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 -- 172.21.15.69:0/68968462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9c90069f50 con 0x7f9c90102320 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8f7fe640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8ffff640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800cad0 tx=0x7f9c8800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800ae20 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9c9006a240 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9c88005020 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9c8800f6f0 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.229+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9c9006a710 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.230+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9c90107a90 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.231+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9c880051d0 con 0x7f9c90102320 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.232+0000 7f9c8cff9640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 0x7f9c7c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.232+0000 7f9c8f7fe640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 0x7f9c7c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.232+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f9c880cf460 con 0x7f9c90102320 2024-05-09T09:18:38.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.233+0000 7f9c8f7fe640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 0x7f9c7c0a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c901017c0 tx=0x7f9c7800a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.236+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9c88014030 con 0x7f9c90102320 2024-05-09T09:18:38.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.414+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f9c9006a9d0 con 0x7f9c90102320 2024-05-09T09:18:38.413 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.415+0000 7f9c8cff9640 1 -- 172.21.15.69:0/68968462 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f9c8809c040 con 0x7f9c90102320 2024-05-09T09:18:38.414 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 msgr2=0x7f9c7c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 0x7f9c7c0a5050 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f9c901017c0 tx=0x7f9c7800a5e0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 msgr2=0x7f9c9019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f9c8800cad0 tx=0x7f9c8800cf90 comp rx=0 tx=0).stop 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 shutdown_connections 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9c7c0a2b90 0x7f9c7c0a5050 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9c90102320 0x7f9c9019dfc0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9c90101960 0x7f9c90196f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 --2- 172.21.15.69:0/68968462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9c90100770 0x7f9c90196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 >> 172.21.15.69:0/68968462 conn(0x7f9c900fbce0 msgr2=0x7f9c90104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 shutdown_connections 2024-05-09T09:18:38.417 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.417+0000 7f9c96142640 1 -- 172.21.15.69:0/68968462 wait complete. 2024-05-09T09:18:38.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:38 smithi069 ceph-mon[30123]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:38.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3853014015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:38.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1688697981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.426 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:38 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/245550328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.427 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:38 smithi103 ceph-mon[34421]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3853014015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1688697981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:38 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/245550328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2182920492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4100790 msgr2=0x7fb0f4100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2182920492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4100790 0x7fb0f4100bb0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4009900 tx=0x7fb0e402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:38.531 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2182920492 shutdown_connections 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2182920492 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4102340 0x7fb0f4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2182920492 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 0x7fb0f4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2182920492 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4100790 0x7fb0f4100bb0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2182920492 >> 172.21.15.69:0/2182920492 conn(0x7fb0f40fbd20 msgr2=0x7fb0f40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2182920492 shutdown_connections 2024-05-09T09:18:38.532 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.532+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2182920492 wait complete. 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 Processor -- start 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 -- start start 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4100790 0x7fb0f4197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 0x7fb0f4198160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410c400 con 0x7fb0f4102340 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410c280 con 0x7fb0f4100790 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f90d4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb0f410c580 con 0x7fb0f4101980 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43030/0 (socket says 172.21.15.69:43030) 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 -- 172.21.15.69:0/2373669387 learned_addr learned my addr 172.21.15.69:0/2373669387 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f2d9d640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4100790 0x7fb0f4197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f259c640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 0x7fb0f4198160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 -- 172.21.15.69:0/2373669387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 msgr2=0x7fb0f4198160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 0x7fb0f4198160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 -- 172.21.15.69:0/2373669387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4100790 msgr2=0x7fb0f4197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4100790 0x7fb0f4197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.533+0000 7fb0f359e640 1 -- 172.21.15.69:0/2373669387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb0f419f830 con 0x7fb0f4102340 2024-05-09T09:18:38.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0f359e640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00efd0 tx=0x7fb0ec00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec003b70 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb0ec003d10 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb0f419fb20 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb0ec0109f0 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.534+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb0f419fff0 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.535+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb0f41a0280 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.536+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb0ec010b90 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.537+0000 7fb0dbfff640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 0x7fb0dc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.537+0000 7fb0f2d9d640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 0x7fb0dc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.537+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fb0ec0cfcd0 con 0x7fb0f4102340 2024-05-09T09:18:38.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.538+0000 7fb0f2d9d640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 0x7fb0dc0a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005ab0 tx=0x7fb0e4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.541+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb0ec09c8b0 con 0x7fb0f4102340 2024-05-09T09:18:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:38 smithi163 ceph-mon[34354]: pgmap v357: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-05-09T09:18:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3853014015' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1688697981' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:38 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/245550328' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:38.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.717+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fb0f41a0450 con 0x7fb0f4102340 2024-05-09T09:18:38.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.717+0000 7fb0dbfff640 1 -- 172.21.15.69:0/2373669387 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7fb0ec09c1b0 con 0x7fb0f4102340 2024-05-09T09:18:38.716 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:38.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 msgr2=0x7fb0dc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 0x7fb0dc0a5050 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fb0e4005ab0 tx=0x7fb0e4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 msgr2=0x7fb0f419f190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fb0ec00efd0 tx=0x7fb0ec00c6a0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 shutdown_connections 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb0dc0a2b90 0x7fb0dc0a5050 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb0f4102340 0x7fb0f419f190 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb0f4101980 0x7fb0f4198160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 --2- 172.21.15.69:0/2373669387 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb0f4100790 0x7fb0f4197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 >> 172.21.15.69:0/2373669387 conn(0x7fb0f40fbd20 msgr2=0x7fb0f4069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 shutdown_connections 2024-05-09T09:18:38.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.719+0000 7fb0f90d4640 1 -- 172.21.15.69:0/2373669387 wait complete. 2024-05-09T09:18:38.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:38.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 -- 172.21.15.69:0/4123501559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 msgr2=0x7f25d01003d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.832 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/4123501559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d01003d0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f25c0009900 tx=0x7f25c002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 -- 172.21.15.69:0/4123501559 shutdown_connections 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/4123501559 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d0108360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/4123501559 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 0x7f25d0101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/4123501559 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d01003d0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 -- 172.21.15.69:0/4123501559 >> 172.21.15.69:0/4123501559 conn(0x7f25d00fb020 msgr2=0x7f25d00fd460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.831+0000 7f25d5b5a640 1 -- 172.21.15.69:0/4123501559 shutdown_connections 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 -- 172.21.15.69:0/4123501559 wait complete. 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 Processor -- start 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 -- start start 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 0x7f25d0198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d019f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010d530 con 0x7f25d00fffb0 2024-05-09T09:18:38.833 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010d3b0 con 0x7f25d0101a40 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25d5b5a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f25d010d6b0 con 0x7f25d0101080 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25cf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25cf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43048/0 (socket says 172.21.15.69:43048) 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25cf7fe640 1 -- 172.21.15.69:0/1558064967 learned_addr learned my addr 172.21.15.69:0/1558064967 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25c7fff640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 0x7f25d0198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.832+0000 7f25cffff640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d019f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 -- 172.21.15.69:0/1558064967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 msgr2=0x7f25d0198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 0x7f25d0198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 -- 172.21.15.69:0/1558064967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 msgr2=0x7f25d019f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d019f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 -- 172.21.15.69:0/1558064967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f25d019f750 con 0x7f25d00fffb0 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cffff640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cf7fe640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f25c0031610 tx=0x7f25c00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c003f030 con 0x7f25d00fffb0 2024-05-09T09:18:38.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25c0003ce0 con 0x7f25d00fffb0 2024-05-09T09:18:38.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f25d019f9e0 con 0x7f25d00fffb0 2024-05-09T09:18:38.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25c0035070 con 0x7f25d00fffb0 2024-05-09T09:18:38.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.833+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f25d019fed0 con 0x7f25d00fffb0 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.834+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f25d0107240 con 0x7f25d00fffb0 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.835+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f25c0042030 con 0x7f25d00fffb0 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.836+0000 7f25cd7fa640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 0x7f25bc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.836+0000 7f25c7fff640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 0x7f25bc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.836+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f25c00efa10 con 0x7f25d00fffb0 2024-05-09T09:18:38.836 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.836+0000 7f25c7fff640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 0x7f25bc0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f25b8005fd0 tx=0x7f25b8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:38.838 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:38.840+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f25c00bc5f0 con 0x7f25d00fffb0 2024-05-09T09:18:39.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.051+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f25d0062430 con 0x7f25d00fffb0 2024-05-09T09:18:39.050 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.052+0000 7f25cd7fa640 1 -- 172.21.15.69:0/1558064967 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 24 v24) ==== 94+0+11879 (secure 0 0 0) 0x7f25c00bbef0 con 0x7f25d00fffb0 2024-05-09T09:18:39.051 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:39.051 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":24,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":24,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:16:53.206774+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:18:39.051 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 24 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 msgr2=0x7f25bc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 0x7f25bc0a5050 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f25b8005fd0 tx=0x7f25b8005850 comp rx=0 tx=0).stop 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 msgr2=0x7f25d0197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f25c0031610 tx=0x7f25c00043b0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 shutdown_connections 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25bc0a2b90 0x7f25bc0a5050 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f25d0101a40 0x7f25d019f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f25d0101080 0x7f25d0198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 --2- 172.21.15.69:0/1558064967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f25d00fffb0 0x7f25d0197b30 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 >> 172.21.15.69:0/1558064967 conn(0x7f25d00fb020 msgr2=0x7f25d0104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 shutdown_connections 2024-05-09T09:18:39.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.054+0000 7f25d5b5a640 1 -- 172.21.15.69:0/1558064967 wait complete. 2024-05-09T09:18:39.068 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:39.171 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 -- 172.21.15.69:0/530295786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8102300 msgr2=0x7f55a81046f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 --2- 172.21.15.69:0/530295786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8102300 0x7f55a81046f0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f5598009900 tx=0x7f559802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 -- 172.21.15.69:0/530295786 shutdown_connections 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 --2- 172.21.15.69:0/530295786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8107560 0x7f55a81099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 --2- 172.21.15.69:0/530295786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8104c30 0x7f55a8107020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 --2- 172.21.15.69:0/530295786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8102300 0x7f55a81046f0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.170+0000 7f55ae453640 1 -- 172.21.15.69:0/530295786 >> 172.21.15.69:0/530295786 conn(0x7f55a80fbd10 msgr2=0x7f55a80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- 172.21.15.69:0/530295786 shutdown_connections 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- 172.21.15.69:0/530295786 wait complete. 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 Processor -- start 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- start start 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 0x7f55a8101c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8107560 0x7f55a81007f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.172 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810d530 con 0x7f55a8104c30 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810d3b0 con 0x7f55a8102300 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.171+0000 7f55ae453640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f55a810d6b0 con 0x7f55a8107560 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43066/0 (socket says 172.21.15.69:43066) 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 -- 172.21.15.69:0/3566377378 learned_addr learned my addr 172.21.15.69:0/3566377378 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55ac9f0640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8107560 0x7f55a81007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a7fff640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 0x7f55a8101c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.173 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 -- 172.21.15.69:0/3566377378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8107560 msgr2=0x7f55a81007f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8107560 0x7f55a81007f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 -- 172.21.15.69:0/3566377378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 msgr2=0x7f55a8101c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 0x7f55a8101c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.174 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 -- 172.21.15.69:0/3566377378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f55a81010b0 con 0x7f55a8104c30 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a7fff640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 0x7f55a8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a77fe640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f559c00d740 tx=0x7f559c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c017070 con 0x7f55a8104c30 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f55a81a1d50 con 0x7f55a8104c30 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f559c00de50 con 0x7f55a8104c30 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f559c004e00 con 0x7f55a8104c30 2024-05-09T09:18:39.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.172+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f55a81a2210 con 0x7f55a8104c30 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.173+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f55a80fd800 con 0x7f55a8104c30 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.174+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f559c004fa0 con 0x7f55a8104c30 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.175+0000 7f55a57fa640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 0x7f55900a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.175+0000 7f55a7fff640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 0x7f55900a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.175+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f559c0ced90 con 0x7f55a8104c30 2024-05-09T09:18:39.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.176+0000 7f55a7fff640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 0x7f55900a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5598000c00 tx=0x7f5598031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.180+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f559c09b970 con 0x7f55a8104c30 2024-05-09T09:18:39.357 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.359+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f55a8062460 con 0x7f55a8104c30 2024-05-09T09:18:39.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.360+0000 7f55a57fa640 1 -- 172.21.15.69:0/3566377378 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7f559c09b270 con 0x7f55a8104c30 2024-05-09T09:18:39.359 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:39.360 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.362+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 msgr2=0x7f55900a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.362+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 0x7f55900a5050 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5598000c00 tx=0x7f5598031400 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.362+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 msgr2=0x7f55a81002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.362+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7f559c00d740 tx=0x7f559c00dc00 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 shutdown_connections 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f55900a2b90 0x7f55900a5050 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f55a8107560 0x7f55a81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f55a8104c30 0x7f55a81002b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 --2- 172.21.15.69:0/3566377378 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f55a8102300 0x7f55a8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 >> 172.21.15.69:0/3566377378 conn(0x7f55a80fbd10 msgr2=0x7f55a81030d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 shutdown_connections 2024-05-09T09:18:39.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.363+0000 7f55ae453640 1 -- 172.21.15.69:0/3566377378 wait complete. 2024-05-09T09:18:39.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/68968462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2373669387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:39 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1558064967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:39.372 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:39 smithi069 ceph-mon[30123]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:18:39.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 24, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-05-09T09:16:32.211035+0000', 'modified': '2024-05-09T09:16:53.206774+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}, 'max_mds': 3, 'in': [0, 1, 2], 'up': {'mds_0': 24527, 'mds_1': 24511, 'mds_2': 24491}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24458': {'gid': 24458, 'name': 'e', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.103:6835/3790151668', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6834', 'nonce': 3790151668}, {'type': 'v1', 'addr': '172.21.15.103:6835', 'nonce': 3790151668}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24478': {'gid': 24478, 'name': 'f', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6835/989258568', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6834', 'nonce': 989258568}, {'type': 'v1', 'addr': '172.21.15.163:6835', 'nonce': 989258568}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24490': {'gid': 24490, 'name': 'c', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.163:6837/3224689113', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6836', 'nonce': 3224689113}, {'type': 'v1', 'addr': '172.21.15.163:6837', 'nonce': 3224689113}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24491': {'gid': 24491, 'name': 'h', 'rank': 2, 'incarnation': 22, 'state': 'up:active', 'state_seq': 11, 'addr': '172.21.15.103:6837/1084447179', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6836', 'nonce': 1084447179}, {'type': 'v1', 'addr': '172.21.15.103:6837', 'nonce': 1084447179}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24511': {'gid': 24511, 'name': 'i', 'rank': 1, 'incarnation': 20, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.163:6839/282166335', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.163:6838', 'nonce': 282166335}, {'type': 'v1', 'addr': '172.21.15.163:6839', 'nonce': 282166335}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}, 'gid_24527': {'gid': 24527, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.103:6839/189860875', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.103:6838', 'nonce': 189860875}, {'type': 'v1', 'addr': '172.21.15.103:6839', 'nonce': 189860875}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments', 'feature_12': 'quiesce subvolumes'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24527, 'qdb_cluster': [24491, 24511, 24527]} 2024-05-09T09:18:39.374 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 3/3 2024-05-09T09:18:39.374 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.h status 2024-05-09T09:18:39.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 -- 172.21.15.69:0/2136477043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8103440 msgr2=0x7f38c8109cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.476 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 --2- 172.21.15.69:0/2136477043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8103440 0x7f38c8109cd0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009ca0 tx=0x7f38c002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 -- 172.21.15.69:0/2136477043 shutdown_connections 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 --2- 172.21.15.69:0/2136477043 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8103440 0x7f38c8109cd0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 --2- 172.21.15.69:0/2136477043 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 --2- 172.21.15.69:0/2136477043 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8101890 0x7f38c8101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 -- 172.21.15.69:0/2136477043 >> 172.21.15.69:0/2136477043 conn(0x7f38c80fce00 msgr2=0x7f38c80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 -- 172.21.15.69:0/2136477043 shutdown_connections 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.477+0000 7f38ccf90640 1 -- 172.21.15.69:0/2136477043 wait complete. 2024-05-09T09:18:39.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 Processor -- start 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 -- start start 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 0x7f38c819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f38c810d510 con 0x7f38c8101890 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f38c810d390 con 0x7f38c8103440 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38ccf90640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f38c810d690 con 0x7f38c8102a80 2024-05-09T09:18:39.479 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43090/0 (socket says 172.21.15.69:43090) 2024-05-09T09:18:39.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 -- 172.21.15.69:0/4090258259 learned_addr learned my addr 172.21.15.69:0/4090258259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 0x7f38c819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.480 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 -- 172.21.15.69:0/4090258259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 msgr2=0x7f38c8198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.481 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 -- 172.21.15.69:0/4090258259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 msgr2=0x7f38c819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 0x7f38c819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c659c640 1 -- 172.21.15.69:0/4090258259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f38c819f770 con 0x7f38c8101890 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 0x7f38c819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.478+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38c659c640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d770 tx=0x7f38b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38bffff640 1 -- 172.21.15.69:0/4090258259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8017070 con 0x7f38c8101890 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f38c819fa60 con 0x7f38c8101890 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38bffff640 1 -- 172.21.15.69:0/4090258259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b8009c30 con 0x7f38c8101890 2024-05-09T09:18:39.482 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38bffff640 1 -- 172.21.15.69:0/4090258259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8003dd0 con 0x7f38c8101890 2024-05-09T09:18:39.483 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.479+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f38c819ffb0 con 0x7f38c8101890 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.481+0000 7f38bffff640 1 -- 172.21.15.69:0/4090258259 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f38b8003fb0 con 0x7f38c8101890 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.481+0000 7f38cfd80740 1 Processor -- start 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.481+0000 7f38cfd80740 1 -- start start 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922d8c0 0x55bb5922dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 0x55bb592358b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb591914d0 con 0x55bb5922e2b0 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb591393e0 con 0x55bb59233420 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38cfd80740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb5918d490 con 0x55bb5922d8c0 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43104/0 (socket says 172.21.15.69:43104) 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 -- 172.21.15.69:0/215466576 learned_addr learned my addr 172.21.15.69:0/215466576 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922d8c0 0x55bb5922dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 0x55bb592358b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 -- 172.21.15.69:0/215466576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922d8c0 msgr2=0x55bb5922dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922d8c0 0x55bb5922dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 -- 172.21.15.69:0/215466576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 msgr2=0x55bb592358b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 0x55bb592358b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 -- 172.21.15.69:0/215466576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb58ea52d0 con 0x55bb5922e2b0 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 0x55bb592358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38bffff640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 0x7f38b40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 0x7f38b40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38c659c640 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f38b80d2f10 tx=0x7f38b80d74b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c5e9d533ddcaa65 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.482+0000 7f38bffff640 1 -- 172.21.15.69:0/4090258259 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f38b80cee00 con 0x7f38c8101890 2024-05-09T09:18:39.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38bcd99640 1 -- 172.21.15.69:0/215466576 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8020070 con 0x55bb5922e2b0 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38bcd99640 1 -- 172.21.15.69:0/215466576 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b8095e50 con 0x55bb5922e2b0 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38bcd99640 1 -- 172.21.15.69:0/215466576 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b809b0b0 con 0x55bb5922e2b0 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 -- 172.21.15.69:0/215466576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 msgr2=0x55bb59232db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 0x7f38b40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38c0031c10 tx=0x7f38c00093f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f38b80d2f10 tx=0x7f38b80d74b0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 -- 172.21.15.69:0/215466576 shutdown_connections 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb59233420 0x55bb592358b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb59232db0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 --2- 172.21.15.69:0/215466576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922d8c0 0x55bb5922dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.483+0000 7f38cfd80740 1 -- 172.21.15.69:0/215466576 >> 172.21.15.69:0/215466576 conn(0x55bb59228f30 msgr2=0x55bb59229330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- 172.21.15.69:0/215466576 shutdown_connections 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- 172.21.15.69:0/215466576 wait complete. 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- start start 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 Processor -- start 2024-05-09T09:18:39.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59237d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb592363d0 con 0x55bb5922e2b0 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb5914c8e0 con 0x55bb5922d8c0 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb59236550 con 0x55bb59233420 2024-05-09T09:18:39.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38cfd80740 10 client.0 _ll_register_callbacks cb 0x55bb591199c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43108/0 (socket says 172.21.15.69:43108) 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 -- 172.21.15.69:0/1675665858 learned_addr learned my addr 172.21.15.69:0/1675665858 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38c659c640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59237d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 -- 172.21.15.69:0/1675665858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 msgr2=0x55bb5923f2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923f2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.484+0000 7f38bdd9b640 1 -- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 msgr2=0x55bb59237d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59237d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38bdd9b640 1 -- 172.21.15.69:0/1675665858 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb58f6f6b0 con 0x55bb5922e2b0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38c659c640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59237d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f38b0010670 tx=0x7f38b0010a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b314f2877de9a6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b000e940 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f38b000eae0 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38b8095e50 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f38b0025050 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 10 client.15393 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bb59242340 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38b8095e50 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f38b000f040 con 0x55bb5922e2b0 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 10 client.15393 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 10 client.15393 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 20 client.15393 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb59244ac0 0x55bb59245f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 4 client.15393 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:39.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.485+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bb5919ec00 con 0x55bb59244ac0 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.486+0000 7f38c659c640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb59244ac0 0x55bb59245f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.486+0000 7f38adffb640 10 client.15393 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.486+0000 7f38c659c640 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb59244ac0 0x55bb59245f30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=87e9c7116a73d684 server_cookie=f2c28f9abbbd9ef4 in_seq=0 out_seq=0 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38adffb640 1 -- 172.21.15.69:0/1675665858 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bb5919ec00 con 0x55bb59244ac0 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38adffb640 10 client.15393 handle_command_reply: tid=0 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 client.15393 shutdown 2024-05-09T09:18:39.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 msgr2=0x55bb592382b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7f38b0010670 tx=0x7f38b0010a70 comp rx=0 tx=0).stop 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 shutdown_connections 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb59244ac0 0x55bb59245f30 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923f2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922e2b0 0x55bb592382b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38adffb640 0 client.15393 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1675665858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59237d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 >> 172.21.15.69:0/1675665858 conn(0x55bb59228f30 msgr2=0x55bb59228000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 shutdown_connections 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 1 -- 172.21.15.69:0/1675665858 wait complete. 2024-05-09T09:18:39.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.487+0000 7f38cfd80740 20 client.15393 trim_cache size 0 max 16384 2024-05-09T09:18:39.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 Processor -- start 2024-05-09T09:18:39.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 -- start start 2024-05-09T09:18:39.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59238090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922e2b0 0x55bb5923f2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb59244270 con 0x55bb59233420 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb59244940 con 0x55bb5922d8c0 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38cfd80740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb59245010 con 0x55bb5922e2b0 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43116/0 (socket says 172.21.15.69:43116) 2024-05-09T09:18:39.513 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 -- 172.21.15.69:0/410656789 learned_addr learned my addr 172.21.15.69:0/410656789 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922e2b0 0x55bb5923f2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c659c640 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59238090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 -- 172.21.15.69:0/410656789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922e2b0 msgr2=0x55bb5923f2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922e2b0 0x55bb5923f2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 -- 172.21.15.69:0/410656789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 msgr2=0x55bb59238090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59238090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.505+0000 7f38c6d9d640 1 -- 172.21.15.69:0/410656789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bb59228a70 con 0x55bb59233420 2024-05-09T09:18:39.514 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38c6d9d640 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009ca0 tx=0x7f38c0040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff8eac74e1819118 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38affff640 1 -- 172.21.15.69:0/410656789 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c0004040 con 0x55bb59233420 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38affff640 1 -- 172.21.15.69:0/410656789 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0036040 con 0x55bb59233420 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38affff640 1 -- 172.21.15.69:0/410656789 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38c002fc40 con 0x55bb59233420 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 -- 172.21.15.69:0/410656789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 msgr2=0x55bb59236980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f38c0009ca0 tx=0x7f38c0040000 comp rx=0 tx=0).stop 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 -- 172.21.15.69:0/410656789 shutdown_connections 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb59233420 0x55bb59236980 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.515 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb5922e2b0 0x55bb5923f2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 --2- 172.21.15.69:0/410656789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922d8c0 0x55bb59238090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 -- 172.21.15.69:0/410656789 >> 172.21.15.69:0/410656789 conn(0x55bb59228f30 msgr2=0x55bb59237980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 -- 172.21.15.69:0/410656789 shutdown_connections 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.506+0000 7f38cfd80740 1 -- 172.21.15.69:0/410656789 wait complete. 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 -- start start 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 Processor -- start 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922e2b0 0x55bb5923b140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.516 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bb59245db0 con 0x55bb5922d8c0 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bb592456e0 con 0x55bb5922e2b0 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bb59246480 con 0x55bb59233420 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38cfd80740 10 client.0 _ll_register_callbacks cb 0x7f38b000b460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43120/0 (socket says 172.21.15.69:43120) 2024-05-09T09:18:39.517 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 -- 172.21.15.69:0/1161903735 learned_addr learned my addr 172.21.15.69:0/1161903735 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922e2b0 0x55bb5923b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 -- 172.21.15.69:0/1161903735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 msgr2=0x55bb5923e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 -- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922e2b0 msgr2=0x55bb5923b140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922e2b0 0x55bb5923b140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 -- 172.21.15.69:0/1161903735 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bb5923e750 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38c659c640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f38b80d79b0 tx=0x7f38b8005f30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1200887279fa5cb7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38adffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.507+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f38b8020040 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f38b80042a0 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f38c0036040 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f38b809ac40 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 10 client.15399 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55bb592aac30 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f38c0036040 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f38b8097950 con 0x55bb5922d8c0 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 10 client.15399 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 10 client.15399 resolve_mds: resolved name 'h' to daemon mds.h 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 20 client.15399 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb592af410 0x55bb592ae120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 4 client.15399 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "status"}) -- 0x55bb59198f50 con 0x55bb592af410 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.508+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb592af410 0x55bb592ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38bdd9b640 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb592af410 0x55bb592ae120 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=fc4419b744dc2a71 server_cookie=52955abdcc9cd03b in_seq=0 out_seq=0 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38adffb640 10 client.15399 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38adffb640 1 -- 172.21.15.69:0/1161903735 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+326 (crc 0 0 0) 0x55bb59198f50 con 0x55bb592af410 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38adffb640 10 client.15399 handle_command_reply: tid=0 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 client.15399 shutdown 2024-05-09T09:18:39.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 msgr2=0x55bb5923ac00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f38b80d79b0 tx=0x7f38b8005f30 comp rx=0 tx=0).stop 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 shutdown_connections 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55bb592af410 0x55bb592ae120 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bb59233420 0x55bb5923e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bb5922e2b0 0x55bb5923b140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38adffb640 0 client.15399 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 --2- 172.21.15.69:0/1161903735 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bb5922d8c0 0x55bb5923ac00 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 >> 172.21.15.69:0/1161903735 conn(0x55bb59228f30 msgr2=0x55bb59236980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 shutdown_connections 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 1 -- 172.21.15.69:0/1161903735 wait complete. 2024-05-09T09:18:39.522 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.509+0000 7f38cfd80740 20 client.15399 trim_cache size 0 max 16384 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 2, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24491, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 107.31427053, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 93, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:39.523 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 144.29905477200001 2024-05-09T09:18:39.524 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 msgr2=0x7f38b40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 0x7f38b40a5050 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f38c0031c10 tx=0x7f38c00093f0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 msgr2=0x7f38c8197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f38b800d770 tx=0x7f38b800dc30 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 shutdown_connections 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f38b40a2b90 0x7f38b40a5050 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f38c8103440 0x7f38c819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f38c8102a80 0x7f38c8198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 --2- 172.21.15.69:0/4090258259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f38c8101890 0x7f38c8197b50 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 >> 172.21.15.69:0/4090258259 conn(0x7f38c80fce00 msgr2=0x7f38c8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 shutdown_connections 2024-05-09T09:18:39.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.510+0000 7f38ccf90640 1 -- 172.21.15.69:0/4090258259 wait complete. 2024-05-09T09:18:39.526 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.i status 2024-05-09T09:18:39.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/68968462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2373669387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:39 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1558064967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:39.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:39 smithi163 ceph-mon[34354]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:18:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/68968462' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2373669387' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:39 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1558064967' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:39 smithi103 ceph-mon[34421]: pgmap v358: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 -- 172.21.15.69:0/1097104722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48102340 msgr2=0x7f7c48108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/1097104722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48102340 0x7f7c48108bd0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009ca0 tx=0x7f7c4002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 -- 172.21.15.69:0/1097104722 shutdown_connections 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/1097104722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48102340 0x7f7c48108bd0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/1097104722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48101980 0x7f7c48101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/1097104722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c48100790 0x7f7c48100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 -- 172.21.15.69:0/1097104722 >> 172.21.15.69:0/1097104722 conn(0x7f7c480fbd00 msgr2=0x7f7c480fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 -- 172.21.15.69:0/1097104722 shutdown_connections 2024-05-09T09:18:39.631 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.629+0000 7f7c4fc53640 1 -- 172.21.15.69:0/1097104722 wait complete. 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 Processor -- start 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 -- start start 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c48100790 0x7f7c48197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48102340 0x7f7c4819f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4810c410 con 0x7f7c48101980 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4810c290 con 0x7f7c48102340 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4fc53640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7c4810c590 con 0x7f7c48100790 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43130/0 (socket says 172.21.15.69:43130) 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/366093882 learned_addr learned my addr 172.21.15.69:0/366093882 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48102340 0x7f7c4819f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/366093882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c48100790 msgr2=0x7f7c48197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.632 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c48100790 0x7f7c48197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/366093882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48102340 msgr2=0x7f7c4819f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48102340 0x7f7c4819f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.630+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/366093882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7c4819f750 con 0x7f7c48101980 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400d950 tx=0x7f7c4400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c3effd640 1 -- 172.21.15.69:0/366093882 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c44017030 con 0x7f7c48101980 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c3effd640 1 -- 172.21.15.69:0/366093882 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c44004020 con 0x7f7c48101980 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c3effd640 1 -- 172.21.15.69:0/366093882 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c440047a0 con 0x7f7c48101980 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7c4819fa40 con 0x7f7c48101980 2024-05-09T09:18:39.633 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.631+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7c4819ffe0 con 0x7f7c48101980 2024-05-09T09:18:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.633+0000 7f7c3effd640 1 -- 172.21.15.69:0/366093882 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7c440041d0 con 0x7f7c48101980 2024-05-09T09:18:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.633+0000 7f7c52a33740 1 Processor -- start 2024-05-09T09:18:39.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 -- start start 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed64f6ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed64fe8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ed645a4d0 con 0x564ed64f72b0 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ed64023e0 con 0x564ed64f68c0 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c52a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564ed6456490 con 0x564ed64fc420 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43146/0 (socket says 172.21.15.69:43146) 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 -- 172.21.15.69:0/1298676630 learned_addr learned my addr 172.21.15.69:0/1298676630 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.641 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c3effd640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 0x7f7c340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed64fe8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed64f6ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 0x7f7c340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 -- 172.21.15.69:0/1298676630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 msgr2=0x564ed64fe8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed64fe8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 -- 172.21.15.69:0/1298676630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 msgr2=0x564ed64f6ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed64f6ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 -- 172.21.15.69:0/1298676630 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed616e2d0 con 0x564ed64f72b0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed64fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c3effd640 1 -- 172.21.15.69:0/366093882 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f7c440cede0 con 0x7f7c48101980 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.634+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800bb00 tx=0x7f7c3800d2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13eaf961ff899269 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c33fff640 1 -- 172.21.15.69:0/1298676630 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38016070 con 0x564ed64f72b0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 0x7f7c340a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7c40031c10 tx=0x7f7c4003f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.642 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c33fff640 1 -- 172.21.15.69:0/1298676630 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3800dd10 con 0x564ed64f72b0 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c33fff640 1 -- 172.21.15.69:0/1298676630 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c38010460 con 0x564ed64f72b0 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 -- 172.21.15.69:0/1298676630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 msgr2=0x564ed64fbdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f7c3800bb00 tx=0x7f7c3800d2e0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 -- 172.21.15.69:0/1298676630 shutdown_connections 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed64fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed64fbdb0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1298676630 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed64f6ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 -- 172.21.15.69:0/1298676630 >> 172.21.15.69:0/1298676630 conn(0x564ed64f1f30 msgr2=0x564ed64f2330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.635+0000 7f7c52a33740 1 -- 172.21.15.69:0/1298676630 shutdown_connections 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 -- 172.21.15.69:0/1298676630 wait complete. 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 -- start start 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 Processor -- start 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.643 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 0x564ed65081f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ed64ff3d0 con 0x564ed64f72b0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ed64158e0 con 0x564ed64fc420 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564ed64ff550 con 0x564ed64f68c0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c52a33740 10 client.0 _ll_register_callbacks cb 0x564ed63e29c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43162/0 (socket says 172.21.15.69:43162) 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/4138414270 learned_addr learned my addr 172.21.15.69:0/4138414270 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.636+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 0x564ed65081f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 msgr2=0x564ed6500cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/4138414270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 msgr2=0x564ed65081f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 0x564ed65081f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.644 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/4138414270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed62386b0 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 0x564ed65081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c44097720 tx=0x7f7c4406a240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13e251668f13512d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c440663a0 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f7c44011d80 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c3800dd10 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7c4406a080 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 10 client.15411 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ed650b1d0 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c3800dd10 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7c44097930 con 0x564ed64f72b0 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 10 client.15411 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 10 client.15411 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 20 client.15411 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed650d8b0 0x564ed650ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 4 client.15411 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.637+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564ed6467c00 con 0x564ed650d8b0 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.638+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed650d8b0 0x564ed650ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.638+0000 7f7c317fa640 10 client.15411 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.638+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed650d8b0 0x564ed650ed20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=8631e25a5141d135 server_cookie=9dc80ad0795e3f28 in_seq=0 out_seq=0 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c317fa640 1 -- 172.21.15.69:0/4138414270 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564ed6467c00 con 0x564ed650d8b0 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c317fa640 10 client.15411 handle_command_reply: tid=0 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 client.15411 shutdown 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 msgr2=0x564ed65011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f7c44097720 tx=0x7f7c4406a240 comp rx=0 tx=0).stop 2024-05-09T09:18:39.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 shutdown_connections 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed650d8b0 0x564ed650ed20 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64fc420 0x564ed65081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c317fa640 0 client.15411 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 --2- 172.21.15.69:0/4138414270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 >> 172.21.15.69:0/4138414270 conn(0x564ed64f1f30 msgr2=0x564ed64f1000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 shutdown_connections 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 1 -- 172.21.15.69:0/4138414270 wait complete. 2024-05-09T09:18:39.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.639+0000 7f7c52a33740 20 client.15411 trim_cache size 0 max 16384 2024-05-09T09:18:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 Processor -- start 2024-05-09T09:18:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 -- start start 2024-05-09T09:18:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.663 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed65080c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ed650d060 con 0x564ed64f72b0 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ed650d730 con 0x564ed64f68c0 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c52a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564ed650de00 con 0x564ed64fc420 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4d1ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43170/0 (socket says 172.21.15.69:43170) 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/1653966304 learned_addr learned my addr 172.21.15.69:0/1653966304 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/1653966304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 msgr2=0x564ed65080c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.657+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed65080c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed65080c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/1653966304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 msgr2=0x564ed6500cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d1ee640 1 -- 172.21.15.69:0/1653966304 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564ed64f2980 con 0x564ed64f72b0 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009610 tx=0x7f7c44005bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=680c06533ab7fff6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c337fe640 1 -- 172.21.15.69:0/1653966304 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c4409ae00 con 0x564ed64f72b0 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c337fe640 1 -- 172.21.15.69:0/1653966304 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4409b9c0 con 0x564ed64f72b0 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c337fe640 1 -- 172.21.15.69:0/1653966304 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c440b4110 con 0x564ed64f72b0 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 -- 172.21.15.69:0/1653966304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 msgr2=0x564ed65011f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f7c44009610 tx=0x7f7c44005bf0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 -- 172.21.15.69:0/1653966304 shutdown_connections 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64fc420 0x564ed65080c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64f72b0 0x564ed65011f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1653966304 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f68c0 0x564ed6500cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 -- 172.21.15.69:0/1653966304 >> 172.21.15.69:0/1653966304 conn(0x564ed64f1f30 msgr2=0x564ed64f8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.658+0000 7f7c52a33740 1 -- 172.21.15.69:0/1653966304 shutdown_connections 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 -- 172.21.15.69:0/1653966304 wait complete. 2024-05-09T09:18:39.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 -- start start 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 Processor -- start 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6504040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 0x564ed6504580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564ed650eba0 con 0x564ed64fc420 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564ed650e4d0 con 0x564ed64f72b0 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564ed650f270 con 0x564ed64f68c0 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c52a33740 10 client.0 _ll_register_callbacks cb 0x7f7c44005130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4e1f0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4e1f0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43182/0 (socket says 172.21.15.69:43182) 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4e1f0640 1 -- 172.21.15.69:0/1598610499 learned_addr learned my addr 172.21.15.69:0/1598610499 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6504040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 0x564ed6504580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.659+0000 7f7c4e1f0640 1 -- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 msgr2=0x564ed6504040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6504040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4e1f0640 1 -- 172.21.15.69:0/1598610499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 msgr2=0x564ed6504580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 0x564ed6504580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4e1f0640 1 -- 172.21.15.69:0/1598610499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564ed6507b40 con 0x564ed64fc420 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6504040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4d1ee640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 0x564ed6504580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c4e1f0640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009ca0 tx=0x7f7c400095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f53a553fbc980b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7c400309d0 con 0x564ed64fc420 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f7c40030b70 con 0x564ed64fc420 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7c4409b9c0 con 0x564ed64fc420 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f7c4005e050 con 0x564ed64fc420 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 10 client.15417 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x564ed6573f10 con 0x564ed64fc420 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7c4409b9c0 con 0x564ed64fc420 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f7c4003cb00 con 0x564ed64fc420 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 10 client.15417 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 10 client.15417 resolve_mds: resolved name 'i' to daemon mds.i 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 20 client.15417 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed6578c00 0x564ed6577910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 4 client.15417 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.660+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "status"}) -- 0x564ed6461f50 con 0x564ed6578c00 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed6578c00 0x564ed6577910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c4d9ef640 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed6578c00 0x564ed6577910 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9c2d8ffc762829 server_cookie=cc5af5401dce069b in_seq=0 out_seq=0 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c317fa640 10 client.15417 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c317fa640 1 -- 172.21.15.69:0/1598610499 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+322 (crc 0 0 0) 0x564ed6461f50 con 0x564ed6578c00 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c317fa640 10 client.15417 handle_command_reply: tid=0 2024-05-09T09:18:39.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 client.15417 shutdown 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 msgr2=0x564ed6507550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f7c40009ca0 tx=0x7f7c400095c0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 shutdown_connections 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x564ed6578c00 0x564ed6577910 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564ed64fc420 0x564ed6507550 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564ed64f72b0 0x564ed6504580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 --2- 172.21.15.69:0/1598610499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564ed64f68c0 0x564ed6504040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c317fa640 0 client.15417 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.661+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 >> 172.21.15.69:0/1598610499 conn(0x564ed64f1f30 msgr2=0x564ed64f8610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 shutdown_connections 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c52a33740 1 -- 172.21.15.69:0/1598610499 wait complete. 2024-05-09T09:18:39.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c52a33740 20 client.15417 trim_cache size 0 max 16384 2024-05-09T09:18:39.670 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:39.670 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:39.670 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 1, 2024-05-09T09:18:39.670 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24511, 2024-05-09T09:18:39.670 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 108.471723318, 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 93, 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 141.643553935 2024-05-09T09:18:39.671 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:39.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 msgr2=0x7f7c340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 0x7f7c340a50d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f7c40031c10 tx=0x7f7c4003f040 comp rx=0 tx=0).stop 2024-05-09T09:18:39.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 msgr2=0x7f7c48198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f7c4400d950 tx=0x7f7c4400de10 comp rx=0 tx=0).stop 2024-05-09T09:18:39.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 shutdown_connections 2024-05-09T09:18:39.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7c340a2c10 0x7f7c340a50d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7c48102340 0x7f7c4819f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7c48101980 0x7f7c48198070 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 --2- 172.21.15.69:0/366093882 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7c48100790 0x7f7c48197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 >> 172.21.15.69:0/366093882 conn(0x7f7c480fbd00 msgr2=0x7f7c48069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 shutdown_connections 2024-05-09T09:18:39.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.662+0000 7f7c4fc53640 1 -- 172.21.15.69:0/366093882 wait complete. 2024-05-09T09:18:39.679 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-05-09T09:18:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 -- 172.21.15.69:0/2099377792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c102ea0 msgr2=0x7f197c1032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 --2- 172.21.15.69:0/2099377792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c102ea0 0x7f197c1032c0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f196c009900 tx=0x7f196c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:39.784 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 -- 172.21.15.69:0/2099377792 shutdown_connections 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 --2- 172.21.15.69:0/2099377792 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 0x7f197c109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 --2- 172.21.15.69:0/2099377792 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c0ff910 0x7f197c0ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 --2- 172.21.15.69:0/2099377792 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c102ea0 0x7f197c1032c0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 -- 172.21.15.69:0/2099377792 >> 172.21.15.69:0/2099377792 conn(0x7f197c0fb470 msgr2=0x7f197c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 -- 172.21.15.69:0/2099377792 shutdown_connections 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.782+0000 7f1981ca5640 1 -- 172.21.15.69:0/2099377792 wait complete. 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 Processor -- start 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 -- start start 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 0x7f197c100be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c102ea0 0x7f197c101120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f197c10d550 con 0x7f197c0ff910 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f197c10d3d0 con 0x7f197c102ea0 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1981ca5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f197c10d6d0 con 0x7f197c1002d0 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43206/0 (socket says 172.21.15.69:43206) 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 -- 172.21.15.69:0/3141264227 learned_addr learned my addr 172.21.15.69:0/3141264227 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f1973fff640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 0x7f197c100be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197bfff640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c102ea0 0x7f197c101120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 -- 172.21.15.69:0/3141264227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 msgr2=0x7f197c100be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 0x7f197c100be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 -- 172.21.15.69:0/3141264227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c102ea0 msgr2=0x7f197c101120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c102ea0 0x7f197c101120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 -- 172.21.15.69:0/3141264227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f197c1019b0 con 0x7f197c0ff910 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.783+0000 7f197b7fe640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f196c031610 tx=0x7f196c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.784+0000 7f19797fa640 1 -- 172.21.15.69:0/3141264227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c03f030 con 0x7f197c0ff910 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.784+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f197c1a1ce0 con 0x7f197c0ff910 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.784+0000 7f19797fa640 1 -- 172.21.15.69:0/3141264227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c003ce0 con 0x7f197c0ff910 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.784+0000 7f19797fa640 1 -- 172.21.15.69:0/3141264227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c035070 con 0x7f197c0ff910 2024-05-09T09:18:39.787 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.784+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f197c1a2180 con 0x7f197c0ff910 2024-05-09T09:18:39.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19797fa640 1 -- 172.21.15.69:0/3141264227 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f196c042030 con 0x7f197c0ff910 2024-05-09T09:18:39.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 Processor -- start 2024-05-09T09:18:39.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 -- start start 2024-05-09T09:18:39.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75eace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75f0420 0x55fee75f28b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fee754e4d0 con 0x55fee75eb2b0 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fee74f63e0 con 0x55fee75ea8c0 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f19849b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fee754a490 con 0x55fee75f0420 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f197b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f197b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43214/0 (socket says 172.21.15.69:43214) 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.786+0000 7f197b7fe640 1 -- 172.21.15.69:0/2513721937 learned_addr learned my addr 172.21.15.69:0/2513721937 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197bfff640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75eace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 -- 172.21.15.69:0/2513721937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75f0420 msgr2=0x55fee75f28b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f1973fff640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75f0420 0x55fee75f28b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75f0420 0x55fee75f28b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 -- 172.21.15.69:0/2513721937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 msgr2=0x55fee75eace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75eace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 -- 172.21.15.69:0/2513721937 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee72622d0 con 0x55fee75eb2b0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197bfff640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75eace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f19797fa640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 0x7f19680a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197bfff640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 0x7f19680a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f196c0f5bc0 tx=0x7f196c0f5cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a15595a8ee552ee2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f19797fa640 1 -- 172.21.15.69:0/3141264227 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f196c0ef9c0 con 0x7f197c0ff910 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f1971ffb640 1 -- 172.21.15.69:0/2513721937 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c049070 con 0x55fee75eb2b0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f1971ffb640 1 -- 172.21.15.69:0/2513721937 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c0c4b10 con 0x55fee75eb2b0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.787+0000 7f1971ffb640 1 -- 172.21.15.69:0/2513721937 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f196c06ee60 con 0x55fee75eb2b0 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- 172.21.15.69:0/2513721937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 msgr2=0x55fee75efdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f197bfff640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 0x7f19680a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55fee75ebb10 tx=0x7f1974009650 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f196c0f5bc0 tx=0x7f196c0f5cb0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- 172.21.15.69:0/2513721937 shutdown_connections 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75f0420 0x55fee75f28b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75efdb0 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 --2- 172.21.15.69:0/2513721937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75eace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- 172.21.15.69:0/2513721937 >> 172.21.15.69:0/2513721937 conn(0x55fee75e5f30 msgr2=0x55fee75e6330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- 172.21.15.69:0/2513721937 shutdown_connections 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- 172.21.15.69:0/2513721937 wait complete. 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 -- start start 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.788+0000 7f19849b3740 1 Processor -- start 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75f4950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75f4e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fee754e4d0 con 0x55fee75f0420 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fee75098e0 con 0x55fee75ea8c0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fee75f3190 con 0x55fee75eb2b0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 10 client.0 _ll_register_callbacks cb 0x55fee74d69c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43230/0 (socket says 172.21.15.69:43230) 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 -- 172.21.15.69:0/1774617709 learned_addr learned my addr 172.21.15.69:0/1774617709 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f1973fff640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75f4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197b7fe640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75f4950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 -- 172.21.15.69:0/1774617709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 msgr2=0x55fee75f4e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75f4e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 -- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 msgr2=0x55fee75f4950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75f4950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 -- 172.21.15.69:0/1774617709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee732c6b0 con 0x55fee75f0420 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f1973fff640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f197bfff640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1974019ef0 tx=0x7f1974004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9925033835896a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f195affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f19740168e0 con 0x55fee75f0420 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f1974016a80 con 0x55fee75f0420 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.789+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f196c0c4b10 con 0x55fee75f0420 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f197402d050 con 0x55fee75f0420 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 10 client.15429 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fee75fee90 con 0x55fee75f0420 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f196c0c4b10 con 0x55fee75f0420 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1974022400 con 0x55fee75f0420 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 10 client.15429 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 10 client.15429 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 20 client.15429 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee76015a0 0x55fee7602a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 4 client.15429 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fee755bc00 con 0x55fee76015a0 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.790+0000 7f197b7fe640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee76015a0 0x55fee7602a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f195affd640 10 client.15429 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f197b7fe640 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee76015a0 0x55fee7602a10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=efb0303f71a238da server_cookie=f5d9518cc5908813 in_seq=0 out_seq=0 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f195affd640 1 -- 172.21.15.69:0/1774617709 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fee755bc00 con 0x55fee76015a0 2024-05-09T09:18:39.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f195affd640 10 client.15429 handle_command_reply: tid=0 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 client.15429 shutdown 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 msgr2=0x55fee75fbed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f1974019ef0 tx=0x7f1974004b50 comp rx=0 tx=0).stop 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 shutdown_connections 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75fbed0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.791+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee76015a0 0x55fee7602a10 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75f4e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f195affd640 0 client.15429 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 1 --2- 172.21.15.69:0/1774617709 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75f4950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 >> 172.21.15.69:0/1774617709 conn(0x55fee75e5f30 msgr2=0x55fee75e5000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 shutdown_connections 2024-05-09T09:18:39.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 1 -- 172.21.15.69:0/1774617709 wait complete. 2024-05-09T09:18:39.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.792+0000 7f19849b3740 20 client.15429 trim_cache size 0 max 16384 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.809+0000 7f19849b3740 1 Processor -- start 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.809+0000 7f19849b3740 1 -- start start 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75efce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75fbe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fee7600d50 con 0x55fee75f0420 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fee7601420 con 0x55fee75ea8c0 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19849b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fee7601af0 con 0x55fee75eb2b0 2024-05-09T09:18:39.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43234/0 (socket says 172.21.15.69:43234) 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 -- 172.21.15.69:0/961808293 learned_addr learned my addr 172.21.15.69:0/961808293 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f1973fff640 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75fbe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 -- 172.21.15.69:0/961808293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 msgr2=0x55fee75fbe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75fbe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 -- 172.21.15.69:0/961808293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 msgr2=0x55fee75efce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75efce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 -- 172.21.15.69:0/961808293 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fee75eb690 con 0x55fee75f0420 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f1973fff640 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f197bfff640 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1974019ec0 tx=0x7f1974019b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbc904cda594a1f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19717fa640 1 -- 172.21.15.69:0/961808293 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974022ae0 con 0x55fee75f0420 2024-05-09T09:18:39.816 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19717fa640 1 -- 172.21.15.69:0/961808293 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1974036070 con 0x55fee75f0420 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.810+0000 7f19717fa640 1 -- 172.21.15.69:0/961808293 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1974031690 con 0x55fee75f0420 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- 172.21.15.69:0/961808293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 msgr2=0x55fee75f3c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f1974019ec0 tx=0x7f1974019b50 comp rx=0 tx=0).stop 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- 172.21.15.69:0/961808293 shutdown_connections 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75f0420 0x55fee75f3c20 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75eb2b0 0x55fee75fbe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 --2- 172.21.15.69:0/961808293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75ea8c0 0x55fee75efce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- 172.21.15.69:0/961808293 >> 172.21.15.69:0/961808293 conn(0x55fee75e5f30 msgr2=0x55fee75ec6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- 172.21.15.69:0/961808293 shutdown_connections 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- 172.21.15.69:0/961808293 wait complete. 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 -- start start 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.811+0000 7f19849b3740 1 Processor -- start 2024-05-09T09:18:39.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75ea8c0 0x55fee75fa0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 0x55fee75fac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fee7602890 con 0x55fee75eb2b0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fee76021c0 con 0x55fee75f0420 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fee7602f60 con 0x55fee75ea8c0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 10 client.0 _ll_register_callbacks cb 0x7f1974009d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43242/0 (socket says 172.21.15.69:43242) 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 -- 172.21.15.69:0/2838269068 learned_addr learned my addr 172.21.15.69:0/2838269068 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75ea8c0 0x55fee75fa0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f197bfff640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 0x55fee75fac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 -- 172.21.15.69:0/2838269068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75ea8c0 msgr2=0x55fee75fa0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75ea8c0 0x55fee75fa0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 -- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 msgr2=0x55fee75fac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 0x55fee75fac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 -- 172.21.15.69:0/2838269068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fee75fb470 con 0x55fee75eb2b0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f197bfff640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 0x55fee75fac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f1973fff640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f196400f6a0 tx=0x7f196400fb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b4fadac6ae83980 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f195affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1964004020 con 0x55fee75eb2b0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f19640041c0 con 0x55fee75eb2b0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.812+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1974036070 con 0x55fee75eb2b0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=24) ==== 24+0+0 (secure 0 0 0) 0x7f1964026050 con 0x55fee75eb2b0 2024-05-09T09:18:39.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 10 client.15435 fetch_fsmap learned FSMap version 24 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=24,monmap=4+}) -- 0x55fee76678d0 con 0x55fee75eb2b0 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1974036070 con 0x55fee75eb2b0 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 24) ==== 8346+0+0 (secure 0 0 0) 0x7f1964003b30 con 0x55fee75eb2b0 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 10 client.15435 fetch_fsmap finished waiting for FSMap version 24 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 10 client.15435 resolve_mds: resolved name 'b' to daemon mds.b 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 20 client.15435 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee766c560 0x55fee766b270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 4 client.15435 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "status"}] 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "status"}) -- 0x55fee7555f50 con 0x55fee766c560 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee766c560 0x55fee766b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f195affd640 10 client.15435 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:39.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.813+0000 7f197b7fe640 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee766c560 0x55fee766b270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63365c3efeec9539 server_cookie=e181d906219a872f in_seq=0 out_seq=0 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f195affd640 1 -- 172.21.15.69:0/2838269068 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+332 (crc 0 0 0) 0x55fee7555f50 con 0x55fee766c560 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f195affd640 10 client.15435 handle_command_reply: tid=0 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 client.15435 shutdown 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 msgr2=0x55fee75fa610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f196400f6a0 tx=0x7f196400fb60 comp rx=0 tx=0).stop 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 shutdown_connections 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fee75f0420 0x55fee75fac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fee75eb2b0 0x55fee75fa610 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fee766c560 0x55fee766b270 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 --2- 172.21.15.69:0/2838269068 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fee75ea8c0 0x55fee75fa0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f195affd640 0 client.15435 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:39.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 >> 172.21.15.69:0/2838269068 conn(0x55fee75e5f30 msgr2=0x55fee75e8ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 shutdown_connections 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 1 -- 172.21.15.69:0/2838269068 wait complete. 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f19849b3740 20 client.15435 trim_cache size 0 max 16384 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout: "cluster_fsid": "8e3a7b52-0de3-11ef-bc98-c7b262605968", 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout: "whoami": 0, 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 24527, 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout: "want_state": "up:active", 2024-05-09T09:18:39.822 INFO:teuthology.orchestra.run.smithi069.stdout: "state": "up:active", 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "fs_name": "cephfs", 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "rank_uptime": 126.95572438799999, 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "mdsmap_epoch": 24, 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch": 93, 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_epoch_barrier": 89, 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout: "uptime": 133.07313503200001 2024-05-09T09:18:39.823 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 msgr2=0x7f19680a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 0x7f19680a5050 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x55fee75ebb10 tx=0x7f1974009650 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.814+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 msgr2=0x7f197c1025a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f196c031610 tx=0x7f196c0043b0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 shutdown_connections 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f19680a2b90 0x7f19680a5050 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f197c102ea0 0x7f197c101120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f197c1002d0 0x7f197c100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 --2- 172.21.15.69:0/3141264227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f197c0ff910 0x7f197c1025a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 >> 172.21.15.69:0/3141264227 conn(0x7f197c0fb470 msgr2=0x7f197c105fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:39.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 shutdown_connections 2024-05-09T09:18:39.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:39.815+0000 7f1981ca5640 1 -- 172.21.15.69:0/3141264227 wait complete. 2024-05-09T09:18:39.826 INFO:teuthology.run_tasks:Running task sequential... 2024-05-09T09:18:39.836 INFO:teuthology.task.sequential:In sequential, running task sequential... 2024-05-09T09:18:39.836 INFO:teuthology.task.sequential:In sequential, running task print... 2024-05-09T09:18:39.836 INFO:teuthology.task.print:Enabling mgr modules 2024-05-09T09:18:39.836 INFO:teuthology.run_tasks:Running task kclient... 2024-05-09T09:18:39.849 INFO:tasks.kclient:Mounting kernel clients... 2024-05-09T09:18:39.849 INFO:tasks.kclient:config is {'client.0': {}} 2024-05-09T09:18:39.850 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:39.850 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-05-09T09:18:39.866 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-05-09T09:18:39.867 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:39.867 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-05-09T09:18:39.938 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:39.938 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-05-09T09:18:40.034 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'wsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-05-09T09:18:40.034 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-05-09T09:18:40.034 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:40.034 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph fs ls 2024-05-09T09:18:40.276 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:18:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:40 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3566377378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:40 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3566377378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:40 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3566377378' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:41.340 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:41 smithi069 ceph-mon[30123]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-09T09:18:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:41 smithi163 ceph-mon[34354]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-09T09:18:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:41 smithi103 ceph-mon[34421]: pgmap v359: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-05-09T09:18:41.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 -- 172.21.15.69:0/2575018840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 msgr2=0x7f3014107670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:41.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 --2- 172.21.15.69:0/2575018840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f3014107670 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f300c00b0a0 tx=0x7f300c00b3b0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 -- 172.21.15.69:0/2575018840 shutdown_connections 2024-05-09T09:18:41.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 --2- 172.21.15.69:0/2575018840 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 0x7f301410e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 --2- 172.21.15.69:0/2575018840 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f3014107670 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 --2- 172.21.15.69:0/2575018840 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 0x7f30141016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 -- 172.21.15.69:0/2575018840 >> 172.21.15.69:0/2575018840 conn(0x7f30140781d0 msgr2=0x7f30140785e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 -- 172.21.15.69:0/2575018840 shutdown_connections 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.872+0000 7f301bfc5700 1 -- 172.21.15.69:0/2575018840 wait complete. 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 Processor -- start 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 -- start start 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 0x7f30141999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 0x7f3014199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3014111580 con 0x7f3014107290 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3014111400 con 0x7f3014101220 2024-05-09T09:18:41.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301bfc5700 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3014111700 con 0x7f3014101be0 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301a562700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301a562700 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43264/0 (socket says 172.21.15.69:43264) 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f301a562700 1 -- 172.21.15.69:0/1018949136 learned_addr learned my addr 172.21.15.69:0/1018949136 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f3019560700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 0x7f3014199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.873+0000 7f3019d61700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 0x7f30141999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 -- 172.21.15.69:0/1018949136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 msgr2=0x7f3014199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 0x7f3014199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 -- 172.21.15.69:0/1018949136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 msgr2=0x7f30141999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 0x7f30141999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 -- 172.21.15.69:0/1018949136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f30141a4f60 con 0x7f3014107290 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301a562700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f300400d980 tx=0x7f300400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004017070 con 0x7f3014107290 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f30141a5190 con 0x7f3014107290 2024-05-09T09:18:41.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3004005020 con 0x7f3014107290 2024-05-09T09:18:41.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3004009b50 con 0x7f3014107290 2024-05-09T09:18:41.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.874+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f30141a56f0 con 0x7f3014107290 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.875+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3014069010 con 0x7f3014107290 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.876+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3004009cf0 con 0x7f3014107290 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.877+0000 7f3012ffd700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 0x7f2ffc0a50e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.877+0000 7f3019d61700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 0x7f2ffc0a50e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.877+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f30040cf1e0 con 0x7f3014107290 2024-05-09T09:18:41.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.877+0000 7f3019d61700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 0x7f2ffc0a50e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f300c006130 tx=0x7f300c0060a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:41.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:41.881+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3004020020 con 0x7f3014107290 2024-05-09T09:18:42.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.123+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs ls"} v 0) -- 0x7f301402db80 con 0x7f3014107290 2024-05-09T09:18:42.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.123+0000 7f3012ffd700 1 -- 172.21.15.69:0/1018949136 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v24) ==== 53+0+88 (secure 0 0 0) 0x7f300409bdc0 con 0x7f3014107290 2024-05-09T09:18:42.122 INFO:teuthology.orchestra.run.smithi069.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-05-09T09:18:42.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 msgr2=0x7f2ffc0a50e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:42.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 0x7f2ffc0a50e0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f300c006130 tx=0x7f300c0060a0 comp rx=0 tx=0).stop 2024-05-09T09:18:42.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 msgr2=0x7f30141a4a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:42.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f300400d980 tx=0x7f300400dc90 comp rx=0 tx=0).stop 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 shutdown_connections 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3014107290 0x7f30141a4a20 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2ffc0a2c20 0x7f2ffc0a50e0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3014101220 0x7f30141999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 --2- 172.21.15.69:0/1018949136 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3014101be0 0x7f3014199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 >> 172.21.15.69:0/1018949136 conn(0x7f30140781d0 msgr2=0x7f30140fff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 shutdown_connections 2024-05-09T09:18:42.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42.125+0000 7f301bfc5700 1 -- 172.21.15.69:0/1018949136 wait complete. 2024-05-09T09:18:42.383 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:42 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1018949136' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:18:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:42 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1018949136' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:18:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:42 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1018949136' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-05-09T09:18:42.771 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi069.front.sepia.ceph.com 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-05-09T09:18:42.772 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-05-09T09:18:42.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:42.772 DEBUG:teuthology.orchestra.run.smithi069:> ip addr 2024-05-09T09:18:42.790 INFO:teuthology.orchestra.run.smithi069.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: inet 127.0.0.1/8 scope host lo 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 ::1/128 scope host 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c2 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f0 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:d8:d9:c3 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: altname enp7s0f1 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-05-09T09:18:42.791 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:36 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout: link/ether 0c:c4:7a:8f:d7:37 brd ff:ff:ff:ff:ff:ff 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout: inet 172.21.15.69/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft 42092sec preferred_lft 42092sec 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout: inet6 fe80::ec4:7aff:fe8f:d737/64 scope link 2024-05-09T09:18:42.792 INFO:teuthology.orchestra.run.smithi069.stdout: valid_lft forever preferred_lft forever 2024-05-09T09:18:42.792 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-05-09T09:18:42.792 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:18:42.792 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:18:42.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add name ceph-brx type bridge 2024-05-09T09:18:42.793 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr flush dev ceph-brx 2024-05-09T09:18:42.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set ceph-brx up 2024-05-09T09:18:42.794 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-05-09T09:18:42.794 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:18:42.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:42.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:42 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:42.932 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:42.932 DEBUG:teuthology.orchestra.run.smithi069:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-05-09T09:18:42.958 INFO:teuthology.orchestra.run.smithi069.stdout:1 2024-05-09T09:18:42.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:42.959 DEBUG:teuthology.orchestra.run.smithi069:> route 2024-05-09T09:18:43.017 INFO:teuthology.orchestra.run.smithi069.stdout:Kernel IP routing table 2024-05-09T09:18:43.017 INFO:teuthology.orchestra.run.smithi069.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-05-09T09:18:43.017 INFO:teuthology.orchestra.run.smithi069.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-05-09T09:18:43.017 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-05-09T09:18:43.018 INFO:teuthology.orchestra.run.smithi069.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-05-09T09:18:43.018 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:18:43.106 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:43.139 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:43.145 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:43.145 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-05-09T09:18:43.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:43.161 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list-id 2024-05-09T09:18:43.217 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:18:43.217 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:18:43.217 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T09:18:43.217 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-05-09T09:18:43.217 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:18:43.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:43.329 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:43.335 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-05-09T09:18:43.336 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:18:43.336 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:18:43.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-05-09T09:18:43.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-05-09T09:18:43.336 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-05-09T09:18:43.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-05-09T09:18:43.337 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-05-09T09:18:43.337 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:18:43.386 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:43.394 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:43 smithi069 ceph-mon[30123]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:43.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:43.452 DEBUG:teuthology.orchestra.run.smithi069:> (cd / && exec stdin-killer --timeout=300 -- bash -c ' 2024-05-09T09:18:43.452 DEBUG:teuthology.orchestra.run.smithi069:> set -e 2024-05-09T09:18:43.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set brx.0 up 2024-05-09T09:18:43.452 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link set dev brx.0 master ceph-brx 2024-05-09T09:18:43.452 DEBUG:teuthology.orchestra.run.smithi069:> ') 2024-05-09T09:18:43.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:43.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:43.529 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T09:18:43.530 DEBUG:teuthology.orchestra.run.smithi069:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:43.545 INFO:teuthology.orchestra.run.smithi069.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-05-09T09:18:43.545 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-05-09T09:18:43.545 DEBUG:teuthology.orchestra.run.smithi069:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:43 smithi163 ceph-mon[34354]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:43.601 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-05-09T09:18:43.601 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-05-09T09:18:43.601 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d 2024-05-09T09:18:43.601 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:43.601 DEBUG:teuthology.orchestra.run.smithi069:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T09:18:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:43 smithi103 ceph-mon[34421]: pgmap v360: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-05-09T09:18:43.684 INFO:teuthology.orchestra.run.smithi069.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T09:18:43.685 INFO:teuthology.orchestra.run.smithi069.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-09T09:18:43.685 INFO:teuthology.orchestra.run.smithi069.stdout:invalid new device string format 2024-05-09T09:18:43.816 INFO:teuthology.orchestra.run.smithi069.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync 2024-05-09T09:18:43.816 INFO:teuthology.orchestra.run.smithi069.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync". 2024-05-09T09:18:43.816 INFO:teuthology.orchestra.run.smithi069.stdout:invalid new device string format 2024-05-09T09:18:43.817 INFO:teuthology.orchestra.run.smithi069.stdout:mount.ceph: resolved to: "172.21.15.69:6789,172.21.15.103:6789,172.21.15.163:6789" 2024-05-09T09:18:43.817 INFO:teuthology.orchestra.run.smithi069.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.69:6789,172.21.15.103:6789,172.21.15.163:6789:/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d 2024-05-09T09:18:43.817 INFO:teuthology.orchestra.run.smithi069.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,wsync,key=0,fsid=8e3a7b52-0de3-11ef-bc98-c7b262605968" will pass to kernel 2024-05-09T09:18:43.818 INFO:teuthology.orchestra.run.smithi069.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-05-09T09:18:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: You just mounted a file system that supports labels which does not 2024-05-09T09:18:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-05-09T09:18:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: applications will generate AVC messages and not be allowed access to 2024-05-09T09:18:43.818 INFO:teuthology.orchestra.run.smithi069.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-05-09T09:18:43.819 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-05-09T09:18:43.819 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:43.819 DEBUG:teuthology.orchestra.run.smithi069:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:43.888 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T09:18:43.978 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T09:18:43.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:43 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:43.986 INFO:teuthology.orchestra.run.smithi069.stdout:client15444 2024-05-09T09:18:43.986 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:18:43.986 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:18:44.004 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T09:18:44.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T09:18:44.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:44.100 INFO:teuthology.orchestra.run.smithi069.stdout:client15444 2024-05-09T09:18:44.100 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=900 -- bash -c 'getfattr --only-values -n ceph.client_id .') 2024-05-09T09:18:44.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: expiration expected; waiting 900 seconds for command to complete 2024-05-09T09:18:44.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:44.156 INFO:teuthology.orchestra.run.smithi069.stdout:client15444 2024-05-09T09:18:44.156 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest/mnt.0 && exec stdin-killer --timeout=300 -- bash -c 'sudo dd if=/sys/kernel/debug/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968.client15444/status') 2024-05-09T09:18:44.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: expiration expected; waiting 300 seconds for command to complete 2024-05-09T09:18:44.217 INFO:teuthology.orchestra.run.smithi069.stdout:instance: client.15444 (3)192.168.0.1:0/350423126 2024-05-09T09:18:44.217 INFO:teuthology.orchestra.run.smithi069.stdout:blocklisted: false 2024-05-09T09:18:44.217 INFO:teuthology.orchestra.run.smithi069.stderr:0+1 records in 2024-05-09T09:18:44.218 INFO:teuthology.orchestra.run.smithi069.stderr:0+1 records out 2024-05-09T09:18:44.218 INFO:teuthology.orchestra.run.smithi069.stderr:69 bytes copied, 2.325e-05 s, 3.0 MB/s 2024-05-09T09:18:44.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-05-09T09:18:44.223 INFO:teuthology.run_tasks:Running task exec... 2024-05-09T09:18:44.236 INFO:teuthology.task.exec:Executing custom commands... 2024-05-09T09:18:44.236 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:18:44.236 DEBUG:teuthology.orchestra.run.smithi069:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate true' 2024-05-09T09:18:44.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 -- 172.21.15.69:0/535898861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c102a80 msgr2=0x7f4a0c102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 --2- 172.21.15.69:0/535898861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c102a80 0x7f4a0c102f00 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7f4a00009900 tx=0x7f4a0002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 -- 172.21.15.69:0/535898861 shutdown_connections 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 --2- 172.21.15.69:0/535898861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c103440 0x7f4a0c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 --2- 172.21.15.69:0/535898861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c102a80 0x7f4a0c102f00 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 --2- 172.21.15.69:0/535898861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 0x7f4a0c101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 -- 172.21.15.69:0/535898861 >> 172.21.15.69:0/535898861 conn(0x7f4a0c0fce20 msgr2=0x7f4a0c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 -- 172.21.15.69:0/535898861 shutdown_connections 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.340+0000 7f4a124a5640 1 -- 172.21.15.69:0/535898861 wait complete. 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 Processor -- start 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 -- start start 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 0x7f4a0c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c102a80 0x7f4a0c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.342 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10d510 con 0x7f4a0c103440 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10d390 con 0x7f4a0c102a80 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a124a5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a0c10d690 con 0x7f4a0c101890 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a10a42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a10a42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43274/0 (socket says 172.21.15.69:43274) 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.341+0000 7f4a10a42640 1 -- 172.21.15.69:0/4238709896 learned_addr learned my addr 172.21.15.69:0/4238709896 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a0bfff640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 0x7f4a0c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a0b7fe640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c102a80 0x7f4a0c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 -- 172.21.15.69:0/4238709896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 msgr2=0x7f4a0c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 0x7f4a0c197a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 -- 172.21.15.69:0/4238709896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c102a80 msgr2=0x7f4a0c197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c102a80 0x7f4a0c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 -- 172.21.15.69:0/4238709896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a0c19f680 con 0x7f4a0c103440 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a10a42640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400cce0 tx=0x7f4a0400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a0400f450 con 0x7f4a0c103440 2024-05-09T09:18:44.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a0400fad0 con 0x7f4a0c103440 2024-05-09T09:18:44.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a0c19f970 con 0x7f4a0c103440 2024-05-09T09:18:44.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a04018c10 con 0x7f4a0c103440 2024-05-09T09:18:44.344 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.342+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a0c19fe70 con 0x7f4a0c103440 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.343+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4a0c108ae0 con 0x7f4a0c103440 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.344+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4a0400f5f0 con 0x7f4a0c103440 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.345+0000 7f4a097fa640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 0x7f49f40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.345+0000 7f4a0bfff640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 0x7f49f40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.345+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f4a040cfab0 con 0x7f4a0c103440 2024-05-09T09:18:44.345 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.346+0000 7f4a0bfff640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 0x7f49f40a50f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f49fc005fd0 tx=0x7f49fc00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:44.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.350+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4a04014030 con 0x7f4a0c103440 2024-05-09T09:18:44.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.567+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"} v 0) -- 0x7f4a0c062430 con 0x7f4a0c103440 2024-05-09T09:18:44.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.619+0000 7f4a097fa640 1 -- 172.21.15.69:0/4238709896 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]=0 v25) ==== 117+0+0 (secure 0 0 0) 0x7f4a0409c690 con 0x7f4a0c103440 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 msgr2=0x7f49f40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 0x7f49f40a50f0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f49fc005fd0 tx=0x7f49fc00b390 comp rx=0 tx=0).stop 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 msgr2=0x7f4a0c19f010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7f4a0400cce0 tx=0x7f4a0400a620 comp rx=0 tx=0).stop 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 shutdown_connections 2024-05-09T09:18:44.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f49f40a2c30 0x7f49f40a50f0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a0c103440 0x7f4a0c19f010 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a0c102a80 0x7f4a0c197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 --2- 172.21.15.69:0/4238709896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a0c101890 0x7f4a0c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 >> 172.21.15.69:0/4238709896 conn(0x7f4a0c0fce20 msgr2=0x7f4a0c105cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 shutdown_connections 2024-05-09T09:18:44.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.621+0000 7f4a124a5640 1 -- 172.21.15.69:0/4238709896 wait complete. 2024-05-09T09:18:44.632 INFO:teuthology.run_tasks:Running task check-counter... 2024-05-09T09:18:44.645 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-05-09T09:18:44.712 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:44.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 -- 172.21.15.69:0/302462561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd241034c0 msgr2=0x7fdd24109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 --2- 172.21.15.69:0/302462561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd241034c0 0x7fdd24109d50 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009930 tx=0x7fdd1c02a420 comp rx=0 tx=0).stop 2024-05-09T09:18:44.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 -- 172.21.15.69:0/302462561 shutdown_connections 2024-05-09T09:18:44.817 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 --2- 172.21.15.69:0/302462561 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd241034c0 0x7fdd24109d50 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 --2- 172.21.15.69:0/302462561 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd24102b00 0x7fdd24102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 --2- 172.21.15.69:0/302462561 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd240ffb30 0x7fdd240fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 -- 172.21.15.69:0/302462561 >> 172.21.15.69:0/302462561 conn(0x7fdd240fb460 msgr2=0x7fdd240fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.815+0000 7fdd29815640 1 -- 172.21.15.69:0/302462561 shutdown_connections 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 -- 172.21.15.69:0/302462561 wait complete. 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 Processor -- start 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 -- start start 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 0x7fdd24102470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd241034c0 0x7fdd24100ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410d5b0 con 0x7fdd24102b00 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410d430 con 0x7fdd241034c0 2024-05-09T09:18:44.818 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd29815640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fdd2410d730 con 0x7fdd240ffb30 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.816+0000 7fdd227fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43294/0 (socket says 172.21.15.69:43294) 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 -- 172.21.15.69:0/2102483658 learned_addr learned my addr 172.21.15.69:0/2102483658 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd22ffd640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 0x7fdd24102470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd237fe640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd241034c0 0x7fdd24100ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 -- 172.21.15.69:0/2102483658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 msgr2=0x7fdd24102470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 0x7fdd24102470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 -- 172.21.15.69:0/2102483658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd241034c0 msgr2=0x7fdd24100ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd241034c0 0x7fdd24100ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 -- 172.21.15.69:0/2102483658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fdd24101880 con 0x7fdd24102b00 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd22ffd640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 0x7fdd24102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd227fc640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00b4a0 tx=0x7fdd0c00b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c014070 con 0x7fdd24102b00 2024-05-09T09:18:44.819 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fdd0c010af0 con 0x7fdd24102b00 2024-05-09T09:18:44.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fdd241a1ce0 con 0x7fdd24102b00 2024-05-09T09:18:44.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fdd0c017d10 con 0x7fdd24102b00 2024-05-09T09:18:44.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.817+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fdd241a2180 con 0x7fdd24102b00 2024-05-09T09:18:44.820 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.818+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fdd24108bd0 con 0x7fdd24102b00 2024-05-09T09:18:44.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.819+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fdd0c017070 con 0x7fdd24102b00 2024-05-09T09:18:44.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.820+0000 7fdd28813640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:44.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.820+0000 7fdd22ffd640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:44.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.820+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fdd0c0ce8d0 con 0x7fdd24102b00 2024-05-09T09:18:44.821 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.821+0000 7fdd22ffd640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 0x7fdd100a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd240687c0 tx=0x7fdd14009540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:44.823 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:44.825+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fdd0c09b4b0 con 0x7fdd24102b00 2024-05-09T09:18:45.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.005+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7fdd240624d0 con 0x7fdd24102b00 2024-05-09T09:18:45.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.007+0000 7fdd28813640 1 -- 172.21.15.69:0/2102483658 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7fdd0c09adb0 con 0x7fdd24102b00 2024-05-09T09:18:45.006 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:45.006 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:45.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 msgr2=0x7fdd100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 0x7fdd100a5050 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7fdd240687c0 tx=0x7fdd14009540 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 msgr2=0x7fdd24100ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fdd0c00b4a0 tx=0x7fdd0c00b960 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 shutdown_connections 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fdd100a2b90 0x7fdd100a5050 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fdd241034c0 0x7fdd24100ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fdd24102b00 0x7fdd24100ab0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 --2- 172.21.15.69:0/2102483658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fdd240ffb30 0x7fdd24102470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 >> 172.21.15.69:0/2102483658 conn(0x7fdd240fb460 msgr2=0x7fdd24105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 shutdown_connections 2024-05-09T09:18:45.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.009+0000 7fdd29815640 1 -- 172.21.15.69:0/2102483658 wait complete. 2024-05-09T09:18:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:44 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T09:18:45.020 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:45.021 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:44 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T09:18:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:44 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]: dispatch 2024-05-09T09:18:45.127 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 -- 172.21.15.69:0/631828665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40101960 msgr2=0x7f3f40101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 --2- 172.21.15.69:0/631828665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40101960 0x7f3f40101de0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f3f34009900 tx=0x7f3f3402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 -- 172.21.15.69:0/631828665 shutdown_connections 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 --2- 172.21.15.69:0/631828665 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40102320 0x7f3f40108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 --2- 172.21.15.69:0/631828665 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40101960 0x7f3f40101de0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 --2- 172.21.15.69:0/631828665 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 0x7f3f40100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 -- 172.21.15.69:0/631828665 >> 172.21.15.69:0/631828665 conn(0x7f3f400fbce0 msgr2=0x7f3f400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 -- 172.21.15.69:0/631828665 shutdown_connections 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.126+0000 7f3f45370640 1 -- 172.21.15.69:0/631828665 wait complete. 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 Processor -- start 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 -- start start 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 0x7f3f40069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40101960 0x7f3f4019c910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.128 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4010c3f0 con 0x7f3f40102320 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4010c270 con 0x7f3f40101960 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f45370640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3f4010c570 con 0x7f3f40100770 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f3f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43306/0 (socket says 172.21.15.69:43306) 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.127+0000 7f3f3f7fe640 1 -- 172.21.15.69:0/3029902295 learned_addr learned my addr 172.21.15.69:0/3029902295 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3effd640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 0x7f3f40069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3e7fc640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40101960 0x7f3f4019c910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 -- 172.21.15.69:0/3029902295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 msgr2=0x7f3f40069550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 0x7f3f40069550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 -- 172.21.15.69:0/3029902295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40101960 msgr2=0x7f3f4019c910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40101960 0x7f3f4019c910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 -- 172.21.15.69:0/3029902295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3f4019f810 con 0x7f3f40102320 2024-05-09T09:18:45.129 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f3f7fe640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d750 tx=0x7f3f3800dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38017070 con 0x7f3f40102320 2024-05-09T09:18:45.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3f38005500 con 0x7f3f40102320 2024-05-09T09:18:45.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3f4019fb00 con 0x7f3f40102320 2024-05-09T09:18:45.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3f38004d90 con 0x7f3f40102320 2024-05-09T09:18:45.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.128+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3f4019ffd0 con 0x7f3f40102320 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.129+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3f40107b80 con 0x7f3f40102320 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.130+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3f38004f30 con 0x7f3f40102320 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.131+0000 7f3f27fff640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 0x7f3f280a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.131+0000 7f3f3effd640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 0x7f3f280a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.131+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3f380ced30 con 0x7f3f40102320 2024-05-09T09:18:45.131 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.131+0000 7f3f3effd640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 0x7f3f280a5050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f30005c40 tx=0x7f3f3000a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.134 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.135+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3f3809b2c0 con 0x7f3f40102320 2024-05-09T09:18:45.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.312+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f3f40062430 con 0x7f3f40102320 2024-05-09T09:18:45.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.312+0000 7f3f27fff640 1 -- 172.21.15.69:0/3029902295 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f3f3809f5c0 con 0x7f3f40102320 2024-05-09T09:18:45.311 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 msgr2=0x7f3f280a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 0x7f3f280a5050 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f3f30005c40 tx=0x7f3f3000a590 comp rx=0 tx=0).stop 2024-05-09T09:18:45.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 msgr2=0x7f3f4019ef50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f3f3800d750 tx=0x7f3f3800dc10 comp rx=0 tx=0).stop 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 shutdown_connections 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3f280a2b90 0x7f3f280a5050 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3f40102320 0x7f3f4019ef50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3f40101960 0x7f3f4019c910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 --2- 172.21.15.69:0/3029902295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3f40100770 0x7f3f40069550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 >> 172.21.15.69:0/3029902295 conn(0x7f3f400fbce0 msgr2=0x7f3f40104d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 shutdown_connections 2024-05-09T09:18:45.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.314+0000 7f3f45370640 1 -- 172.21.15.69:0/3029902295 wait complete. 2024-05-09T09:18:45.325 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/32789852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8101880 msgr2=0x7fb2c8101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/32789852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8101880 0x7fb2c8101ca0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8009900 tx=0x7fb2b802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/32789852 shutdown_connections 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/32789852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8103430 0x7fb2c8109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/32789852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8102a70 0x7fb2c8102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/32789852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8101880 0x7fb2c8101ca0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/32789852 >> 172.21.15.69:0/32789852 conn(0x7fb2c80fce10 msgr2=0x7fb2c80ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/32789852 shutdown_connections 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/32789852 wait complete. 2024-05-09T09:18:45.432 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 Processor -- start 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- start start 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8101880 0x7fb2c8197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8103430 0x7fb2c819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c810d4f0 con 0x7fb2c8102a70 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c810d370 con 0x7fb2c8101880 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.431+0000 7fb2ce4ee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2c810d670 con 0x7fb2c8103430 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43316/0 (socket says 172.21.15.69:43316) 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 -- 172.21.15.69:0/3999298034 learned_addr learned my addr 172.21.15.69:0/3999298034 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2cca8b640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8103430 0x7fb2c819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2c7fff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8101880 0x7fb2c8197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 -- 172.21.15.69:0/3999298034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8103430 msgr2=0x7fb2c819f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8103430 0x7fb2c819f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 -- 172.21.15.69:0/3999298034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8101880 msgr2=0x7fb2c8197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8101880 0x7fb2c8197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 -- 172.21.15.69:0/3999298034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2c819f7e0 con 0x7fb2c8102a70 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2bffff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000e9f0 tx=0x7fb2b000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b000cd80 con 0x7fb2c8102a70 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2b0005500 con 0x7fb2c8102a70 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2c819fad0 con 0x7fb2c8102a70 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2b0010690 con 0x7fb2c8102a70 2024-05-09T09:18:45.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.432+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2c81a0020 con 0x7fb2c8102a70 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.433+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2c8108bf0 con 0x7fb2c8102a70 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.434+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb2b00046c0 con 0x7fb2c8102a70 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.435+0000 7fb2c5ffb640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 0x7fb2b40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.435+0000 7fb2c7fff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 0x7fb2b40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.435+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fb2b00cf740 con 0x7fb2c8102a70 2024-05-09T09:18:45.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.435+0000 7fb2c7fff640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 0x7fb2b40a50d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8005ab0 tx=0x7fb2b8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.439+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb2b009c320 con 0x7fb2c8102a70 2024-05-09T09:18:45.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.614+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7fb2c8062430 con 0x7fb2c8102a70 2024-05-09T09:18:45.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.614+0000 7fb2c5ffb640 1 -- 172.21.15.69:0/3999298034 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7fb2b00860a0 con 0x7fb2c8102a70 2024-05-09T09:18:45.613 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 msgr2=0x7fb2b40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 0x7fb2b40a50d0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fb2b8005ab0 tx=0x7fb2b8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 msgr2=0x7fb2c8198100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7fb2b000e9f0 tx=0x7fb2b000edf0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 shutdown_connections 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2b40a2c10 0x7fb2b40a50d0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2c8103430 0x7fb2c819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2c8102a70 0x7fb2c8198100 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 --2- 172.21.15.69:0/3999298034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2c8101880 0x7fb2c8197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.615 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 >> 172.21.15.69:0/3999298034 conn(0x7fb2c80fce10 msgr2=0x7fb2c8105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 shutdown_connections 2024-05-09T09:18:45.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.616+0000 7fb2ce4ee640 1 -- 172.21.15.69:0/3999298034 wait complete. 2024-05-09T09:18:45.626 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 -- 172.21.15.69:0/2367813140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84102340 msgr2=0x7ffa84108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/2367813140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84102340 0x7ffa84108bd0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7ffa7c009900 tx=0x7ffa7c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 -- 172.21.15.69:0/2367813140 shutdown_connections 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/2367813140 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84102340 0x7ffa84108bd0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/2367813140 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa84101980 0x7ffa84101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/2367813140 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84100790 0x7ffa84100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 -- 172.21.15.69:0/2367813140 >> 172.21.15.69:0/2367813140 conn(0x7ffa840fbd20 msgr2=0x7ffa840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 -- 172.21.15.69:0/2367813140 shutdown_connections 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.731+0000 7ffa88b7e640 1 -- 172.21.15.69:0/2367813140 wait complete. 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 Processor -- start 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 -- start start 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 0x7ffa84196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa84102340 0x7ffa8419dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8410c410 con 0x7ffa84100790 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8410c290 con 0x7ffa84101980 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa88b7e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa8410c590 con 0x7ffa84102340 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43338/0 (socket says 172.21.15.69:43338) 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 -- 172.21.15.69:0/554257088 learned_addr learned my addr 172.21.15.69:0/554257088 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa81d9b640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 0x7ffa84196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 -- 172.21.15.69:0/554257088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa84102340 msgr2=0x7ffa8419dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa84102340 0x7ffa8419dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 -- 172.21.15.69:0/554257088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 msgr2=0x7ffa84196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 0x7ffa84196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 -- 172.21.15.69:0/554257088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa8419e650 con 0x7ffa84100790 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa81d9b640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 0x7ffa84196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa8259c640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400ccb0 tx=0x7ffa7400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.732+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa74003990 con 0x7ffa84100790 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.733+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa74003b30 con 0x7ffa84100790 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.733+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa8419e940 con 0x7ffa84100790 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.733+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa7400f7b0 con 0x7ffa84100790 2024-05-09T09:18:45.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.733+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa8419ee90 con 0x7ffa84100790 2024-05-09T09:18:45.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.734+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7ffa84107ab0 con 0x7ffa84100790 2024-05-09T09:18:45.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.735+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ffa7400f950 con 0x7ffa84100790 2024-05-09T09:18:45.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.736+0000 7ffa7b7fe640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:45.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.736+0000 7ffa81d9b640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:45.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.736+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7ffa740cf560 con 0x7ffa84100790 2024-05-09T09:18:45.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.736+0000 7ffa81d9b640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 0x7ffa700a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0058c0 tx=0x7ffa6c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:45.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.740+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7ffa74014030 con 0x7ffa84100790 2024-05-09T09:18:45.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.917+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7ffa8402d1c0 con 0x7ffa84100790 2024-05-09T09:18:45.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.918+0000 7ffa7b7fe640 1 -- 172.21.15.69:0/554257088 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7ffa8402d1c0 con 0x7ffa84100790 2024-05-09T09:18:45.917 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:45.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 msgr2=0x7ffa700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.919 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 0x7ffa700a5050 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7ffa6c0058c0 tx=0x7ffa6c005850 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 msgr2=0x7ffa84196a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7ffa7400ccb0 tx=0x7ffa7400a620 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 shutdown_connections 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa700a2b90 0x7ffa700a5050 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa84102340 0x7ffa8419dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa84101980 0x7ffa84196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 --2- 172.21.15.69:0/554257088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa84100790 0x7ffa84196a30 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:45.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 >> 172.21.15.69:0/554257088 conn(0x7ffa840fbd20 msgr2=0x7ffa84104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:45.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 shutdown_connections 2024-05-09T09:18:45.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:45.920+0000 7ffa88b7e640 1 -- 172.21.15.69:0/554257088 wait complete. 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2102483658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3029902295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:45.929 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:45 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3999298034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:45.931 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:46.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- 172.21.15.69:0/3815782418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920102320 msgr2=0x7f5920108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- 172.21.15.69:0/3815782418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920102320 0x7f5920108bb0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f5918009ca0 tx=0x7f591802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- 172.21.15.69:0/3815782418 shutdown_connections 2024-05-09T09:18:46.038 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- 172.21.15.69:0/3815782418 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920102320 0x7f5920108bb0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- 172.21.15.69:0/3815782418 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 0x7f5920101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- 172.21.15.69:0/3815782418 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920100770 0x7f5920100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- 172.21.15.69:0/3815782418 >> 172.21.15.69:0/3815782418 conn(0x7f59200fbce0 msgr2=0x7f59200fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- 172.21.15.69:0/3815782418 shutdown_connections 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- 172.21.15.69:0/3815782418 wait complete. 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 Processor -- start 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- start start 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 0x7f59201980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 0x7f592019f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c3f0 con 0x7f5920100770 2024-05-09T09:18:46.039 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c270 con 0x7f5920101960 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5926d5a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f592010c570 con 0x7f5920102320 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5924af6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5924af6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43356/0 (socket says 172.21.15.69:43356) 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.036+0000 7f5924af6640 1 -- 172.21.15.69:0/2244019757 learned_addr learned my addr 172.21.15.69:0/2244019757 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5917fff640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 0x7f59201980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f59252f7640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 0x7f592019f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 -- 172.21.15.69:0/2244019757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 msgr2=0x7f592019f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 0x7f592019f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 -- 172.21.15.69:0/2244019757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 msgr2=0x7f59201980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 0x7f59201980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 -- 172.21.15.69:0/2244019757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f592019f780 con 0x7f5920100770 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f59252f7640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 0x7f592019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5924af6640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f591c00d950 tx=0x7f591c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.040 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c017070 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f591c009c30 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f592019fa70 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f591c00f600 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.037+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f592019ffc0 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.038+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f59201a01f0 con 0x7f5920100770 2024-05-09T09:18:46.041 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.039+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f591c01a070 con 0x7f5920100770 2024-05-09T09:18:46.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.040+0000 7f5915ffb640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 0x7f59080a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.041+0000 7f5917fff640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 0x7f59080a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.041+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f591c0cede0 con 0x7f5920100770 2024-05-09T09:18:46.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.041+0000 7f5917fff640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 0x7f59080a50d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5920199070 tx=0x7f5910005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.045+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f591c009de0 con 0x7f5920100770 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2102483658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3029902295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:45 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3999298034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4238709896' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "true"}]': finished 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2102483658' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: pgmap v361: 129 pgs: 129 active+clean; 643 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3029902295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:45 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3999298034' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.219+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f59201a0940 con 0x7f5920100770 2024-05-09T09:18:46.222 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.220+0000 7f5915ffb640 1 -- 172.21.15.69:0/2244019757 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f591c09b9c0 con 0x7f5920100770 2024-05-09T09:18:46.222 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:46.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 msgr2=0x7f59080a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 0x7f59080a50d0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f5920199070 tx=0x7f5910005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:46.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 msgr2=0x7f5920197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f591c00d950 tx=0x7f591c00de10 comp rx=0 tx=0).stop 2024-05-09T09:18:46.223 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 shutdown_connections 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f59080a2c10 0x7f59080a50d0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5920102320 0x7f592019f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5920101960 0x7f59201980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 --2- 172.21.15.69:0/2244019757 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5920100770 0x7f5920197b60 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 >> 172.21.15.69:0/2244019757 conn(0x7f59200fbce0 msgr2=0x7f59200689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 shutdown_connections 2024-05-09T09:18:46.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.222+0000 7f5926d5a640 1 -- 172.21.15.69:0/2244019757 wait complete. 2024-05-09T09:18:46.233 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:46.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 -- 172.21.15.69:0/2926579440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc102330 msgr2=0x7febfc108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 --2- 172.21.15.69:0/2926579440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc102330 0x7febfc108bc0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7febf4009ca0 tx=0x7febf402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 -- 172.21.15.69:0/2926579440 shutdown_connections 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 --2- 172.21.15.69:0/2926579440 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc102330 0x7febfc108bc0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 --2- 172.21.15.69:0/2926579440 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 0x7febfc101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 --2- 172.21.15.69:0/2926579440 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc100780 0x7febfc100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 -- 172.21.15.69:0/2926579440 >> 172.21.15.69:0/2926579440 conn(0x7febfc0fbd10 msgr2=0x7febfc0fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 -- 172.21.15.69:0/2926579440 shutdown_connections 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.336+0000 7fec02961640 1 -- 172.21.15.69:0/2926579440 wait complete. 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 Processor -- start 2024-05-09T09:18:46.338 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 -- start start 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 0x7febfc1980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc102330 0x7febfc19f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c400 con 0x7febfc100780 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c280 con 0x7febfc102330 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec02961640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7febfc10c580 con 0x7febfc101970 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43378/0 (socket says 172.21.15.69:43378) 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 -- 172.21.15.69:0/909983206 learned_addr learned my addr 172.21.15.69:0/909983206 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7fec00efe640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc102330 0x7febfc19f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfb7fe640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 0x7febfc1980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 -- 172.21.15.69:0/909983206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 msgr2=0x7febfc1980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.339 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 0x7febfc1980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 -- 172.21.15.69:0/909983206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc102330 msgr2=0x7febfc19f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc102330 0x7febfc19f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 -- 172.21.15.69:0/909983206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7febfc19f7b0 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.337+0000 7febfbfff640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7febec00d770 tx=0x7febec00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.338+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec017070 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.338+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7febec009c30 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.338+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7febfc19faa0 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.338+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7febec003dd0 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.338+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7febfc19fff0 con 0x7febfc100780 2024-05-09T09:18:46.340 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.339+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7febfc1a0220 con 0x7febfc100780 2024-05-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.340+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7febec003fb0 con 0x7febfc100780 2024-05-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.340+0000 7febf97fa640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.340+0000 7febfb7fe640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.340+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7febec0cee00 con 0x7febfc100780 2024-05-09T09:18:46.341 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.341+0000 7febfb7fe640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 0x7febe40a5050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7febfc1990a0 tx=0x7febf0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.345+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7febec09b9e0 con 0x7febfc100780 2024-05-09T09:18:46.521 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.522+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7febfc1a0970 con 0x7febfc100780 2024-05-09T09:18:46.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.524+0000 7febf97fa640 1 -- 172.21.15.69:0/909983206 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7febec009de0 con 0x7febfc100780 2024-05-09T09:18:46.523 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:46.523 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:46.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 msgr2=0x7febe40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 0x7febe40a5050 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7febfc1990a0 tx=0x7febf0005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:46.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 msgr2=0x7febfc197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7febec00d770 tx=0x7febec00dc30 comp rx=0 tx=0).stop 2024-05-09T09:18:46.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 shutdown_connections 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7febe40a2b90 0x7febe40a5050 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7febfc102330 0x7febfc19f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7febfc101970 0x7febfc1980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 --2- 172.21.15.69:0/909983206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7febfc100780 0x7febfc197b90 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 >> 172.21.15.69:0/909983206 conn(0x7febfc0fbd10 msgr2=0x7febfc0fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 shutdown_connections 2024-05-09T09:18:46.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.526+0000 7fec02961640 1 -- 172.21.15.69:0/909983206 wait complete. 2024-05-09T09:18:46.537 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:46.537 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 -- 172.21.15.69:0/839197035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101d10 msgr2=0x7f35f0102130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/839197035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101d10 0x7f35f0102130 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f35e0009900 tx=0x7f35e002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 -- 172.21.15.69:0/839197035 shutdown_connections 2024-05-09T09:18:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/839197035 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101400 0x7f35f0108ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/839197035 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 0x7f35f0100ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/839197035 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101d10 0x7f35f0102130 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 -- 172.21.15.69:0/839197035 >> 172.21.15.69:0/839197035 conn(0x7f35f00fb470 msgr2=0x7f35f00fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 -- 172.21.15.69:0/839197035 shutdown_connections 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 -- 172.21.15.69:0/839197035 wait complete. 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.643+0000 7f35f5a7d640 1 Processor -- start 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 -- start start 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 0x7f35f01004c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 0x7f35f019efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010c460 con 0x7f35f0101400 2024-05-09T09:18:46.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010c2e0 con 0x7f35f0100a40 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35f010c5e0 con 0x7f35f0101d10 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43396/0 (socket says 172.21.15.69:43396) 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 -- 172.21.15.69:0/3087688646 learned_addr learned my addr 172.21.15.69:0/3087688646 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35ef7fe640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 0x7f35f01004c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35effff640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 0x7f35f019efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 -- 172.21.15.69:0/3087688646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 msgr2=0x7f35f019efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 0x7f35f019efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 -- 172.21.15.69:0/3087688646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 msgr2=0x7f35f01004c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.649 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 0x7f35f01004c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 -- 172.21.15.69:0/3087688646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35f019f880 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35effff640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 0x7f35f019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35e7fff640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f35d800e9f0 tx=0x7f35d800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d800cd80 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35d8010910 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35f019fb70 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.644+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35d8018990 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.645+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35f01a0110 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.646+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f35f0107c70 con 0x7f35f0101400 2024-05-09T09:18:46.650 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.646+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f35d8010ac0 con 0x7f35f0101400 2024-05-09T09:18:46.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.647+0000 7f35ed7fa640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 0x7f35dc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.647+0000 7f35ef7fe640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 0x7f35dc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.647+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f35d80cf740 con 0x7f35f0101400 2024-05-09T09:18:46.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.648+0000 7f35ef7fe640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 0x7f35dc0a50d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35e0005ab0 tx=0x7f35e0000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.651 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.651+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f35d809c2c0 con 0x7f35f0101400 2024-05-09T09:18:46.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.827+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f35f0062430 con 0x7f35f0101400 2024-05-09T09:18:46.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.827+0000 7f35ed7fa640 1 -- 172.21.15.69:0/3087688646 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f35d809bbc0 con 0x7f35f0101400 2024-05-09T09:18:46.826 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:46.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 msgr2=0x7f35dc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 0x7f35dc0a50d0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f35e0005ab0 tx=0x7f35e0000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 msgr2=0x7f35f019c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f35d800e9f0 tx=0x7f35d800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 shutdown_connections 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35dc0a2c10 0x7f35dc0a50d0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35f0101d10 0x7f35f019efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35f0101400 0x7f35f019c980 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 --2- 172.21.15.69:0/3087688646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35f0100a40 0x7f35f01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 >> 172.21.15.69:0/3087688646 conn(0x7f35f00fb470 msgr2=0x7f35f0104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 shutdown_connections 2024-05-09T09:18:46.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.829+0000 7f35f5a7d640 1 -- 172.21.15.69:0/3087688646 wait complete. 2024-05-09T09:18:46.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/554257088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2244019757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:46.839 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:46 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/909983206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:46.840 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 -- 172.21.15.69:0/1307689851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98101880 msgr2=0x7f2e98101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/1307689851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98101880 0x7f2e98101ca0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f2e88009900 tx=0x7f2e8802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 -- 172.21.15.69:0/1307689851 shutdown_connections 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/1307689851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98103430 0x7f2e98109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/1307689851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98102a70 0x7f2e98102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/1307689851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98101880 0x7f2e98101ca0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 -- 172.21.15.69:0/1307689851 >> 172.21.15.69:0/1307689851 conn(0x7f2e980fcdd0 msgr2=0x7f2e980ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:46.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 -- 172.21.15.69:0/1307689851 shutdown_connections 2024-05-09T09:18:46.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.945+0000 7f2e9cc88640 1 -- 172.21.15.69:0/1307689851 wait complete. 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 Processor -- start 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 -- start start 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98101880 0x7f2e980755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98102a70 0x7f2e98073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810d500 con 0x7f2e98103430 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810d380 con 0x7f2e98101880 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9cc88640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2e9810d680 con 0x7f2e98102a70 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43404/0 (socket says 172.21.15.69:43404) 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 -- 172.21.15.69:0/4046656876 learned_addr learned my addr 172.21.15.69:0/4046656876 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:46.947 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e9659c640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98101880 0x7f2e980755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e95d9b640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98102a70 0x7f2e98073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 -- 172.21.15.69:0/4046656876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98102a70 msgr2=0x7f2e98073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98102a70 0x7f2e98073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 -- 172.21.15.69:0/4046656876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98101880 msgr2=0x7f2e980755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98101880 0x7f2e980755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 -- 172.21.15.69:0/4046656876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2e980749f0 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.946+0000 7f2e96d9d640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000eaa0 tx=0x7f2e9000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.947+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e9000cdb0 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.947+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2e90005020 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.947+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2e981a1c30 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.947+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2e900116b0 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.947+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2e981a2180 con 0x7f2e98103430 2024-05-09T09:18:46.948 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.948+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2e98108ba0 con 0x7f2e98103430 2024-05-09T09:18:46.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.949+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2e90005560 con 0x7f2e98103430 2024-05-09T09:18:46.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.950+0000 7f2e877fe640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 0x7f2e800a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:46.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.950+0000 7f2e9659c640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 0x7f2e800a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:46.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.950+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2e900cf6b0 con 0x7f2e98103430 2024-05-09T09:18:46.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.950+0000 7f2e9659c640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 0x7f2e800a5050 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2e88005ab0 tx=0x7f2e88000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:46.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:46.955+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2e90016030 con 0x7f2e98103430 2024-05-09T09:18:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/554257088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2244019757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:46 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/909983206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/554257088' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2244019757' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:46 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/909983206' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.131+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f2e980624d0 con 0x7f2e98103430 2024-05-09T09:18:47.130 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.131+0000 7f2e877fe640 1 -- 172.21.15.69:0/4046656876 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f2e9009c290 con 0x7f2e98103430 2024-05-09T09:18:47.130 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:47.132 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 msgr2=0x7f2e800a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 0x7f2e800a5050 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f2e88005ab0 tx=0x7f2e88000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 msgr2=0x7f2e98074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f2e9000eaa0 tx=0x7f2e9000ef60 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 shutdown_connections 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e800a2b90 0x7f2e800a5050 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2e98103430 0x7f2e98074160 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2e98102a70 0x7f2e98073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 --2- 172.21.15.69:0/4046656876 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2e98101880 0x7f2e980755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 >> 172.21.15.69:0/4046656876 conn(0x7f2e980fcdd0 msgr2=0x7f2e98105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 shutdown_connections 2024-05-09T09:18:47.133 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.133+0000 7f2e9cc88640 1 -- 172.21.15.69:0/4046656876 wait complete. 2024-05-09T09:18:47.145 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:47.252 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 -- 172.21.15.69:0/3423938277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 msgr2=0x7f2fc0108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 --2- 172.21.15.69:0/3423938277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc0108bd0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8009ca0 tx=0x7f2fb802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 -- 172.21.15.69:0/3423938277 shutdown_connections 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 --2- 172.21.15.69:0/3423938277 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc0108bd0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 --2- 172.21.15.69:0/3423938277 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 0x7f2fc0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 --2- 172.21.15.69:0/3423938277 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 0x7f2fc0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 -- 172.21.15.69:0/3423938277 >> 172.21.15.69:0/3423938277 conn(0x7f2fc00fbd00 msgr2=0x7f2fc00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 -- 172.21.15.69:0/3423938277 shutdown_connections 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.251+0000 7f2fc7530640 1 -- 172.21.15.69:0/3423938277 wait complete. 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 Processor -- start 2024-05-09T09:18:47.253 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 -- start start 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 0x7f2fc0196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 0x7f2fc0196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc010c410 con 0x7f2fc0102340 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc010c290 con 0x7f2fc0100790 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc7530640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2fc010c590 con 0x7f2fc0101980 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43422/0 (socket says 172.21.15.69:43422) 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 -- 172.21.15.69:0/4082981019 learned_addr learned my addr 172.21.15.69:0/4082981019 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc52cc640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 0x7f2fc0196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.254 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc4acb640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 0x7f2fc0196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 -- 172.21.15.69:0/4082981019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 msgr2=0x7f2fc0196fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 0x7f2fc0196fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 -- 172.21.15.69:0/4082981019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 msgr2=0x7f2fc0196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 0x7f2fc0196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 -- 172.21.15.69:0/4082981019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2fc019e6a0 con 0x7f2fc0102340 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fc5acd640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8005e30 tx=0x7f2fb8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.255 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.252+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb8003c30 con 0x7f2fc0102340 2024-05-09T09:18:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.253+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2fb8003dd0 con 0x7f2fc0102340 2024-05-09T09:18:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.253+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2fc019e930 con 0x7f2fc0102340 2024-05-09T09:18:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.253+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2fb802fde0 con 0x7f2fc0102340 2024-05-09T09:18:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.253+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2fc019ee80 con 0x7f2fc0102340 2024-05-09T09:18:47.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.253+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2fc0107ab0 con 0x7f2fc0102340 2024-05-09T09:18:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.255+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2fb8004050 con 0x7f2fc0102340 2024-05-09T09:18:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.255+0000 7f2fb27fc640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 0x7f2fac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.255+0000 7f2fc52cc640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 0x7f2fac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.255+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f2fb80f0730 con 0x7f2fc0102340 2024-05-09T09:18:47.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.256+0000 7f2fc52cc640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 0x7f2fac0a5050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2fc01017e0 tx=0x7f2fb400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.259+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2fb80bd310 con 0x7f2fc0102340 2024-05-09T09:18:47.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.436+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f2fc002d3a0 con 0x7f2fc0102340 2024-05-09T09:18:47.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.437+0000 7f2fb27fc640 1 -- 172.21.15.69:0/4082981019 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f2fb80bcc10 con 0x7f2fc0102340 2024-05-09T09:18:47.436 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:47.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 msgr2=0x7f2fac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 0x7f2fac0a5050 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f2fc01017e0 tx=0x7f2fb400b390 comp rx=0 tx=0).stop 2024-05-09T09:18:47.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 msgr2=0x7f2fc019e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f2fb8005e30 tx=0x7f2fb8031ee0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 shutdown_connections 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2fac0a2b90 0x7f2fac0a5050 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2fc0102340 0x7f2fc019e030 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2fc0101980 0x7f2fc0196fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 --2- 172.21.15.69:0/4082981019 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2fc0100790 0x7f2fc0196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 >> 172.21.15.69:0/4082981019 conn(0x7f2fc00fbd00 msgr2=0x7f2fc0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 shutdown_connections 2024-05-09T09:18:47.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.439+0000 7f2fc7530640 1 -- 172.21.15.69:0/4082981019 wait complete. 2024-05-09T09:18:47.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:47.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 -- 172.21.15.69:0/1933612727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c103430 msgr2=0x7efd2c109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 --2- 172.21.15.69:0/1933612727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c103430 0x7efd2c109cc0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7efd24009ca0 tx=0x7efd2402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 -- 172.21.15.69:0/1933612727 shutdown_connections 2024-05-09T09:18:47.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 --2- 172.21.15.69:0/1933612727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c103430 0x7efd2c109cc0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 --2- 172.21.15.69:0/1933612727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c102a70 0x7efd2c102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 --2- 172.21.15.69:0/1933612727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd2c101880 0x7efd2c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.555+0000 7efd31dee640 1 -- 172.21.15.69:0/1933612727 >> 172.21.15.69:0/1933612727 conn(0x7efd2c0fce10 msgr2=0x7efd2c0ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- 172.21.15.69:0/1933612727 shutdown_connections 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- 172.21.15.69:0/1933612727 wait complete. 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 Processor -- start 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- start start 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 0x7efd2c197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd2c103430 0x7efd2c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c10d570 con 0x7efd2c102a70 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c10d3f0 con 0x7efd2c101880 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.556+0000 7efd31dee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd2c10d6f0 con 0x7efd2c103430 2024-05-09T09:18:47.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39934/0 (socket says 172.21.15.69:39934) 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 -- 172.21.15.69:0/159002233 learned_addr learned my addr 172.21.15.69:0/159002233 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 -- 172.21.15.69:0/159002233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd2c103430 msgr2=0x7efd2c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd2c103430 0x7efd2c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2b7fe640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 0x7efd2c197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 -- 172.21.15.69:0/159002233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 msgr2=0x7efd2c197c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 0x7efd2c197c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 -- 172.21.15.69:0/159002233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd2c19f840 con 0x7efd2c102a70 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2b7fe640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 0x7efd2c197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd2affd640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c890 tx=0x7efd1400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd14004220 con 0x7efd2c102a70 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efd140043c0 con 0x7efd2c102a70 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd2c19fb30 con 0x7efd2c102a70 2024-05-09T09:18:47.558 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efd140049e0 con 0x7efd2c102a70 2024-05-09T09:18:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.557+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd2c1a0100 con 0x7efd2c102a70 2024-05-09T09:18:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.558+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd2c108ba0 con 0x7efd2c102a70 2024-05-09T09:18:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.559+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efd140079b0 con 0x7efd2c102a70 2024-05-09T09:18:47.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.560+0000 7efd28ff9640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 0x7efd180a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.560+0000 7efd2b7fe640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 0x7efd180a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.560+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7efd140cf5c0 con 0x7efd2c102a70 2024-05-09T09:18:47.560 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.560+0000 7efd2b7fe640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 0x7efd180a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006fd0 tx=0x7efd1c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.564+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efd14014030 con 0x7efd2c102a70 2024-05-09T09:18:47.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.742+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7efd2c062530 con 0x7efd2c102a70 2024-05-09T09:18:47.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.743+0000 7efd28ff9640 1 -- 172.21.15.69:0/159002233 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7efd1409c140 con 0x7efd2c102a70 2024-05-09T09:18:47.743 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 msgr2=0x7efd180a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 0x7efd180a5050 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7efd1c006fd0 tx=0x7efd1c008040 comp rx=0 tx=0).stop 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 msgr2=0x7efd2c198170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7efd1400c890 tx=0x7efd1400cd50 comp rx=0 tx=0).stop 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 shutdown_connections 2024-05-09T09:18:47.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efd180a2b90 0x7efd180a5050 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd2c103430 0x7efd2c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd2c102a70 0x7efd2c198170 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 --2- 172.21.15.69:0/159002233 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd2c101880 0x7efd2c197c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 >> 172.21.15.69:0/159002233 conn(0x7efd2c0fce10 msgr2=0x7efd2c105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 shutdown_connections 2024-05-09T09:18:47.746 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.745+0000 7efd31dee640 1 -- 172.21.15.69:0/159002233 wait complete. 2024-05-09T09:18:47.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3087688646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4046656876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:47 smithi069 ceph-mon[30123]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:47.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:47 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4082981019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:47.756 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-05-09T09:18:47.756 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 -- 172.21.15.69:0/3853767906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc24102360 msgr2=0x7fcc24108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 --2- 172.21.15.69:0/3853767906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc24102360 0x7fcc24108bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7fcc1c009ca0 tx=0x7fcc1c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 -- 172.21.15.69:0/3853767906 shutdown_connections 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 --2- 172.21.15.69:0/3853767906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc24102360 0x7fcc24108bf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 --2- 172.21.15.69:0/3853767906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241019a0 0x7fcc24101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 --2- 172.21.15.69:0/3853767906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc241007b0 0x7fcc24100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.858+0000 7fcc2959e640 1 -- 172.21.15.69:0/3853767906 >> 172.21.15.69:0/3853767906 conn(0x7fcc240fbd20 msgr2=0x7fcc240fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:47.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- 172.21.15.69:0/3853767906 shutdown_connections 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- 172.21.15.69:0/3853767906 wait complete. 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 Processor -- start 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- start start 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241007b0 0x7fcc2406a6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc24102360 0x7fcc2419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2410c4e0 con 0x7fcc241019a0 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2410c360 con 0x7fcc241007b0 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.859+0000 7fcc2959e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcc2410c660 con 0x7fcc24102360 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39960/0 (socket says 172.21.15.69:39960) 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 -- 172.21.15.69:0/2696353069 learned_addr learned my addr 172.21.15.69:0/2696353069 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc237fe640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc24102360 0x7fcc2419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc22ffd640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241007b0 0x7fcc2406a6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 -- 172.21.15.69:0/2696353069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc24102360 msgr2=0x7fcc2419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc24102360 0x7fcc2419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 -- 172.21.15.69:0/2696353069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241007b0 msgr2=0x7fcc2406a6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241007b0 0x7fcc2406a6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 -- 172.21.15.69:0/2696353069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcc2419f830 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc227fc640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800b4a0 tx=0x7fcc1800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18014070 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcc18010af0 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcc2419fb20 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcc18017d10 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.860+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcc241a0070 con 0x7fcc241019a0 2024-05-09T09:18:47.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.861+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fcc24107ad0 con 0x7fcc241019a0 2024-05-09T09:18:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.862+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcc18017070 con 0x7fcc241019a0 2024-05-09T09:18:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.863+0000 7fcc0bfff640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 0x7fcc0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.863+0000 7fcc22ffd640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 0x7fcc0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.863+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fcc180ce730 con 0x7fcc241019a0 2024-05-09T09:18:47.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.864+0000 7fcc22ffd640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 0x7fcc0c0a5050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc24101800 tx=0x7fcc14005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:47.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:47.868+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fcc1809b310 con 0x7fcc241019a0 2024-05-09T09:18:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3087688646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4046656876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:47 smithi163 ceph-mon[34354]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:47 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4082981019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.081+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fcc2402d1c0 con 0x7fcc241019a0 2024-05-09T09:18:48.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.081+0000 7fcc0bfff640 1 -- 172.21.15.69:0/2696353069 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11879 (secure 0 0 0) 0x7fcc1809ac10 con 0x7fcc241019a0 2024-05-09T09:18:48.081 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:48.081 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:18:44.567983+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24491,24511]},"id":1}]} 2024-05-09T09:18:48.082 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 msgr2=0x7fcc0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 0x7fcc0c0a5050 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7fcc24101800 tx=0x7fcc14005c60 comp rx=0 tx=0).stop 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 msgr2=0x7fcc2419c930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fcc1800b4a0 tx=0x7fcc1800b960 comp rx=0 tx=0).stop 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 shutdown_connections 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcc0c0a2b90 0x7fcc0c0a5050 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcc24102360 0x7fcc2419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcc241019a0 0x7fcc2419c930 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 --2- 172.21.15.69:0/2696353069 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcc241007b0 0x7fcc2406a6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.083+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 >> 172.21.15.69:0/2696353069 conn(0x7fcc240fbd20 msgr2=0x7fcc24104ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.084+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 shutdown_connections 2024-05-09T09:18:48.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.084+0000 7fcc2959e640 1 -- 172.21.15.69:0/2696353069 wait complete. 2024-05-09T09:18:48.096 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-05-09T09:18:48.096 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-05-09T09:18:48.096 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T09:18:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3087688646' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4046656876' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:47 smithi103 ceph-mon[34421]: pgmap v362: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:18:48.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:47 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4082981019' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 -- 172.21.15.69:0/2290346283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410100770 msgr2=0x7f1410100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 --2- 172.21.15.69:0/2290346283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410100770 0x7f1410100b90 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f1400009900 tx=0x7f140002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 -- 172.21.15.69:0/2290346283 shutdown_connections 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 --2- 172.21.15.69:0/2290346283 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410102320 0x7f1410108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 --2- 172.21.15.69:0/2290346283 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410101960 0x7f1410101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 --2- 172.21.15.69:0/2290346283 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410100770 0x7f1410100b90 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 -- 172.21.15.69:0/2290346283 >> 172.21.15.69:0/2290346283 conn(0x7f14100fbce0 msgr2=0x7f14100fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 -- 172.21.15.69:0/2290346283 shutdown_connections 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.197+0000 7f1414de7640 1 -- 172.21.15.69:0/2290346283 wait complete. 2024-05-09T09:18:48.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 Processor -- start 2024-05-09T09:18:48.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 -- start start 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410100770 0x7f1410197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 0x7f141019f150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f141010d570 con 0x7f1410101960 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f141010d3f0 con 0x7f1410102320 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.198+0000 7f1414de7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f141010d6f0 con 0x7f1410100770 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39986/0 (socket says 172.21.15.69:39986) 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 -- 172.21.15.69:0/390347364 learned_addr learned my addr 172.21.15.69:0/390347364 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f140e59c640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410100770 0x7f1410197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f140ed9d640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 0x7f141019f150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 -- 172.21.15.69:0/390347364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410100770 msgr2=0x7f1410197ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410100770 0x7f1410197ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 -- 172.21.15.69:0/390347364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 msgr2=0x7f141019f150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 0x7f141019f150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 -- 172.21.15.69:0/390347364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f141019f7c0 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f140ed9d640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 0x7f141019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1405d9b640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f13f800d950 tx=0x7f13f800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f8017070 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f141019fab0 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13f8009c30 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f13f800f6b0 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.199+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f14101a0050 con 0x7f1410101960 2024-05-09T09:18:48.202 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.200+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1410107a90 con 0x7f1410101960 2024-05-09T09:18:48.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.201+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f13f80109d0 con 0x7f1410101960 2024-05-09T09:18:48.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.202+0000 7f1407fff640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 0x7f13fc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.202+0000 7f140e59c640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 0x7f13fc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.202+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f13f80cefe0 con 0x7f1410101960 2024-05-09T09:18:48.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.203+0000 7f140e59c640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 0x7f13fc0a5050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f140002aa00 tx=0x7f14000315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.206+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f13f8009de0 con 0x7f1410101960 2024-05-09T09:18:48.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.381+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f1410061c70 con 0x7f13fc0a2b90 2024-05-09T09:18:48.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.391+0000 7f1407fff640 1 -- 172.21.15.69:0/390347364 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340780 (secure 0 0 0) 0x7f1410061c70 con 0x7f13fc0a2b90 2024-05-09T09:18:48.395 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:48.397 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:18:48.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 msgr2=0x7f13fc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 0x7f13fc0a5050 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f140002aa00 tx=0x7f14000315b0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 msgr2=0x7f14101980e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f13f800d950 tx=0x7f13f800de10 comp rx=0 tx=0).stop 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 shutdown_connections 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f13fc0a2b90 0x7f13fc0a5050 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1410102320 0x7f141019f150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1410101960 0x7f14101980e0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 --2- 172.21.15.69:0/390347364 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1410100770 0x7f1410197ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 >> 172.21.15.69:0/390347364 conn(0x7f14100fbce0 msgr2=0x7f1410104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 shutdown_connections 2024-05-09T09:18:48.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.394+0000 7f1414de7640 1 -- 172.21.15.69:0/390347364 wait complete. 2024-05-09T09:18:48.406 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-05-09T09:18:47.136643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658965,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":826,"num_read_kb":533,"num_write":356,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181771,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353312,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588512,"statfs":{"total":1151940427776,"available":1151578636288,"internally_reserved":0,"allocated":12738560,"data_stored":8185613,"data_compressed":430872,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":42,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164962+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:18:31.164962+0000","last_peered":"2024-05-09T09:18:31.164962+0000","last_clean":"2024-05-09T09:18:31.164962+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:18:31.164962+0000","last_undegraded":"2024-05-09T09:18:31.164962+0000","last_fullsized":"2024-05-09T09:18:31.164962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981364+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:18:30.981364+0000","last_peered":"2024-05-09T09:18:30.981364+0000","last_clean":"2024-05-09T09:18:30.981364+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:18:30.981364+0000","last_undegraded":"2024-05-09T09:18:30.981364+0000","last_fullsized":"2024-05-09T09:18:30.981364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981277+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:18:30.981277+0000","last_peered":"2024-05-09T09:18:30.981277+0000","last_clean":"2024-05-09T09:18:30.981277+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:18:30.981277+0000","last_undegraded":"2024-05-09T09:18:30.981277+0000","last_fullsized":"2024-05-09T09:18:30.981277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229778+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:18:30.229778+0000","last_peered":"2024-05-09T09:18:30.229778+0000","last_clean":"2024-05-09T09:18:30.229778+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:18:30.229778+0000","last_undegraded":"2024-05-09T09:18:30.229778+0000","last_fullsized":"2024-05-09T09:18:30.229778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512597+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:18:30.512597+0000","last_peered":"2024-05-09T09:18:30.512597+0000","last_clean":"2024-05-09T09:18:30.512597+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:18:30.512597+0000","last_undegraded":"2024-05-09T09:18:30.512597+0000","last_fullsized":"2024-05-09T09:18:30.512597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226905+0000","last_change":"2024-05-09T09:17:30.205057+0000","last_active":"2024-05-09T09:18:30.226905+0000","last_peered":"2024-05-09T09:18:30.226905+0000","last_clean":"2024-05-09T09:18:30.226905+0000","last_became_active":"2024-05-09T09:17:30.204736+0000","last_became_peered":"2024-05-09T09:17:30.204736+0000","last_unstale":"2024-05-09T09:18:30.226905+0000","last_undegraded":"2024-05-09T09:18:30.226905+0000","last_fullsized":"2024-05-09T09:18:30.226905+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:00.112812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492692+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:18:30.492692+0000","last_peered":"2024-05-09T09:18:30.492692+0000","last_clean":"2024-05-09T09:18:30.492692+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:18:30.492692+0000","last_undegraded":"2024-05-09T09:18:30.492692+0000","last_fullsized":"2024-05-09T09:18:30.492692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980080+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:18:30.980080+0000","last_peered":"2024-05-09T09:18:30.980080+0000","last_clean":"2024-05-09T09:18:30.980080+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:18:30.980080+0000","last_undegraded":"2024-05-09T09:18:30.980080+0000","last_fullsized":"2024-05-09T09:18:30.980080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980964+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:18:30.980964+0000","last_peered":"2024-05-09T09:18:30.980964+0000","last_clean":"2024-05-09T09:18:30.980964+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:18:30.980964+0000","last_undegraded":"2024-05-09T09:18:30.980964+0000","last_fullsized":"2024-05-09T09:18:30.980964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227506+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:18:30.227506+0000","last_peered":"2024-05-09T09:18:30.227506+0000","last_clean":"2024-05-09T09:18:30.227506+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:18:30.227506+0000","last_undegraded":"2024-05-09T09:18:30.227506+0000","last_fullsized":"2024-05-09T09:18:30.227506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493143+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:18:30.493143+0000","last_peered":"2024-05-09T09:18:30.493143+0000","last_clean":"2024-05-09T09:18:30.493143+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:18:30.493143+0000","last_undegraded":"2024-05-09T09:18:30.493143+0000","last_fullsized":"2024-05-09T09:18:30.493143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529489+0000","last_change":"2024-05-09T09:17:30.205228+0000","last_active":"2024-05-09T09:18:30.529489+0000","last_peered":"2024-05-09T09:18:30.529489+0000","last_clean":"2024-05-09T09:18:30.529489+0000","last_became_active":"2024-05-09T09:17:30.204824+0000","last_became_peered":"2024-05-09T09:17:30.204824+0000","last_unstale":"2024-05-09T09:18:30.529489+0000","last_undegraded":"2024-05-09T09:18:30.529489+0000","last_fullsized":"2024-05-09T09:18:30.529489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:28.158394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223560+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:18:30.223560+0000","last_peered":"2024-05-09T09:18:30.223560+0000","last_clean":"2024-05-09T09:18:30.223560+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:18:30.223560+0000","last_undegraded":"2024-05-09T09:18:30.223560+0000","last_fullsized":"2024-05-09T09:18:30.223560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980510+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:18:30.980510+0000","last_peered":"2024-05-09T09:18:30.980510+0000","last_clean":"2024-05-09T09:18:30.980510+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:18:30.980510+0000","last_undegraded":"2024-05-09T09:18:30.980510+0000","last_fullsized":"2024-05-09T09:18:30.980510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980927+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:18:30.980927+0000","last_peered":"2024-05-09T09:18:30.980927+0000","last_clean":"2024-05-09T09:18:30.980927+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:18:30.980927+0000","last_undegraded":"2024-05-09T09:18:30.980927+0000","last_fullsized":"2024-05-09T09:18:30.980927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980335+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:18:30.980335+0000","last_peered":"2024-05-09T09:18:30.980335+0000","last_clean":"2024-05-09T09:18:30.980335+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:18:30.980335+0000","last_undegraded":"2024-05-09T09:18:30.980335+0000","last_fullsized":"2024-05-09T09:18:30.980335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164757+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:18:31.164757+0000","last_peered":"2024-05-09T09:18:31.164757+0000","last_clean":"2024-05-09T09:18:31.164757+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:18:31.164757+0000","last_undegraded":"2024-05-09T09:18:31.164757+0000","last_fullsized":"2024-05-09T09:18:31.164757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529708+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:18:30.529708+0000","last_peered":"2024-05-09T09:18:30.529708+0000","last_clean":"2024-05-09T09:18:30.529708+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:18:30.529708+0000","last_undegraded":"2024-05-09T09:18:30.529708+0000","last_fullsized":"2024-05-09T09:18:30.529708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226663+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:18:30.226663+0000","last_peered":"2024-05-09T09:18:30.226663+0000","last_clean":"2024-05-09T09:18:30.226663+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:18:30.226663+0000","last_undegraded":"2024-05-09T09:18:30.226663+0000","last_fullsized":"2024-05-09T09:18:30.226663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226902+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:18:30.226902+0000","last_peered":"2024-05-09T09:18:30.226902+0000","last_clean":"2024-05-09T09:18:30.226902+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:18:30.226902+0000","last_undegraded":"2024-05-09T09:18:30.226902+0000","last_fullsized":"2024-05-09T09:18:30.226902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529614+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:18:30.529614+0000","last_peered":"2024-05-09T09:18:30.529614+0000","last_clean":"2024-05-09T09:18:30.529614+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:18:30.529614+0000","last_undegraded":"2024-05-09T09:18:30.529614+0000","last_fullsized":"2024-05-09T09:18:30.529614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980873+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:18:30.980873+0000","last_peered":"2024-05-09T09:18:30.980873+0000","last_clean":"2024-05-09T09:18:30.980873+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:18:30.980873+0000","last_undegraded":"2024-05-09T09:18:30.980873+0000","last_fullsized":"2024-05-09T09:18:30.980873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228966+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:18:30.228966+0000","last_peered":"2024-05-09T09:18:30.228966+0000","last_clean":"2024-05-09T09:18:30.228966+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:18:30.228966+0000","last_undegraded":"2024-05-09T09:18:30.228966+0000","last_fullsized":"2024-05-09T09:18:30.228966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981126+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:18:30.981126+0000","last_peered":"2024-05-09T09:18:30.981126+0000","last_clean":"2024-05-09T09:18:30.981126+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:18:30.981126+0000","last_undegraded":"2024-05-09T09:18:30.981126+0000","last_fullsized":"2024-05-09T09:18:30.981126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980521+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:18:30.980521+0000","last_peered":"2024-05-09T09:18:30.980521+0000","last_clean":"2024-05-09T09:18:30.980521+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:18:30.980521+0000","last_undegraded":"2024-05-09T09:18:30.980521+0000","last_fullsized":"2024-05-09T09:18:30.980521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229455+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:18:30.229455+0000","last_peered":"2024-05-09T09:18:30.229455+0000","last_clean":"2024-05-09T09:18:30.229455+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:18:30.229455+0000","last_undegraded":"2024-05-09T09:18:30.229455+0000","last_fullsized":"2024-05-09T09:18:30.229455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164671+0000","last_change":"2024-05-09T09:17:30.213383+0000","last_active":"2024-05-09T09:18:31.164671+0000","last_peered":"2024-05-09T09:18:31.164671+0000","last_clean":"2024-05-09T09:18:31.164671+0000","last_became_active":"2024-05-09T09:17:30.205007+0000","last_became_peered":"2024-05-09T09:17:30.205007+0000","last_unstale":"2024-05-09T09:18:31.164671+0000","last_undegraded":"2024-05-09T09:18:31.164671+0000","last_fullsized":"2024-05-09T09:18:31.164671+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:36.820105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980640+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:18:30.980640+0000","last_peered":"2024-05-09T09:18:30.980640+0000","last_clean":"2024-05-09T09:18:30.980640+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:18:30.980640+0000","last_undegraded":"2024-05-09T09:18:30.980640+0000","last_fullsized":"2024-05-09T09:18:30.980640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229018+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:18:30.229018+0000","last_peered":"2024-05-09T09:18:30.229018+0000","last_clean":"2024-05-09T09:18:30.229018+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:18:30.229018+0000","last_undegraded":"2024-05-09T09:18:30.229018+0000","last_fullsized":"2024-05-09T09:18:30.229018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512765+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:18:30.512765+0000","last_peered":"2024-05-09T09:18:30.512765+0000","last_clean":"2024-05-09T09:18:30.512765+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:18:30.512765+0000","last_undegraded":"2024-05-09T09:18:30.512765+0000","last_fullsized":"2024-05-09T09:18:30.512765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227762+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:18:30.227762+0000","last_peered":"2024-05-09T09:18:30.227762+0000","last_clean":"2024-05-09T09:18:30.227762+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:18:30.227762+0000","last_undegraded":"2024-05-09T09:18:30.227762+0000","last_fullsized":"2024-05-09T09:18:30.227762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226533+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:18:30.226533+0000","last_peered":"2024-05-09T09:18:30.226533+0000","last_clean":"2024-05-09T09:18:30.226533+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:18:30.226533+0000","last_undegraded":"2024-05-09T09:18:30.226533+0000","last_fullsized":"2024-05-09T09:18:30.226533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389904+0000","last_change":"2024-05-09T09:17:30.281412+0000","last_active":"2024-05-09T09:18:45.389904+0000","last_peered":"2024-05-09T09:18:45.389904+0000","last_clean":"2024-05-09T09:18:45.389904+0000","last_became_active":"2024-05-09T09:17:30.277253+0000","last_became_peered":"2024-05-09T09:17:30.277253+0000","last_unstale":"2024-05-09T09:18:45.389904+0000","last_undegraded":"2024-05-09T09:18:45.389904+0000","last_fullsized":"2024-05-09T09:18:45.389904+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:48.491205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512581+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:18:30.512581+0000","last_peered":"2024-05-09T09:18:30.512581+0000","last_clean":"2024-05-09T09:18:30.512581+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:18:30.512581+0000","last_undegraded":"2024-05-09T09:18:30.512581+0000","last_fullsized":"2024-05-09T09:18:30.512581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229789+0000","last_change":"2024-05-09T09:18:30.229789+0000","last_active":"2024-05-09T09:18:30.229789+0000","last_peered":"2024-05-09T09:18:30.229789+0000","last_clean":"2024-05-09T09:18:30.229789+0000","last_became_active":"2024-05-09T09:18:30.229281+0000","last_became_peered":"2024-05-09T09:18:30.229281+0000","last_unstale":"2024-05-09T09:18:30.229789+0000","last_undegraded":"2024-05-09T09:18:30.229789+0000","last_fullsized":"2024-05-09T09:18:30.229789+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:55.816495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226891+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:18:30.226891+0000","last_peered":"2024-05-09T09:18:30.226891+0000","last_clean":"2024-05-09T09:18:30.226891+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:18:30.226891+0000","last_undegraded":"2024-05-09T09:18:30.226891+0000","last_fullsized":"2024-05-09T09:18:30.226891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227826+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:18:30.227826+0000","last_peered":"2024-05-09T09:18:30.227826+0000","last_clean":"2024-05-09T09:18:30.227826+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:18:30.227826+0000","last_undegraded":"2024-05-09T09:18:30.227826+0000","last_fullsized":"2024-05-09T09:18:30.227826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164767+0000","last_change":"2024-05-09T09:17:30.277465+0000","last_active":"2024-05-09T09:18:31.164767+0000","last_peered":"2024-05-09T09:18:31.164767+0000","last_clean":"2024-05-09T09:18:31.164767+0000","last_became_active":"2024-05-09T09:17:30.277014+0000","last_became_peered":"2024-05-09T09:17:30.277014+0000","last_unstale":"2024-05-09T09:18:31.164767+0000","last_undegraded":"2024-05-09T09:18:31.164767+0000","last_fullsized":"2024-05-09T09:18:31.164767+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:36:04.346440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223244+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:18:30.223244+0000","last_peered":"2024-05-09T09:18:30.223244+0000","last_clean":"2024-05-09T09:18:30.223244+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:18:30.223244+0000","last_undegraded":"2024-05-09T09:18:30.223244+0000","last_fullsized":"2024-05-09T09:18:30.223244+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980233+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:18:30.980233+0000","last_peered":"2024-05-09T09:18:30.980233+0000","last_clean":"2024-05-09T09:18:30.980233+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:18:30.980233+0000","last_undegraded":"2024-05-09T09:18:30.980233+0000","last_fullsized":"2024-05-09T09:18:30.980233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981359+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:18:30.981359+0000","last_peered":"2024-05-09T09:18:30.981359+0000","last_clean":"2024-05-09T09:18:30.981359+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:18:30.981359+0000","last_undegraded":"2024-05-09T09:18:30.981359+0000","last_fullsized":"2024-05-09T09:18:30.981359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227863+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:18:30.227863+0000","last_peered":"2024-05-09T09:18:30.227863+0000","last_clean":"2024-05-09T09:18:30.227863+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:18:30.227863+0000","last_undegraded":"2024-05-09T09:18:30.227863+0000","last_fullsized":"2024-05-09T09:18:30.227863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529239+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:18:30.529239+0000","last_peered":"2024-05-09T09:18:30.529239+0000","last_clean":"2024-05-09T09:18:30.529239+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:18:30.529239+0000","last_undegraded":"2024-05-09T09:18:30.529239+0000","last_fullsized":"2024-05-09T09:18:30.529239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980320+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:18:30.980320+0000","last_peered":"2024-05-09T09:18:30.980320+0000","last_clean":"2024-05-09T09:18:30.980320+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:18:30.980320+0000","last_undegraded":"2024-05-09T09:18:30.980320+0000","last_fullsized":"2024-05-09T09:18:30.980320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389402+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:18:45.389402+0000","last_peered":"2024-05-09T09:18:45.389402+0000","last_clean":"2024-05-09T09:18:45.389402+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:18:45.389402+0000","last_undegraded":"2024-05-09T09:18:45.389402+0000","last_fullsized":"2024-05-09T09:18:45.389402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980894+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:18:30.980894+0000","last_peered":"2024-05-09T09:18:30.980894+0000","last_clean":"2024-05-09T09:18:30.980894+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:18:30.980894+0000","last_undegraded":"2024-05-09T09:18:30.980894+0000","last_fullsized":"2024-05-09T09:18:30.980894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529090+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:18:30.529090+0000","last_peered":"2024-05-09T09:18:30.529090+0000","last_clean":"2024-05-09T09:18:30.529090+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:18:30.529090+0000","last_undegraded":"2024-05-09T09:18:30.529090+0000","last_fullsized":"2024-05-09T09:18:30.529090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493132+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:18:30.493132+0000","last_peered":"2024-05-09T09:18:30.493132+0000","last_clean":"2024-05-09T09:18:30.493132+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:18:30.493132+0000","last_undegraded":"2024-05-09T09:18:30.493132+0000","last_fullsized":"2024-05-09T09:18:30.493132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980583+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:18:30.980583+0000","last_peered":"2024-05-09T09:18:30.980583+0000","last_clean":"2024-05-09T09:18:30.980583+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:18:30.980583+0000","last_undegraded":"2024-05-09T09:18:30.980583+0000","last_fullsized":"2024-05-09T09:18:30.980583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229610+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:18:30.229610+0000","last_peered":"2024-05-09T09:18:30.229610+0000","last_clean":"2024-05-09T09:18:30.229610+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:18:30.229610+0000","last_undegraded":"2024-05-09T09:18:30.229610+0000","last_fullsized":"2024-05-09T09:18:30.229610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227664+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:18:30.227664+0000","last_peered":"2024-05-09T09:18:30.227664+0000","last_clean":"2024-05-09T09:18:30.227664+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:18:30.227664+0000","last_undegraded":"2024-05-09T09:18:30.227664+0000","last_fullsized":"2024-05-09T09:18:30.227664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980767+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:18:30.980767+0000","last_peered":"2024-05-09T09:18:30.980767+0000","last_clean":"2024-05-09T09:18:30.980767+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:18:30.980767+0000","last_undegraded":"2024-05-09T09:18:30.980767+0000","last_fullsized":"2024-05-09T09:18:30.980767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529402+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:18:30.529402+0000","last_peered":"2024-05-09T09:18:30.529402+0000","last_clean":"2024-05-09T09:18:30.529402+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:18:30.529402+0000","last_undegraded":"2024-05-09T09:18:30.529402+0000","last_fullsized":"2024-05-09T09:18:30.529402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:46.390433+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:18:46.390433+0000","last_peered":"2024-05-09T09:18:46.390433+0000","last_clean":"2024-05-09T09:18:46.390433+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:18:46.390433+0000","last_undegraded":"2024-05-09T09:18:46.390433+0000","last_fullsized":"2024-05-09T09:18:46.390433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63913,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":181,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512855+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:18:30.512855+0000","last_peered":"2024-05-09T09:18:30.512855+0000","last_clean":"2024-05-09T09:18:30.512855+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:18:30.512855+0000","last_undegraded":"2024-05-09T09:18:30.512855+0000","last_fullsized":"2024-05-09T09:18:30.512855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228102+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:18:30.228102+0000","last_peered":"2024-05-09T09:18:30.228102+0000","last_clean":"2024-05-09T09:18:30.228102+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:18:30.228102+0000","last_undegraded":"2024-05-09T09:18:30.228102+0000","last_fullsized":"2024-05-09T09:18:30.228102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226401+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:18:30.226401+0000","last_peered":"2024-05-09T09:18:30.226401+0000","last_clean":"2024-05-09T09:18:30.226401+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:18:30.226401+0000","last_undegraded":"2024-05-09T09:18:30.226401+0000","last_fullsized":"2024-05-09T09:18:30.226401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223357+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:18:30.223357+0000","last_peered":"2024-05-09T09:18:30.223357+0000","last_clean":"2024-05-09T09:18:30.223357+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:18:30.223357+0000","last_undegraded":"2024-05-09T09:18:30.223357+0000","last_fullsized":"2024-05-09T09:18:30.223357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164421+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:18:31.164421+0000","last_peered":"2024-05-09T09:18:31.164421+0000","last_clean":"2024-05-09T09:18:31.164421+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:18:31.164421+0000","last_undegraded":"2024-05-09T09:18:31.164421+0000","last_fullsized":"2024-05-09T09:18:31.164421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227484+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:18:30.227484+0000","last_peered":"2024-05-09T09:18:30.227484+0000","last_clean":"2024-05-09T09:18:30.227484+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:18:30.227484+0000","last_undegraded":"2024-05-09T09:18:30.227484+0000","last_fullsized":"2024-05-09T09:18:30.227484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492851+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:18:30.492851+0000","last_peered":"2024-05-09T09:18:30.492851+0000","last_clean":"2024-05-09T09:18:30.492851+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:18:30.492851+0000","last_undegraded":"2024-05-09T09:18:30.492851+0000","last_fullsized":"2024-05-09T09:18:30.492851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513032+0000","last_change":"2024-05-09T09:17:30.465425+0000","last_active":"2024-05-09T09:18:30.513032+0000","last_peered":"2024-05-09T09:18:30.513032+0000","last_clean":"2024-05-09T09:18:30.513032+0000","last_became_active":"2024-05-09T09:17:30.465043+0000","last_became_peered":"2024-05-09T09:17:30.465043+0000","last_unstale":"2024-05-09T09:18:30.513032+0000","last_undegraded":"2024-05-09T09:18:30.513032+0000","last_fullsized":"2024-05-09T09:18:30.513032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:16:54.706845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:40.381710+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:18:40.381710+0000","last_peered":"2024-05-09T09:18:40.381710+0000","last_clean":"2024-05-09T09:18:40.381710+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:18:40.381710+0000","last_undegraded":"2024-05-09T09:18:40.381710+0000","last_fullsized":"2024-05-09T09:18:40.381710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492685+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:18:30.492685+0000","last_peered":"2024-05-09T09:18:30.492685+0000","last_clean":"2024-05-09T09:18:30.492685+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:18:30.492685+0000","last_undegraded":"2024-05-09T09:18:30.492685+0000","last_fullsized":"2024-05-09T09:18:30.492685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223712+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:18:30.223712+0000","last_peered":"2024-05-09T09:18:30.223712+0000","last_clean":"2024-05-09T09:18:30.223712+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:18:30.223712+0000","last_undegraded":"2024-05-09T09:18:30.223712+0000","last_fullsized":"2024-05-09T09:18:30.223712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512424+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:18:30.512424+0000","last_peered":"2024-05-09T09:18:30.512424+0000","last_clean":"2024-05-09T09:18:30.512424+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:18:30.512424+0000","last_undegraded":"2024-05-09T09:18:30.512424+0000","last_fullsized":"2024-05-09T09:18:30.512424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227187+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:18:30.227187+0000","last_peered":"2024-05-09T09:18:30.227187+0000","last_clean":"2024-05-09T09:18:30.227187+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:18:30.227187+0000","last_undegraded":"2024-05-09T09:18:30.227187+0000","last_fullsized":"2024-05-09T09:18:30.227187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227206+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:18:30.227206+0000","last_peered":"2024-05-09T09:18:30.227206+0000","last_clean":"2024-05-09T09:18:30.227206+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:18:30.227206+0000","last_undegraded":"2024-05-09T09:18:30.227206+0000","last_fullsized":"2024-05-09T09:18:30.227206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513190+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:18:30.513190+0000","last_peered":"2024-05-09T09:18:30.513190+0000","last_clean":"2024-05-09T09:18:30.513190+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:18:30.513190+0000","last_undegraded":"2024-05-09T09:18:30.513190+0000","last_fullsized":"2024-05-09T09:18:30.513190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164659+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:18:31.164659+0000","last_peered":"2024-05-09T09:18:31.164659+0000","last_clean":"2024-05-09T09:18:31.164659+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:18:31.164659+0000","last_undegraded":"2024-05-09T09:18:31.164659+0000","last_fullsized":"2024-05-09T09:18:31.164659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512820+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:18:30.512820+0000","last_peered":"2024-05-09T09:18:30.512820+0000","last_clean":"2024-05-09T09:18:30.512820+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:18:30.512820+0000","last_undegraded":"2024-05-09T09:18:30.512820+0000","last_fullsized":"2024-05-09T09:18:30.512820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529852+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:18:30.529852+0000","last_peered":"2024-05-09T09:18:30.529852+0000","last_clean":"2024-05-09T09:18:30.529852+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:18:30.529852+0000","last_undegraded":"2024-05-09T09:18:30.529852+0000","last_fullsized":"2024-05-09T09:18:30.529852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227403+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:18:30.227403+0000","last_peered":"2024-05-09T09:18:30.227403+0000","last_clean":"2024-05-09T09:18:30.227403+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:18:30.227403+0000","last_undegraded":"2024-05-09T09:18:30.227403+0000","last_fullsized":"2024-05-09T09:18:30.227403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223570+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:18:30.223570+0000","last_peered":"2024-05-09T09:18:30.223570+0000","last_clean":"2024-05-09T09:18:30.223570+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:18:30.223570+0000","last_undegraded":"2024-05-09T09:18:30.223570+0000","last_fullsized":"2024-05-09T09:18:30.223570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226975+0000","last_change":"2024-05-09T09:17:30.203699+0000","last_active":"2024-05-09T09:18:30.226975+0000","last_peered":"2024-05-09T09:18:30.226975+0000","last_clean":"2024-05-09T09:18:30.226975+0000","last_became_active":"2024-05-09T09:17:30.203312+0000","last_became_peered":"2024-05-09T09:17:30.203312+0000","last_unstale":"2024-05-09T09:18:30.226975+0000","last_undegraded":"2024-05-09T09:18:30.226975+0000","last_fullsized":"2024-05-09T09:18:30.226975+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529073+0000","last_change":"2024-05-09T09:17:30.276150+0000","last_active":"2024-05-09T09:18:30.529073+0000","last_peered":"2024-05-09T09:18:30.529073+0000","last_clean":"2024-05-09T09:18:30.529073+0000","last_became_active":"2024-05-09T09:17:30.275868+0000","last_became_peered":"2024-05-09T09:17:30.275868+0000","last_unstale":"2024-05-09T09:18:30.529073+0000","last_undegraded":"2024-05-09T09:18:30.529073+0000","last_fullsized":"2024-05-09T09:18:30.529073+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:18:08.892380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":145,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:41.382493+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:18:41.382493+0000","last_peered":"2024-05-09T09:18:41.382493+0000","last_clean":"2024-05-09T09:18:41.382493+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:18:41.382493+0000","last_undegraded":"2024-05-09T09:18:41.382493+0000","last_fullsized":"2024-05-09T09:18:41.382493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493069+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:18:30.493069+0000","last_peered":"2024-05-09T09:18:30.493069+0000","last_clean":"2024-05-09T09:18:30.493069+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:18:30.493069+0000","last_undegraded":"2024-05-09T09:18:30.493069+0000","last_fullsized":"2024-05-09T09:18:30.493069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980587+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:18:30.980587+0000","last_peered":"2024-05-09T09:18:30.980587+0000","last_clean":"2024-05-09T09:18:30.980587+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:18:30.980587+0000","last_undegraded":"2024-05-09T09:18:30.980587+0000","last_fullsized":"2024-05-09T09:18:30.980587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229495+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:18:30.229495+0000","last_peered":"2024-05-09T09:18:30.229495+0000","last_clean":"2024-05-09T09:18:30.229495+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:18:30.229495+0000","last_undegraded":"2024-05-09T09:18:30.229495+0000","last_fullsized":"2024-05-09T09:18:30.229495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492528+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:18:30.492528+0000","last_peered":"2024-05-09T09:18:30.492528+0000","last_clean":"2024-05-09T09:18:30.492528+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:18:30.492528+0000","last_undegraded":"2024-05-09T09:18:30.492528+0000","last_fullsized":"2024-05-09T09:18:30.492528+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227346+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:18:30.227346+0000","last_peered":"2024-05-09T09:18:30.227346+0000","last_clean":"2024-05-09T09:18:30.227346+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:18:30.227346+0000","last_undegraded":"2024-05-09T09:18:30.227346+0000","last_fullsized":"2024-05-09T09:18:30.227346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223208+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:18:30.223208+0000","last_peered":"2024-05-09T09:18:30.223208+0000","last_clean":"2024-05-09T09:18:30.223208+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:18:30.223208+0000","last_undegraded":"2024-05-09T09:18:30.223208+0000","last_fullsized":"2024-05-09T09:18:30.223208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492715+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:18:30.492715+0000","last_peered":"2024-05-09T09:18:30.492715+0000","last_clean":"2024-05-09T09:18:30.492715+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:18:30.492715+0000","last_undegraded":"2024-05-09T09:18:30.492715+0000","last_fullsized":"2024-05-09T09:18:30.492715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164409+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:18:31.164409+0000","last_peered":"2024-05-09T09:18:31.164409+0000","last_clean":"2024-05-09T09:18:31.164409+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:18:31.164409+0000","last_undegraded":"2024-05-09T09:18:31.164409+0000","last_fullsized":"2024-05-09T09:18:31.164409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164541+0000","last_change":"2024-05-09T09:17:30.208053+0000","last_active":"2024-05-09T09:18:31.164541+0000","last_peered":"2024-05-09T09:18:31.164541+0000","last_clean":"2024-05-09T09:18:31.164541+0000","last_became_active":"2024-05-09T09:17:30.203628+0000","last_became_peered":"2024-05-09T09:17:30.203628+0000","last_unstale":"2024-05-09T09:18:31.164541+0000","last_undegraded":"2024-05-09T09:18:31.164541+0000","last_fullsized":"2024-05-09T09:18:31.164541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:37.871338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980275+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:18:30.980275+0000","last_peered":"2024-05-09T09:18:30.980275+0000","last_clean":"2024-05-09T09:18:30.980275+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:18:30.980275+0000","last_undegraded":"2024-05-09T09:18:30.980275+0000","last_fullsized":"2024-05-09T09:18:30.980275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'10","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:43.881851+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:18:43.881851+0000","last_peered":"2024-05-09T09:18:43.881851+0000","last_clean":"2024-05-09T09:18:43.881851+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:18:43.881851+0000","last_undegraded":"2024-05-09T09:18:43.881851+0000","last_fullsized":"2024-05-09T09:18:43.881851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981095+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:18:30.981095+0000","last_peered":"2024-05-09T09:18:30.981095+0000","last_clean":"2024-05-09T09:18:30.981095+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:18:30.981095+0000","last_undegraded":"2024-05-09T09:18:30.981095+0000","last_fullsized":"2024-05-09T09:18:30.981095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226060+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:18:30.226060+0000","last_peered":"2024-05-09T09:18:30.226060+0000","last_clean":"2024-05-09T09:18:30.226060+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:18:30.226060+0000","last_undegraded":"2024-05-09T09:18:30.226060+0000","last_fullsized":"2024-05-09T09:18:30.226060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229540+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:18:30.229540+0000","last_peered":"2024-05-09T09:18:30.229540+0000","last_clean":"2024-05-09T09:18:30.229540+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:18:30.229540+0000","last_undegraded":"2024-05-09T09:18:30.229540+0000","last_fullsized":"2024-05-09T09:18:30.229540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981427+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:18:30.981427+0000","last_peered":"2024-05-09T09:18:30.981427+0000","last_clean":"2024-05-09T09:18:30.981427+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:18:30.981427+0000","last_undegraded":"2024-05-09T09:18:30.981427+0000","last_fullsized":"2024-05-09T09:18:30.981427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.530637+0000","last_change":"2024-05-09T09:18:30.530637+0000","last_active":"2024-05-09T09:18:30.530637+0000","last_peered":"2024-05-09T09:18:30.530637+0000","last_clean":"2024-05-09T09:18:30.530637+0000","last_became_active":"2024-05-09T09:18:30.530213+0000","last_became_peered":"2024-05-09T09:18:30.530213+0000","last_unstale":"2024-05-09T09:18:30.530637+0000","last_undegraded":"2024-05-09T09:18:30.530637+0000","last_fullsized":"2024-05-09T09:18:30.530637+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:58:48.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980453+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:18:30.980453+0000","last_peered":"2024-05-09T09:18:30.980453+0000","last_clean":"2024-05-09T09:18:30.980453+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:18:30.980453+0000","last_undegraded":"2024-05-09T09:18:30.980453+0000","last_fullsized":"2024-05-09T09:18:30.980453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512512+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:18:30.512512+0000","last_peered":"2024-05-09T09:18:30.512512+0000","last_clean":"2024-05-09T09:18:30.512512+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:18:30.512512+0000","last_undegraded":"2024-05-09T09:18:30.512512+0000","last_fullsized":"2024-05-09T09:18:30.512512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529569+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:18:30.529569+0000","last_peered":"2024-05-09T09:18:30.529569+0000","last_clean":"2024-05-09T09:18:30.529569+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:18:30.529569+0000","last_undegraded":"2024-05-09T09:18:30.529569+0000","last_fullsized":"2024-05-09T09:18:30.529569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980366+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:18:30.980366+0000","last_peered":"2024-05-09T09:18:30.980366+0000","last_clean":"2024-05-09T09:18:30.980366+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:18:30.980366+0000","last_undegraded":"2024-05-09T09:18:30.980366+0000","last_fullsized":"2024-05-09T09:18:30.980366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164377+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:18:31.164377+0000","last_peered":"2024-05-09T09:18:31.164377+0000","last_clean":"2024-05-09T09:18:31.164377+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:18:31.164377+0000","last_undegraded":"2024-05-09T09:18:31.164377+0000","last_fullsized":"2024-05-09T09:18:31.164377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229286+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:18:30.229286+0000","last_peered":"2024-05-09T09:18:30.229286+0000","last_clean":"2024-05-09T09:18:30.229286+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:18:30.229286+0000","last_undegraded":"2024-05-09T09:18:30.229286+0000","last_fullsized":"2024-05-09T09:18:30.229286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226564+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:18:30.226564+0000","last_peered":"2024-05-09T09:18:30.226564+0000","last_clean":"2024-05-09T09:18:30.226564+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:18:30.226564+0000","last_undegraded":"2024-05-09T09:18:30.226564+0000","last_fullsized":"2024-05-09T09:18:30.226564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980483+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:18:30.980483+0000","last_peered":"2024-05-09T09:18:30.980483+0000","last_clean":"2024-05-09T09:18:30.980483+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:18:30.980483+0000","last_undegraded":"2024-05-09T09:18:30.980483+0000","last_fullsized":"2024-05-09T09:18:30.980483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229268+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:18:30.229268+0000","last_peered":"2024-05-09T09:18:30.229268+0000","last_clean":"2024-05-09T09:18:30.229268+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:18:30.229268+0000","last_undegraded":"2024-05-09T09:18:30.229268+0000","last_fullsized":"2024-05-09T09:18:30.229268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512609+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:18:30.512609+0000","last_peered":"2024-05-09T09:18:30.512609+0000","last_clean":"2024-05-09T09:18:30.512609+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:18:30.512609+0000","last_undegraded":"2024-05-09T09:18:30.512609+0000","last_fullsized":"2024-05-09T09:18:30.512609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980379+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:18:30.980379+0000","last_peered":"2024-05-09T09:18:30.980379+0000","last_clean":"2024-05-09T09:18:30.980379+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:18:30.980379+0000","last_undegraded":"2024-05-09T09:18:30.980379+0000","last_fullsized":"2024-05-09T09:18:30.980379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227058+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:18:30.227058+0000","last_peered":"2024-05-09T09:18:30.227058+0000","last_clean":"2024-05-09T09:18:30.227058+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:18:30.227058+0000","last_undegraded":"2024-05-09T09:18:30.227058+0000","last_fullsized":"2024-05-09T09:18:30.227058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493221+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:18:30.493221+0000","last_peered":"2024-05-09T09:18:30.493221+0000","last_clean":"2024-05-09T09:18:30.493221+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:18:30.493221+0000","last_undegraded":"2024-05-09T09:18:30.493221+0000","last_fullsized":"2024-05-09T09:18:30.493221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980643+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:18:30.980643+0000","last_peered":"2024-05-09T09:18:30.980643+0000","last_clean":"2024-05-09T09:18:30.980643+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:18:30.980643+0000","last_undegraded":"2024-05-09T09:18:30.980643+0000","last_fullsized":"2024-05-09T09:18:30.980643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227912+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:18:30.227912+0000","last_peered":"2024-05-09T09:18:30.227912+0000","last_clean":"2024-05-09T09:18:30.227912+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:18:30.227912+0000","last_undegraded":"2024-05-09T09:18:30.227912+0000","last_fullsized":"2024-05-09T09:18:30.227912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492929+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:18:30.492929+0000","last_peered":"2024-05-09T09:18:30.492929+0000","last_clean":"2024-05-09T09:18:30.492929+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:18:30.492929+0000","last_undegraded":"2024-05-09T09:18:30.492929+0000","last_fullsized":"2024-05-09T09:18:30.492929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227073+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:18:30.227073+0000","last_peered":"2024-05-09T09:18:30.227073+0000","last_clean":"2024-05-09T09:18:30.227073+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:18:30.227073+0000","last_undegraded":"2024-05-09T09:18:30.227073+0000","last_fullsized":"2024-05-09T09:18:30.227073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223662+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:18:30.223662+0000","last_peered":"2024-05-09T09:18:30.223662+0000","last_clean":"2024-05-09T09:18:30.223662+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:18:30.223662+0000","last_undegraded":"2024-05-09T09:18:30.223662+0000","last_fullsized":"2024-05-09T09:18:30.223662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493231+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:18:30.493231+0000","last_peered":"2024-05-09T09:18:30.493231+0000","last_clean":"2024-05-09T09:18:30.493231+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:18:30.493231+0000","last_undegraded":"2024-05-09T09:18:30.493231+0000","last_fullsized":"2024-05-09T09:18:30.493231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164450+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:18:31.164450+0000","last_peered":"2024-05-09T09:18:31.164450+0000","last_clean":"2024-05-09T09:18:31.164450+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:18:31.164450+0000","last_undegraded":"2024-05-09T09:18:31.164450+0000","last_fullsized":"2024-05-09T09:18:31.164450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227152+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:18:30.227152+0000","last_peered":"2024-05-09T09:18:30.227152+0000","last_clean":"2024-05-09T09:18:30.227152+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:18:30.227152+0000","last_undegraded":"2024-05-09T09:18:30.227152+0000","last_fullsized":"2024-05-09T09:18:30.227152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226406+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:18:30.226406+0000","last_peered":"2024-05-09T09:18:30.226406+0000","last_clean":"2024-05-09T09:18:30.226406+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:18:30.226406+0000","last_undegraded":"2024-05-09T09:18:30.226406+0000","last_fullsized":"2024-05-09T09:18:30.226406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980983+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:18:30.980983+0000","last_peered":"2024-05-09T09:18:30.980983+0000","last_clean":"2024-05-09T09:18:30.980983+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:18:30.980983+0000","last_undegraded":"2024-05-09T09:18:30.980983+0000","last_fullsized":"2024-05-09T09:18:30.980983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223725+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:18:30.223725+0000","last_peered":"2024-05-09T09:18:30.223725+0000","last_clean":"2024-05-09T09:18:30.223725+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:18:30.223725+0000","last_undegraded":"2024-05-09T09:18:30.223725+0000","last_fullsized":"2024-05-09T09:18:30.223725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164624+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:18:31.164624+0000","last_peered":"2024-05-09T09:18:31.164624+0000","last_clean":"2024-05-09T09:18:31.164624+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:18:31.164624+0000","last_undegraded":"2024-05-09T09:18:31.164624+0000","last_fullsized":"2024-05-09T09:18:31.164624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227381+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:18:30.227381+0000","last_peered":"2024-05-09T09:18:30.227381+0000","last_clean":"2024-05-09T09:18:30.227381+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:18:30.227381+0000","last_undegraded":"2024-05-09T09:18:30.227381+0000","last_fullsized":"2024-05-09T09:18:30.227381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981199+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:18:30.981199+0000","last_peered":"2024-05-09T09:18:30.981199+0000","last_clean":"2024-05-09T09:18:30.981199+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:18:30.981199+0000","last_undegraded":"2024-05-09T09:18:30.981199+0000","last_fullsized":"2024-05-09T09:18:30.981199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228022+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:18:30.228022+0000","last_peered":"2024-05-09T09:18:30.228022+0000","last_clean":"2024-05-09T09:18:30.228022+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:18:30.228022+0000","last_undegraded":"2024-05-09T09:18:30.228022+0000","last_fullsized":"2024-05-09T09:18:30.228022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980884+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:18:30.980884+0000","last_peered":"2024-05-09T09:18:30.980884+0000","last_clean":"2024-05-09T09:18:30.980884+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:18:30.980884+0000","last_undegraded":"2024-05-09T09:18:30.980884+0000","last_fullsized":"2024-05-09T09:18:30.980884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980558+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:18:30.980558+0000","last_peered":"2024-05-09T09:18:30.980558+0000","last_clean":"2024-05-09T09:18:30.980558+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:18:30.980558+0000","last_undegraded":"2024-05-09T09:18:30.980558+0000","last_fullsized":"2024-05-09T09:18:30.980558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223438+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:18:30.223438+0000","last_peered":"2024-05-09T09:18:30.223438+0000","last_clean":"2024-05-09T09:18:30.223438+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:18:30.223438+0000","last_undegraded":"2024-05-09T09:18:30.223438+0000","last_fullsized":"2024-05-09T09:18:30.223438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164557+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:18:31.164557+0000","last_peered":"2024-05-09T09:18:31.164557+0000","last_clean":"2024-05-09T09:18:31.164557+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:18:31.164557+0000","last_undegraded":"2024-05-09T09:18:31.164557+0000","last_fullsized":"2024-05-09T09:18:31.164557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68363,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":700,"num_read_kb":424,"num_write":173,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1035,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":209538,"data_compressed":4887,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":519498,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1170657,"data_compressed":63914,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523808,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":519610,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115682,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":514888,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":578403,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516276,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":579439,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515082,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":520692,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1111578,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63605,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70014,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65491,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:18:48.413 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.548+0000 7f15b99a0640 1 -- 172.21.15.69:0/1502323795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b41005b0 msgr2=0x7f15b41009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.548+0000 7f15b99a0640 1 --2- 172.21.15.69:0/1502323795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b41005b0 0x7f15b41009d0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f15ac009900 tx=0x7f15ac02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- 172.21.15.69:0/1502323795 shutdown_connections 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- 172.21.15.69:0/1502323795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b4102160 0x7f15b41089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- 172.21.15.69:0/1502323795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41017a0 0x7f15b4101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- 172.21.15.69:0/1502323795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b41005b0 0x7f15b41009d0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- 172.21.15.69:0/1502323795 >> 172.21.15.69:0/1502323795 conn(0x7f15b40fbb00 msgr2=0x7f15b40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- 172.21.15.69:0/1502323795 shutdown_connections 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- 172.21.15.69:0/1502323795 wait complete. 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 Processor -- start 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- start start 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41005b0 0x7f15b4197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b41017a0 0x7f15b4197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410d3b0 con 0x7f15b4102160 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410d230 con 0x7f15b41017a0 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b99a0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15b410d530 con 0x7f15b41005b0 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b919f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b919f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40000/0 (socket says 172.21.15.69:40000) 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.549+0000 7f15b919f640 1 -- 172.21.15.69:0/3788403014 learned_addr learned my addr 172.21.15.69:0/3788403014 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b899e640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41005b0 0x7f15b4197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 -- 172.21.15.69:0/3788403014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41005b0 msgr2=0x7f15b4197a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41005b0 0x7f15b4197a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 -- 172.21.15.69:0/3788403014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b41017a0 msgr2=0x7f15b4197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b41017a0 0x7f15b4197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 -- 172.21.15.69:0/3788403014 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15b419f670 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b919f640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f159c009fd0 tx=0x7f159c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c011830 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f159c016070 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f15b419f960 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f159c00f2e0 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.550+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15b419fee0 con 0x7f15b4102160 2024-05-09T09:18:48.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.551+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f15b41078d0 con 0x7f15b4102160 2024-05-09T09:18:48.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.552+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f159c0119d0 con 0x7f15b4102160 2024-05-09T09:18:48.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.553+0000 7f15b1ffb640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 0x7f15a00a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.553+0000 7f15b899e640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 0x7f15a00a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.553+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f159c0ceb80 con 0x7f15b4102160 2024-05-09T09:18:48.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.553+0000 7f15b899e640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 0x7f15a00a50d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15ac006010 tx=0x7f15ac000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.558+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f159c09b700 con 0x7f15b4102160 2024-05-09T09:18:48.732 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.732+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7f15b4061c70 con 0x7f15a00a2c10 2024-05-09T09:18:48.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.741+0000 7f15b1ffb640 1 -- 172.21.15.69:0/3788403014 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340780 (secure 0 0 0) 0x7f15b4061c70 con 0x7f15a00a2c10 2024-05-09T09:18:48.746 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:48.748 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:18:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 msgr2=0x7f15a00a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 0x7f15a00a50d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f15ac006010 tx=0x7f15ac000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 msgr2=0x7f15b419f000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.751 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f159c009fd0 tx=0x7f159c00b8e0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 shutdown_connections 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15a00a2c10 0x7f15a00a50d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15b4102160 0x7f15b419f000 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15b41017a0 0x7f15b4197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 --2- 172.21.15.69:0/3788403014 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f15b41005b0 0x7f15b4197a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 >> 172.21.15.69:0/3788403014 conn(0x7f15b40fbb00 msgr2=0x7f15b4104aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 shutdown_connections 2024-05-09T09:18:48.752 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.745+0000 7f15b99a0640 1 -- 172.21.15.69:0/3788403014 wait complete. 2024-05-09T09:18:48.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/159002233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:48.755 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:48 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2696353069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:48.756 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-05-09T09:18:47.136643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658965,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":826,"num_read_kb":533,"num_write":356,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181771,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353312,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588512,"statfs":{"total":1151940427776,"available":1151578636288,"internally_reserved":0,"allocated":12738560,"data_stored":8185613,"data_compressed":430872,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":42,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164962+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:18:31.164962+0000","last_peered":"2024-05-09T09:18:31.164962+0000","last_clean":"2024-05-09T09:18:31.164962+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:18:31.164962+0000","last_undegraded":"2024-05-09T09:18:31.164962+0000","last_fullsized":"2024-05-09T09:18:31.164962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981364+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:18:30.981364+0000","last_peered":"2024-05-09T09:18:30.981364+0000","last_clean":"2024-05-09T09:18:30.981364+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:18:30.981364+0000","last_undegraded":"2024-05-09T09:18:30.981364+0000","last_fullsized":"2024-05-09T09:18:30.981364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981277+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:18:30.981277+0000","last_peered":"2024-05-09T09:18:30.981277+0000","last_clean":"2024-05-09T09:18:30.981277+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:18:30.981277+0000","last_undegraded":"2024-05-09T09:18:30.981277+0000","last_fullsized":"2024-05-09T09:18:30.981277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229778+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:18:30.229778+0000","last_peered":"2024-05-09T09:18:30.229778+0000","last_clean":"2024-05-09T09:18:30.229778+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:18:30.229778+0000","last_undegraded":"2024-05-09T09:18:30.229778+0000","last_fullsized":"2024-05-09T09:18:30.229778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512597+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:18:30.512597+0000","last_peered":"2024-05-09T09:18:30.512597+0000","last_clean":"2024-05-09T09:18:30.512597+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:18:30.512597+0000","last_undegraded":"2024-05-09T09:18:30.512597+0000","last_fullsized":"2024-05-09T09:18:30.512597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226905+0000","last_change":"2024-05-09T09:17:30.205057+0000","last_active":"2024-05-09T09:18:30.226905+0000","last_peered":"2024-05-09T09:18:30.226905+0000","last_clean":"2024-05-09T09:18:30.226905+0000","last_became_active":"2024-05-09T09:17:30.204736+0000","last_became_peered":"2024-05-09T09:17:30.204736+0000","last_unstale":"2024-05-09T09:18:30.226905+0000","last_undegraded":"2024-05-09T09:18:30.226905+0000","last_fullsized":"2024-05-09T09:18:30.226905+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:00.112812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492692+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:18:30.492692+0000","last_peered":"2024-05-09T09:18:30.492692+0000","last_clean":"2024-05-09T09:18:30.492692+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:18:30.492692+0000","last_undegraded":"2024-05-09T09:18:30.492692+0000","last_fullsized":"2024-05-09T09:18:30.492692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980080+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:18:30.980080+0000","last_peered":"2024-05-09T09:18:30.980080+0000","last_clean":"2024-05-09T09:18:30.980080+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:18:30.980080+0000","last_undegraded":"2024-05-09T09:18:30.980080+0000","last_fullsized":"2024-05-09T09:18:30.980080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980964+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:18:30.980964+0000","last_peered":"2024-05-09T09:18:30.980964+0000","last_clean":"2024-05-09T09:18:30.980964+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:18:30.980964+0000","last_undegraded":"2024-05-09T09:18:30.980964+0000","last_fullsized":"2024-05-09T09:18:30.980964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227506+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:18:30.227506+0000","last_peered":"2024-05-09T09:18:30.227506+0000","last_clean":"2024-05-09T09:18:30.227506+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:18:30.227506+0000","last_undegraded":"2024-05-09T09:18:30.227506+0000","last_fullsized":"2024-05-09T09:18:30.227506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493143+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:18:30.493143+0000","last_peered":"2024-05-09T09:18:30.493143+0000","last_clean":"2024-05-09T09:18:30.493143+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:18:30.493143+0000","last_undegraded":"2024-05-09T09:18:30.493143+0000","last_fullsized":"2024-05-09T09:18:30.493143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529489+0000","last_change":"2024-05-09T09:17:30.205228+0000","last_active":"2024-05-09T09:18:30.529489+0000","last_peered":"2024-05-09T09:18:30.529489+0000","last_clean":"2024-05-09T09:18:30.529489+0000","last_became_active":"2024-05-09T09:17:30.204824+0000","last_became_peered":"2024-05-09T09:17:30.204824+0000","last_unstale":"2024-05-09T09:18:30.529489+0000","last_undegraded":"2024-05-09T09:18:30.529489+0000","last_fullsized":"2024-05-09T09:18:30.529489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:28.158394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223560+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:18:30.223560+0000","last_peered":"2024-05-09T09:18:30.223560+0000","last_clean":"2024-05-09T09:18:30.223560+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:18:30.223560+0000","last_undegraded":"2024-05-09T09:18:30.223560+0000","last_fullsized":"2024-05-09T09:18:30.223560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980510+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:18:30.980510+0000","last_peered":"2024-05-09T09:18:30.980510+0000","last_clean":"2024-05-09T09:18:30.980510+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:18:30.980510+0000","last_undegraded":"2024-05-09T09:18:30.980510+0000","last_fullsized":"2024-05-09T09:18:30.980510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980927+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:18:30.980927+0000","last_peered":"2024-05-09T09:18:30.980927+0000","last_clean":"2024-05-09T09:18:30.980927+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:18:30.980927+0000","last_undegraded":"2024-05-09T09:18:30.980927+0000","last_fullsized":"2024-05-09T09:18:30.980927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980335+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:18:30.980335+0000","last_peered":"2024-05-09T09:18:30.980335+0000","last_clean":"2024-05-09T09:18:30.980335+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:18:30.980335+0000","last_undegraded":"2024-05-09T09:18:30.980335+0000","last_fullsized":"2024-05-09T09:18:30.980335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164757+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:18:31.164757+0000","last_peered":"2024-05-09T09:18:31.164757+0000","last_clean":"2024-05-09T09:18:31.164757+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:18:31.164757+0000","last_undegraded":"2024-05-09T09:18:31.164757+0000","last_fullsized":"2024-05-09T09:18:31.164757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529708+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:18:30.529708+0000","last_peered":"2024-05-09T09:18:30.529708+0000","last_clean":"2024-05-09T09:18:30.529708+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:18:30.529708+0000","last_undegraded":"2024-05-09T09:18:30.529708+0000","last_fullsized":"2024-05-09T09:18:30.529708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226663+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:18:30.226663+0000","last_peered":"2024-05-09T09:18:30.226663+0000","last_clean":"2024-05-09T09:18:30.226663+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:18:30.226663+0000","last_undegraded":"2024-05-09T09:18:30.226663+0000","last_fullsized":"2024-05-09T09:18:30.226663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226902+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:18:30.226902+0000","last_peered":"2024-05-09T09:18:30.226902+0000","last_clean":"2024-05-09T09:18:30.226902+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:18:30.226902+0000","last_undegraded":"2024-05-09T09:18:30.226902+0000","last_fullsized":"2024-05-09T09:18:30.226902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529614+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:18:30.529614+0000","last_peered":"2024-05-09T09:18:30.529614+0000","last_clean":"2024-05-09T09:18:30.529614+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:18:30.529614+0000","last_undegraded":"2024-05-09T09:18:30.529614+0000","last_fullsized":"2024-05-09T09:18:30.529614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980873+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:18:30.980873+0000","last_peered":"2024-05-09T09:18:30.980873+0000","last_clean":"2024-05-09T09:18:30.980873+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:18:30.980873+0000","last_undegraded":"2024-05-09T09:18:30.980873+0000","last_fullsized":"2024-05-09T09:18:30.980873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228966+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:18:30.228966+0000","last_peered":"2024-05-09T09:18:30.228966+0000","last_clean":"2024-05-09T09:18:30.228966+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:18:30.228966+0000","last_undegraded":"2024-05-09T09:18:30.228966+0000","last_fullsized":"2024-05-09T09:18:30.228966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981126+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:18:30.981126+0000","last_peered":"2024-05-09T09:18:30.981126+0000","last_clean":"2024-05-09T09:18:30.981126+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:18:30.981126+0000","last_undegraded":"2024-05-09T09:18:30.981126+0000","last_fullsized":"2024-05-09T09:18:30.981126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980521+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:18:30.980521+0000","last_peered":"2024-05-09T09:18:30.980521+0000","last_clean":"2024-05-09T09:18:30.980521+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:18:30.980521+0000","last_undegraded":"2024-05-09T09:18:30.980521+0000","last_fullsized":"2024-05-09T09:18:30.980521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229455+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:18:30.229455+0000","last_peered":"2024-05-09T09:18:30.229455+0000","last_clean":"2024-05-09T09:18:30.229455+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:18:30.229455+0000","last_undegraded":"2024-05-09T09:18:30.229455+0000","last_fullsized":"2024-05-09T09:18:30.229455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164671+0000","last_change":"2024-05-09T09:17:30.213383+0000","last_active":"2024-05-09T09:18:31.164671+0000","last_peered":"2024-05-09T09:18:31.164671+0000","last_clean":"2024-05-09T09:18:31.164671+0000","last_became_active":"2024-05-09T09:17:30.205007+0000","last_became_peered":"2024-05-09T09:17:30.205007+0000","last_unstale":"2024-05-09T09:18:31.164671+0000","last_undegraded":"2024-05-09T09:18:31.164671+0000","last_fullsized":"2024-05-09T09:18:31.164671+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:36.820105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980640+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:18:30.980640+0000","last_peered":"2024-05-09T09:18:30.980640+0000","last_clean":"2024-05-09T09:18:30.980640+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:18:30.980640+0000","last_undegraded":"2024-05-09T09:18:30.980640+0000","last_fullsized":"2024-05-09T09:18:30.980640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229018+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:18:30.229018+0000","last_peered":"2024-05-09T09:18:30.229018+0000","last_clean":"2024-05-09T09:18:30.229018+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:18:30.229018+0000","last_undegraded":"2024-05-09T09:18:30.229018+0000","last_fullsized":"2024-05-09T09:18:30.229018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512765+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:18:30.512765+0000","last_peered":"2024-05-09T09:18:30.512765+0000","last_clean":"2024-05-09T09:18:30.512765+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:18:30.512765+0000","last_undegraded":"2024-05-09T09:18:30.512765+0000","last_fullsized":"2024-05-09T09:18:30.512765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227762+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:18:30.227762+0000","last_peered":"2024-05-09T09:18:30.227762+0000","last_clean":"2024-05-09T09:18:30.227762+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:18:30.227762+0000","last_undegraded":"2024-05-09T09:18:30.227762+0000","last_fullsized":"2024-05-09T09:18:30.227762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226533+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:18:30.226533+0000","last_peered":"2024-05-09T09:18:30.226533+0000","last_clean":"2024-05-09T09:18:30.226533+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:18:30.226533+0000","last_undegraded":"2024-05-09T09:18:30.226533+0000","last_fullsized":"2024-05-09T09:18:30.226533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389904+0000","last_change":"2024-05-09T09:17:30.281412+0000","last_active":"2024-05-09T09:18:45.389904+0000","last_peered":"2024-05-09T09:18:45.389904+0000","last_clean":"2024-05-09T09:18:45.389904+0000","last_became_active":"2024-05-09T09:17:30.277253+0000","last_became_peered":"2024-05-09T09:17:30.277253+0000","last_unstale":"2024-05-09T09:18:45.389904+0000","last_undegraded":"2024-05-09T09:18:45.389904+0000","last_fullsized":"2024-05-09T09:18:45.389904+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:48.491205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512581+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:18:30.512581+0000","last_peered":"2024-05-09T09:18:30.512581+0000","last_clean":"2024-05-09T09:18:30.512581+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:18:30.512581+0000","last_undegraded":"2024-05-09T09:18:30.512581+0000","last_fullsized":"2024-05-09T09:18:30.512581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229789+0000","last_change":"2024-05-09T09:18:30.229789+0000","last_active":"2024-05-09T09:18:30.229789+0000","last_peered":"2024-05-09T09:18:30.229789+0000","last_clean":"2024-05-09T09:18:30.229789+0000","last_became_active":"2024-05-09T09:18:30.229281+0000","last_became_peered":"2024-05-09T09:18:30.229281+0000","last_unstale":"2024-05-09T09:18:30.229789+0000","last_undegraded":"2024-05-09T09:18:30.229789+0000","last_fullsized":"2024-05-09T09:18:30.229789+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:55.816495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226891+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:18:30.226891+0000","last_peered":"2024-05-09T09:18:30.226891+0000","last_clean":"2024-05-09T09:18:30.226891+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:18:30.226891+0000","last_undegraded":"2024-05-09T09:18:30.226891+0000","last_fullsized":"2024-05-09T09:18:30.226891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227826+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:18:30.227826+0000","last_peered":"2024-05-09T09:18:30.227826+0000","last_clean":"2024-05-09T09:18:30.227826+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:18:30.227826+0000","last_undegraded":"2024-05-09T09:18:30.227826+0000","last_fullsized":"2024-05-09T09:18:30.227826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164767+0000","last_change":"2024-05-09T09:17:30.277465+0000","last_active":"2024-05-09T09:18:31.164767+0000","last_peered":"2024-05-09T09:18:31.164767+0000","last_clean":"2024-05-09T09:18:31.164767+0000","last_became_active":"2024-05-09T09:17:30.277014+0000","last_became_peered":"2024-05-09T09:17:30.277014+0000","last_unstale":"2024-05-09T09:18:31.164767+0000","last_undegraded":"2024-05-09T09:18:31.164767+0000","last_fullsized":"2024-05-09T09:18:31.164767+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:36:04.346440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223244+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:18:30.223244+0000","last_peered":"2024-05-09T09:18:30.223244+0000","last_clean":"2024-05-09T09:18:30.223244+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:18:30.223244+0000","last_undegraded":"2024-05-09T09:18:30.223244+0000","last_fullsized":"2024-05-09T09:18:30.223244+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980233+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:18:30.980233+0000","last_peered":"2024-05-09T09:18:30.980233+0000","last_clean":"2024-05-09T09:18:30.980233+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:18:30.980233+0000","last_undegraded":"2024-05-09T09:18:30.980233+0000","last_fullsized":"2024-05-09T09:18:30.980233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981359+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:18:30.981359+0000","last_peered":"2024-05-09T09:18:30.981359+0000","last_clean":"2024-05-09T09:18:30.981359+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:18:30.981359+0000","last_undegraded":"2024-05-09T09:18:30.981359+0000","last_fullsized":"2024-05-09T09:18:30.981359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227863+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:18:30.227863+0000","last_peered":"2024-05-09T09:18:30.227863+0000","last_clean":"2024-05-09T09:18:30.227863+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:18:30.227863+0000","last_undegraded":"2024-05-09T09:18:30.227863+0000","last_fullsized":"2024-05-09T09:18:30.227863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529239+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:18:30.529239+0000","last_peered":"2024-05-09T09:18:30.529239+0000","last_clean":"2024-05-09T09:18:30.529239+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:18:30.529239+0000","last_undegraded":"2024-05-09T09:18:30.529239+0000","last_fullsized":"2024-05-09T09:18:30.529239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980320+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:18:30.980320+0000","last_peered":"2024-05-09T09:18:30.980320+0000","last_clean":"2024-05-09T09:18:30.980320+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:18:30.980320+0000","last_undegraded":"2024-05-09T09:18:30.980320+0000","last_fullsized":"2024-05-09T09:18:30.980320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389402+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:18:45.389402+0000","last_peered":"2024-05-09T09:18:45.389402+0000","last_clean":"2024-05-09T09:18:45.389402+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:18:45.389402+0000","last_undegraded":"2024-05-09T09:18:45.389402+0000","last_fullsized":"2024-05-09T09:18:45.389402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980894+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:18:30.980894+0000","last_peered":"2024-05-09T09:18:30.980894+0000","last_clean":"2024-05-09T09:18:30.980894+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:18:30.980894+0000","last_undegraded":"2024-05-09T09:18:30.980894+0000","last_fullsized":"2024-05-09T09:18:30.980894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529090+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:18:30.529090+0000","last_peered":"2024-05-09T09:18:30.529090+0000","last_clean":"2024-05-09T09:18:30.529090+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:18:30.529090+0000","last_undegraded":"2024-05-09T09:18:30.529090+0000","last_fullsized":"2024-05-09T09:18:30.529090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493132+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:18:30.493132+0000","last_peered":"2024-05-09T09:18:30.493132+0000","last_clean":"2024-05-09T09:18:30.493132+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:18:30.493132+0000","last_undegraded":"2024-05-09T09:18:30.493132+0000","last_fullsized":"2024-05-09T09:18:30.493132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980583+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:18:30.980583+0000","last_peered":"2024-05-09T09:18:30.980583+0000","last_clean":"2024-05-09T09:18:30.980583+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:18:30.980583+0000","last_undegraded":"2024-05-09T09:18:30.980583+0000","last_fullsized":"2024-05-09T09:18:30.980583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229610+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:18:30.229610+0000","last_peered":"2024-05-09T09:18:30.229610+0000","last_clean":"2024-05-09T09:18:30.229610+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:18:30.229610+0000","last_undegraded":"2024-05-09T09:18:30.229610+0000","last_fullsized":"2024-05-09T09:18:30.229610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227664+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:18:30.227664+0000","last_peered":"2024-05-09T09:18:30.227664+0000","last_clean":"2024-05-09T09:18:30.227664+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:18:30.227664+0000","last_undegraded":"2024-05-09T09:18:30.227664+0000","last_fullsized":"2024-05-09T09:18:30.227664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980767+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:18:30.980767+0000","last_peered":"2024-05-09T09:18:30.980767+0000","last_clean":"2024-05-09T09:18:30.980767+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:18:30.980767+0000","last_undegraded":"2024-05-09T09:18:30.980767+0000","last_fullsized":"2024-05-09T09:18:30.980767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529402+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:18:30.529402+0000","last_peered":"2024-05-09T09:18:30.529402+0000","last_clean":"2024-05-09T09:18:30.529402+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:18:30.529402+0000","last_undegraded":"2024-05-09T09:18:30.529402+0000","last_fullsized":"2024-05-09T09:18:30.529402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:46.390433+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:18:46.390433+0000","last_peered":"2024-05-09T09:18:46.390433+0000","last_clean":"2024-05-09T09:18:46.390433+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:18:46.390433+0000","last_undegraded":"2024-05-09T09:18:46.390433+0000","last_fullsized":"2024-05-09T09:18:46.390433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63913,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":181,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512855+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:18:30.512855+0000","last_peered":"2024-05-09T09:18:30.512855+0000","last_clean":"2024-05-09T09:18:30.512855+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:18:30.512855+0000","last_undegraded":"2024-05-09T09:18:30.512855+0000","last_fullsized":"2024-05-09T09:18:30.512855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228102+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:18:30.228102+0000","last_peered":"2024-05-09T09:18:30.228102+0000","last_clean":"2024-05-09T09:18:30.228102+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:18:30.228102+0000","last_undegraded":"2024-05-09T09:18:30.228102+0000","last_fullsized":"2024-05-09T09:18:30.228102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226401+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:18:30.226401+0000","last_peered":"2024-05-09T09:18:30.226401+0000","last_clean":"2024-05-09T09:18:30.226401+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:18:30.226401+0000","last_undegraded":"2024-05-09T09:18:30.226401+0000","last_fullsized":"2024-05-09T09:18:30.226401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223357+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:18:30.223357+0000","last_peered":"2024-05-09T09:18:30.223357+0000","last_clean":"2024-05-09T09:18:30.223357+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:18:30.223357+0000","last_undegraded":"2024-05-09T09:18:30.223357+0000","last_fullsized":"2024-05-09T09:18:30.223357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164421+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:18:31.164421+0000","last_peered":"2024-05-09T09:18:31.164421+0000","last_clean":"2024-05-09T09:18:31.164421+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:18:31.164421+0000","last_undegraded":"2024-05-09T09:18:31.164421+0000","last_fullsized":"2024-05-09T09:18:31.164421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227484+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:18:30.227484+0000","last_peered":"2024-05-09T09:18:30.227484+0000","last_clean":"2024-05-09T09:18:30.227484+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:18:30.227484+0000","last_undegraded":"2024-05-09T09:18:30.227484+0000","last_fullsized":"2024-05-09T09:18:30.227484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492851+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:18:30.492851+0000","last_peered":"2024-05-09T09:18:30.492851+0000","last_clean":"2024-05-09T09:18:30.492851+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:18:30.492851+0000","last_undegraded":"2024-05-09T09:18:30.492851+0000","last_fullsized":"2024-05-09T09:18:30.492851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513032+0000","last_change":"2024-05-09T09:17:30.465425+0000","last_active":"2024-05-09T09:18:30.513032+0000","last_peered":"2024-05-09T09:18:30.513032+0000","last_clean":"2024-05-09T09:18:30.513032+0000","last_became_active":"2024-05-09T09:17:30.465043+0000","last_became_peered":"2024-05-09T09:17:30.465043+0000","last_unstale":"2024-05-09T09:18:30.513032+0000","last_undegraded":"2024-05-09T09:18:30.513032+0000","last_fullsized":"2024-05-09T09:18:30.513032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:16:54.706845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:40.381710+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:18:40.381710+0000","last_peered":"2024-05-09T09:18:40.381710+0000","last_clean":"2024-05-09T09:18:40.381710+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:18:40.381710+0000","last_undegraded":"2024-05-09T09:18:40.381710+0000","last_fullsized":"2024-05-09T09:18:40.381710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492685+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:18:30.492685+0000","last_peered":"2024-05-09T09:18:30.492685+0000","last_clean":"2024-05-09T09:18:30.492685+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:18:30.492685+0000","last_undegraded":"2024-05-09T09:18:30.492685+0000","last_fullsized":"2024-05-09T09:18:30.492685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223712+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:18:30.223712+0000","last_peered":"2024-05-09T09:18:30.223712+0000","last_clean":"2024-05-09T09:18:30.223712+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:18:30.223712+0000","last_undegraded":"2024-05-09T09:18:30.223712+0000","last_fullsized":"2024-05-09T09:18:30.223712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512424+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:18:30.512424+0000","last_peered":"2024-05-09T09:18:30.512424+0000","last_clean":"2024-05-09T09:18:30.512424+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:18:30.512424+0000","last_undegraded":"2024-05-09T09:18:30.512424+0000","last_fullsized":"2024-05-09T09:18:30.512424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227187+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:18:30.227187+0000","last_peered":"2024-05-09T09:18:30.227187+0000","last_clean":"2024-05-09T09:18:30.227187+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:18:30.227187+0000","last_undegraded":"2024-05-09T09:18:30.227187+0000","last_fullsized":"2024-05-09T09:18:30.227187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227206+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:18:30.227206+0000","last_peered":"2024-05-09T09:18:30.227206+0000","last_clean":"2024-05-09T09:18:30.227206+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:18:30.227206+0000","last_undegraded":"2024-05-09T09:18:30.227206+0000","last_fullsized":"2024-05-09T09:18:30.227206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513190+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:18:30.513190+0000","last_peered":"2024-05-09T09:18:30.513190+0000","last_clean":"2024-05-09T09:18:30.513190+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:18:30.513190+0000","last_undegraded":"2024-05-09T09:18:30.513190+0000","last_fullsized":"2024-05-09T09:18:30.513190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164659+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:18:31.164659+0000","last_peered":"2024-05-09T09:18:31.164659+0000","last_clean":"2024-05-09T09:18:31.164659+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:18:31.164659+0000","last_undegraded":"2024-05-09T09:18:31.164659+0000","last_fullsized":"2024-05-09T09:18:31.164659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512820+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:18:30.512820+0000","last_peered":"2024-05-09T09:18:30.512820+0000","last_clean":"2024-05-09T09:18:30.512820+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:18:30.512820+0000","last_undegraded":"2024-05-09T09:18:30.512820+0000","last_fullsized":"2024-05-09T09:18:30.512820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529852+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:18:30.529852+0000","last_peered":"2024-05-09T09:18:30.529852+0000","last_clean":"2024-05-09T09:18:30.529852+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:18:30.529852+0000","last_undegraded":"2024-05-09T09:18:30.529852+0000","last_fullsized":"2024-05-09T09:18:30.529852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227403+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:18:30.227403+0000","last_peered":"2024-05-09T09:18:30.227403+0000","last_clean":"2024-05-09T09:18:30.227403+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:18:30.227403+0000","last_undegraded":"2024-05-09T09:18:30.227403+0000","last_fullsized":"2024-05-09T09:18:30.227403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223570+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:18:30.223570+0000","last_peered":"2024-05-09T09:18:30.223570+0000","last_clean":"2024-05-09T09:18:30.223570+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:18:30.223570+0000","last_undegraded":"2024-05-09T09:18:30.223570+0000","last_fullsized":"2024-05-09T09:18:30.223570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226975+0000","last_change":"2024-05-09T09:17:30.203699+0000","last_active":"2024-05-09T09:18:30.226975+0000","last_peered":"2024-05-09T09:18:30.226975+0000","last_clean":"2024-05-09T09:18:30.226975+0000","last_became_active":"2024-05-09T09:17:30.203312+0000","last_became_peered":"2024-05-09T09:17:30.203312+0000","last_unstale":"2024-05-09T09:18:30.226975+0000","last_undegraded":"2024-05-09T09:18:30.226975+0000","last_fullsized":"2024-05-09T09:18:30.226975+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529073+0000","last_change":"2024-05-09T09:17:30.276150+0000","last_active":"2024-05-09T09:18:30.529073+0000","last_peered":"2024-05-09T09:18:30.529073+0000","last_clean":"2024-05-09T09:18:30.529073+0000","last_became_active":"2024-05-09T09:17:30.275868+0000","last_became_peered":"2024-05-09T09:17:30.275868+0000","last_unstale":"2024-05-09T09:18:30.529073+0000","last_undegraded":"2024-05-09T09:18:30.529073+0000","last_fullsized":"2024-05-09T09:18:30.529073+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:18:08.892380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":145,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:41.382493+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:18:41.382493+0000","last_peered":"2024-05-09T09:18:41.382493+0000","last_clean":"2024-05-09T09:18:41.382493+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:18:41.382493+0000","last_undegraded":"2024-05-09T09:18:41.382493+0000","last_fullsized":"2024-05-09T09:18:41.382493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493069+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:18:30.493069+0000","last_peered":"2024-05-09T09:18:30.493069+0000","last_clean":"2024-05-09T09:18:30.493069+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:18:30.493069+0000","last_undegraded":"2024-05-09T09:18:30.493069+0000","last_fullsized":"2024-05-09T09:18:30.493069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980587+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:18:30.980587+0000","last_peered":"2024-05-09T09:18:30.980587+0000","last_clean":"2024-05-09T09:18:30.980587+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:18:30.980587+0000","last_undegraded":"2024-05-09T09:18:30.980587+0000","last_fullsized":"2024-05-09T09:18:30.980587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229495+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:18:30.229495+0000","last_peered":"2024-05-09T09:18:30.229495+0000","last_clean":"2024-05-09T09:18:30.229495+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:18:30.229495+0000","last_undegraded":"2024-05-09T09:18:30.229495+0000","last_fullsized":"2024-05-09T09:18:30.229495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492528+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:18:30.492528+0000","last_peered":"2024-05-09T09:18:30.492528+0000","last_clean":"2024-05-09T09:18:30.492528+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:18:30.492528+0000","last_undegraded":"2024-05-09T09:18:30.492528+0000","last_fullsized":"2024-05-09T09:18:30.492528+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227346+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:18:30.227346+0000","last_peered":"2024-05-09T09:18:30.227346+0000","last_clean":"2024-05-09T09:18:30.227346+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:18:30.227346+0000","last_undegraded":"2024-05-09T09:18:30.227346+0000","last_fullsized":"2024-05-09T09:18:30.227346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223208+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:18:30.223208+0000","last_peered":"2024-05-09T09:18:30.223208+0000","last_clean":"2024-05-09T09:18:30.223208+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:18:30.223208+0000","last_undegraded":"2024-05-09T09:18:30.223208+0000","last_fullsized":"2024-05-09T09:18:30.223208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492715+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:18:30.492715+0000","last_peered":"2024-05-09T09:18:30.492715+0000","last_clean":"2024-05-09T09:18:30.492715+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:18:30.492715+0000","last_undegraded":"2024-05-09T09:18:30.492715+0000","last_fullsized":"2024-05-09T09:18:30.492715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164409+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:18:31.164409+0000","last_peered":"2024-05-09T09:18:31.164409+0000","last_clean":"2024-05-09T09:18:31.164409+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:18:31.164409+0000","last_undegraded":"2024-05-09T09:18:31.164409+0000","last_fullsized":"2024-05-09T09:18:31.164409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164541+0000","last_change":"2024-05-09T09:17:30.208053+0000","last_active":"2024-05-09T09:18:31.164541+0000","last_peered":"2024-05-09T09:18:31.164541+0000","last_clean":"2024-05-09T09:18:31.164541+0000","last_became_active":"2024-05-09T09:17:30.203628+0000","last_became_peered":"2024-05-09T09:17:30.203628+0000","last_unstale":"2024-05-09T09:18:31.164541+0000","last_undegraded":"2024-05-09T09:18:31.164541+0000","last_fullsized":"2024-05-09T09:18:31.164541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:37.871338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980275+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:18:30.980275+0000","last_peered":"2024-05-09T09:18:30.980275+0000","last_clean":"2024-05-09T09:18:30.980275+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:18:30.980275+0000","last_undegraded":"2024-05-09T09:18:30.980275+0000","last_fullsized":"2024-05-09T09:18:30.980275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'10","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:43.881851+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:18:43.881851+0000","last_peered":"2024-05-09T09:18:43.881851+0000","last_clean":"2024-05-09T09:18:43.881851+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:18:43.881851+0000","last_undegraded":"2024-05-09T09:18:43.881851+0000","last_fullsized":"2024-05-09T09:18:43.881851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981095+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:18:30.981095+0000","last_peered":"2024-05-09T09:18:30.981095+0000","last_clean":"2024-05-09T09:18:30.981095+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:18:30.981095+0000","last_undegraded":"2024-05-09T09:18:30.981095+0000","last_fullsized":"2024-05-09T09:18:30.981095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226060+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:18:30.226060+0000","last_peered":"2024-05-09T09:18:30.226060+0000","last_clean":"2024-05-09T09:18:30.226060+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:18:30.226060+0000","last_undegraded":"2024-05-09T09:18:30.226060+0000","last_fullsized":"2024-05-09T09:18:30.226060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229540+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:18:30.229540+0000","last_peered":"2024-05-09T09:18:30.229540+0000","last_clean":"2024-05-09T09:18:30.229540+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:18:30.229540+0000","last_undegraded":"2024-05-09T09:18:30.229540+0000","last_fullsized":"2024-05-09T09:18:30.229540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981427+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:18:30.981427+0000","last_peered":"2024-05-09T09:18:30.981427+0000","last_clean":"2024-05-09T09:18:30.981427+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:18:30.981427+0000","last_undegraded":"2024-05-09T09:18:30.981427+0000","last_fullsized":"2024-05-09T09:18:30.981427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.530637+0000","last_change":"2024-05-09T09:18:30.530637+0000","last_active":"2024-05-09T09:18:30.530637+0000","last_peered":"2024-05-09T09:18:30.530637+0000","last_clean":"2024-05-09T09:18:30.530637+0000","last_became_active":"2024-05-09T09:18:30.530213+0000","last_became_peered":"2024-05-09T09:18:30.530213+0000","last_unstale":"2024-05-09T09:18:30.530637+0000","last_undegraded":"2024-05-09T09:18:30.530637+0000","last_fullsized":"2024-05-09T09:18:30.530637+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:58:48.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980453+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:18:30.980453+0000","last_peered":"2024-05-09T09:18:30.980453+0000","last_clean":"2024-05-09T09:18:30.980453+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:18:30.980453+0000","last_undegraded":"2024-05-09T09:18:30.980453+0000","last_fullsized":"2024-05-09T09:18:30.980453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512512+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:18:30.512512+0000","last_peered":"2024-05-09T09:18:30.512512+0000","last_clean":"2024-05-09T09:18:30.512512+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:18:30.512512+0000","last_undegraded":"2024-05-09T09:18:30.512512+0000","last_fullsized":"2024-05-09T09:18:30.512512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529569+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:18:30.529569+0000","last_peered":"2024-05-09T09:18:30.529569+0000","last_clean":"2024-05-09T09:18:30.529569+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:18:30.529569+0000","last_undegraded":"2024-05-09T09:18:30.529569+0000","last_fullsized":"2024-05-09T09:18:30.529569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980366+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:18:30.980366+0000","last_peered":"2024-05-09T09:18:30.980366+0000","last_clean":"2024-05-09T09:18:30.980366+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:18:30.980366+0000","last_undegraded":"2024-05-09T09:18:30.980366+0000","last_fullsized":"2024-05-09T09:18:30.980366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164377+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:18:31.164377+0000","last_peered":"2024-05-09T09:18:31.164377+0000","last_clean":"2024-05-09T09:18:31.164377+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:18:31.164377+0000","last_undegraded":"2024-05-09T09:18:31.164377+0000","last_fullsized":"2024-05-09T09:18:31.164377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229286+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:18:30.229286+0000","last_peered":"2024-05-09T09:18:30.229286+0000","last_clean":"2024-05-09T09:18:30.229286+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:18:30.229286+0000","last_undegraded":"2024-05-09T09:18:30.229286+0000","last_fullsized":"2024-05-09T09:18:30.229286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226564+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:18:30.226564+0000","last_peered":"2024-05-09T09:18:30.226564+0000","last_clean":"2024-05-09T09:18:30.226564+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:18:30.226564+0000","last_undegraded":"2024-05-09T09:18:30.226564+0000","last_fullsized":"2024-05-09T09:18:30.226564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980483+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:18:30.980483+0000","last_peered":"2024-05-09T09:18:30.980483+0000","last_clean":"2024-05-09T09:18:30.980483+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:18:30.980483+0000","last_undegraded":"2024-05-09T09:18:30.980483+0000","last_fullsized":"2024-05-09T09:18:30.980483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229268+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:18:30.229268+0000","last_peered":"2024-05-09T09:18:30.229268+0000","last_clean":"2024-05-09T09:18:30.229268+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:18:30.229268+0000","last_undegraded":"2024-05-09T09:18:30.229268+0000","last_fullsized":"2024-05-09T09:18:30.229268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512609+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:18:30.512609+0000","last_peered":"2024-05-09T09:18:30.512609+0000","last_clean":"2024-05-09T09:18:30.512609+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:18:30.512609+0000","last_undegraded":"2024-05-09T09:18:30.512609+0000","last_fullsized":"2024-05-09T09:18:30.512609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980379+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:18:30.980379+0000","last_peered":"2024-05-09T09:18:30.980379+0000","last_clean":"2024-05-09T09:18:30.980379+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:18:30.980379+0000","last_undegraded":"2024-05-09T09:18:30.980379+0000","last_fullsized":"2024-05-09T09:18:30.980379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227058+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:18:30.227058+0000","last_peered":"2024-05-09T09:18:30.227058+0000","last_clean":"2024-05-09T09:18:30.227058+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:18:30.227058+0000","last_undegraded":"2024-05-09T09:18:30.227058+0000","last_fullsized":"2024-05-09T09:18:30.227058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493221+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:18:30.493221+0000","last_peered":"2024-05-09T09:18:30.493221+0000","last_clean":"2024-05-09T09:18:30.493221+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:18:30.493221+0000","last_undegraded":"2024-05-09T09:18:30.493221+0000","last_fullsized":"2024-05-09T09:18:30.493221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980643+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:18:30.980643+0000","last_peered":"2024-05-09T09:18:30.980643+0000","last_clean":"2024-05-09T09:18:30.980643+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:18:30.980643+0000","last_undegraded":"2024-05-09T09:18:30.980643+0000","last_fullsized":"2024-05-09T09:18:30.980643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227912+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:18:30.227912+0000","last_peered":"2024-05-09T09:18:30.227912+0000","last_clean":"2024-05-09T09:18:30.227912+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:18:30.227912+0000","last_undegraded":"2024-05-09T09:18:30.227912+0000","last_fullsized":"2024-05-09T09:18:30.227912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492929+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:18:30.492929+0000","last_peered":"2024-05-09T09:18:30.492929+0000","last_clean":"2024-05-09T09:18:30.492929+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:18:30.492929+0000","last_undegraded":"2024-05-09T09:18:30.492929+0000","last_fullsized":"2024-05-09T09:18:30.492929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227073+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:18:30.227073+0000","last_peered":"2024-05-09T09:18:30.227073+0000","last_clean":"2024-05-09T09:18:30.227073+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:18:30.227073+0000","last_undegraded":"2024-05-09T09:18:30.227073+0000","last_fullsized":"2024-05-09T09:18:30.227073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223662+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:18:30.223662+0000","last_peered":"2024-05-09T09:18:30.223662+0000","last_clean":"2024-05-09T09:18:30.223662+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:18:30.223662+0000","last_undegraded":"2024-05-09T09:18:30.223662+0000","last_fullsized":"2024-05-09T09:18:30.223662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493231+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:18:30.493231+0000","last_peered":"2024-05-09T09:18:30.493231+0000","last_clean":"2024-05-09T09:18:30.493231+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:18:30.493231+0000","last_undegraded":"2024-05-09T09:18:30.493231+0000","last_fullsized":"2024-05-09T09:18:30.493231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164450+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:18:31.164450+0000","last_peered":"2024-05-09T09:18:31.164450+0000","last_clean":"2024-05-09T09:18:31.164450+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:18:31.164450+0000","last_undegraded":"2024-05-09T09:18:31.164450+0000","last_fullsized":"2024-05-09T09:18:31.164450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227152+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:18:30.227152+0000","last_peered":"2024-05-09T09:18:30.227152+0000","last_clean":"2024-05-09T09:18:30.227152+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:18:30.227152+0000","last_undegraded":"2024-05-09T09:18:30.227152+0000","last_fullsized":"2024-05-09T09:18:30.227152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226406+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:18:30.226406+0000","last_peered":"2024-05-09T09:18:30.226406+0000","last_clean":"2024-05-09T09:18:30.226406+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:18:30.226406+0000","last_undegraded":"2024-05-09T09:18:30.226406+0000","last_fullsized":"2024-05-09T09:18:30.226406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980983+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:18:30.980983+0000","last_peered":"2024-05-09T09:18:30.980983+0000","last_clean":"2024-05-09T09:18:30.980983+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:18:30.980983+0000","last_undegraded":"2024-05-09T09:18:30.980983+0000","last_fullsized":"2024-05-09T09:18:30.980983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223725+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:18:30.223725+0000","last_peered":"2024-05-09T09:18:30.223725+0000","last_clean":"2024-05-09T09:18:30.223725+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:18:30.223725+0000","last_undegraded":"2024-05-09T09:18:30.223725+0000","last_fullsized":"2024-05-09T09:18:30.223725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164624+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:18:31.164624+0000","last_peered":"2024-05-09T09:18:31.164624+0000","last_clean":"2024-05-09T09:18:31.164624+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:18:31.164624+0000","last_undegraded":"2024-05-09T09:18:31.164624+0000","last_fullsized":"2024-05-09T09:18:31.164624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227381+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:18:30.227381+0000","last_peered":"2024-05-09T09:18:30.227381+0000","last_clean":"2024-05-09T09:18:30.227381+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:18:30.227381+0000","last_undegraded":"2024-05-09T09:18:30.227381+0000","last_fullsized":"2024-05-09T09:18:30.227381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981199+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:18:30.981199+0000","last_peered":"2024-05-09T09:18:30.981199+0000","last_clean":"2024-05-09T09:18:30.981199+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:18:30.981199+0000","last_undegraded":"2024-05-09T09:18:30.981199+0000","last_fullsized":"2024-05-09T09:18:30.981199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228022+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:18:30.228022+0000","last_peered":"2024-05-09T09:18:30.228022+0000","last_clean":"2024-05-09T09:18:30.228022+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:18:30.228022+0000","last_undegraded":"2024-05-09T09:18:30.228022+0000","last_fullsized":"2024-05-09T09:18:30.228022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980884+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:18:30.980884+0000","last_peered":"2024-05-09T09:18:30.980884+0000","last_clean":"2024-05-09T09:18:30.980884+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:18:30.980884+0000","last_undegraded":"2024-05-09T09:18:30.980884+0000","last_fullsized":"2024-05-09T09:18:30.980884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980558+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:18:30.980558+0000","last_peered":"2024-05-09T09:18:30.980558+0000","last_clean":"2024-05-09T09:18:30.980558+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:18:30.980558+0000","last_undegraded":"2024-05-09T09:18:30.980558+0000","last_fullsized":"2024-05-09T09:18:30.980558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223438+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:18:30.223438+0000","last_peered":"2024-05-09T09:18:30.223438+0000","last_clean":"2024-05-09T09:18:30.223438+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:18:30.223438+0000","last_undegraded":"2024-05-09T09:18:30.223438+0000","last_fullsized":"2024-05-09T09:18:30.223438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164557+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:18:31.164557+0000","last_peered":"2024-05-09T09:18:31.164557+0000","last_clean":"2024-05-09T09:18:31.164557+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:18:31.164557+0000","last_undegraded":"2024-05-09T09:18:31.164557+0000","last_fullsized":"2024-05-09T09:18:31.164557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68363,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":700,"num_read_kb":424,"num_write":173,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1035,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":209538,"data_compressed":4887,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":519498,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1170657,"data_compressed":63914,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523808,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":519610,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115682,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":514888,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":578403,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516276,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":579439,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515082,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":520692,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1111578,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63605,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70014,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65491,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:18:48.763 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-05-09T09:18:48.763 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-05-09T09:18:48.900 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 -- 172.21.15.69:0/3294463398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c107650 msgr2=0x7fa26c109a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 --2- 172.21.15.69:0/3294463398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c107650 0x7fa26c109a40 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa264009ca0 tx=0x7fa26402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 -- 172.21.15.69:0/3294463398 shutdown_connections 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 --2- 172.21.15.69:0/3294463398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c107650 0x7fa26c109a40 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 --2- 172.21.15.69:0/3294463398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 0x7fa26c107110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 --2- 172.21.15.69:0/3294463398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c1023f0 0x7fa26c1047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 -- 172.21.15.69:0/3294463398 >> 172.21.15.69:0/3294463398 conn(0x7fa26c0fbd20 msgr2=0x7fa26c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 -- 172.21.15.69:0/3294463398 shutdown_connections 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.900+0000 7fa270e38640 1 -- 172.21.15.69:0/3294463398 wait complete. 2024-05-09T09:18:48.901 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 Processor -- start 2024-05-09T09:18:48.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 -- start start 2024-05-09T09:18:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 0x7fa26c198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c107650 0x7fa26c19f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c10d560 con 0x7fa26c1023f0 2024-05-09T09:18:48.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c10d3e0 con 0x7fa26c107650 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa270e38640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa26c10d6e0 con 0x7fa26c104d20 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40016/0 (socket says 172.21.15.69:40016) 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 -- 172.21.15.69:0/2134331270 learned_addr learned my addr 172.21.15.69:0/2134331270 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa261d9b640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 0x7fa26c198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26ad9d640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c107650 0x7fa26c19f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 -- 172.21.15.69:0/2134331270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 msgr2=0x7fa26c198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 0x7fa26c198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 -- 172.21.15.69:0/2134331270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c107650 msgr2=0x7fa26c19f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c107650 0x7fa26c19f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.901+0000 7fa26a59c640 1 -- 172.21.15.69:0/2134331270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa26c19f800 con 0x7fa26c1023f0 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa26a59c640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00d950 tx=0x7fa25c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c017070 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa26c19faf0 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa25c009c30 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa25c00f6b0 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.902+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa26c1a0040 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.903+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fa26c0fd810 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.904+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa25c0109d0 con 0x7fa26c1023f0 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.905+0000 7fa263fff640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 0x7fa2580a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:48.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.905+0000 7fa261d9b640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 0x7fa2580a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:48.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.905+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa25c0cf030 con 0x7fa26c1023f0 2024-05-09T09:18:48.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.905+0000 7fa261d9b640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 0x7fa2580a5050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa26c1990f0 tx=0x7fa25400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:48.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:48.909+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fa25c009de0 con 0x7fa26c1023f0 2024-05-09T09:18:49.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/159002233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:49.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:48 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2696353069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:49.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.086+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 --> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) -- 0x7fa26c061c70 con 0x7fa2580a2b90 2024-05-09T09:18:49.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.095+0000 7fa263fff640 1 -- 172.21.15.69:0/2134331270 <== mgr.14152 v2:172.21.15.69:6800/3550836912 1 ==== mgr_command_reply(tid 0: 0 dumped all) ==== 18+0+340780 (secure 0 0 0) 0x7fa26c061c70 con 0x7fa2580a2b90 2024-05-09T09:18:49.102 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:49.104 INFO:teuthology.orchestra.run.smithi069.stderr:dumped all 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 msgr2=0x7fa2580a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 0x7fa2580a5050 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fa26c1990f0 tx=0x7fa25400b390 comp rx=0 tx=0).stop 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 msgr2=0x7fa26c197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa25c00d950 tx=0x7fa25c00de10 comp rx=0 tx=0).stop 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 shutdown_connections 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa2580a2b90 0x7fa2580a5050 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.107 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa26c107650 0x7fa26c19f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa26c104d20 0x7fa26c198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 --2- 172.21.15.69:0/2134331270 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa26c1023f0 0x7fa26c197be0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 >> 172.21.15.69:0/2134331270 conn(0x7fa26c0fbd20 msgr2=0x7fa26c102d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 shutdown_connections 2024-05-09T09:18:49.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.099+0000 7fa270e38640 1 -- 172.21.15.69:0/2134331270 wait complete. 2024-05-09T09:18:49.111 INFO:teuthology.orchestra.run.smithi069.stdout:{"pg_ready":true,"pg_map":{"version":362,"stamp":"2024-05-09T09:18:47.136643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":658965,"num_objects":64,"num_object_clones":0,"num_object_copies":194,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":64,"num_whiteouts":0,"num_read":826,"num_read_kb":533,"num_write":356,"num_write_kb":3065,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1181771,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":789,"ondisk_log_size":789,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":353312,"kb_used_data":12440,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124588512,"statfs":{"total":1151940427776,"available":1151578636288,"internally_reserved":0,"allocated":12738560,"data_stored":8185613,"data_compressed":430872,"data_compressed_allocated":589824,"data_compressed_original":2678784,"omap_allocated":19138,"internal_metadata":348894526},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":616,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":82,"num_read_kb":42,"num_write":6,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002885"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164962+0000","last_change":"2024-05-09T09:16:32.883968+0000","last_active":"2024-05-09T09:18:31.164962+0000","last_peered":"2024-05-09T09:18:31.164962+0000","last_clean":"2024-05-09T09:18:31.164962+0000","last_became_active":"2024-05-09T09:16:32.883102+0000","last_became_peered":"2024-05-09T09:16:32.883102+0000","last_unstale":"2024-05-09T09:18:31.164962+0000","last_undegraded":"2024-05-09T09:18:31.164962+0000","last_fullsized":"2024-05-09T09:18:31.164962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:07:01.609073+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981364+0000","last_change":"2024-05-09T09:16:32.888064+0000","last_active":"2024-05-09T09:18:30.981364+0000","last_peered":"2024-05-09T09:18:30.981364+0000","last_clean":"2024-05-09T09:18:30.981364+0000","last_became_active":"2024-05-09T09:16:32.887736+0000","last_became_peered":"2024-05-09T09:16:32.887736+0000","last_unstale":"2024-05-09T09:18:30.981364+0000","last_undegraded":"2024-05-09T09:18:30.981364+0000","last_fullsized":"2024-05-09T09:18:30.981364+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:59:27.689901+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981277+0000","last_change":"2024-05-09T09:16:32.888602+0000","last_active":"2024-05-09T09:18:30.981277+0000","last_peered":"2024-05-09T09:18:30.981277+0000","last_clean":"2024-05-09T09:18:30.981277+0000","last_became_active":"2024-05-09T09:16:32.884511+0000","last_became_peered":"2024-05-09T09:16:32.884511+0000","last_unstale":"2024-05-09T09:18:30.981277+0000","last_undegraded":"2024-05-09T09:18:30.981277+0000","last_fullsized":"2024-05-09T09:18:30.981277+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:45:18.070367+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229778+0000","last_change":"2024-05-09T09:16:32.889674+0000","last_active":"2024-05-09T09:18:30.229778+0000","last_peered":"2024-05-09T09:18:30.229778+0000","last_clean":"2024-05-09T09:18:30.229778+0000","last_became_active":"2024-05-09T09:16:32.889360+0000","last_became_peered":"2024-05-09T09:16:32.889360+0000","last_unstale":"2024-05-09T09:18:30.229778+0000","last_undegraded":"2024-05-09T09:18:30.229778+0000","last_fullsized":"2024-05-09T09:18:30.229778+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:25.364579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512597+0000","last_change":"2024-05-09T09:16:32.881690+0000","last_active":"2024-05-09T09:18:30.512597+0000","last_peered":"2024-05-09T09:18:30.512597+0000","last_clean":"2024-05-09T09:18:30.512597+0000","last_became_active":"2024-05-09T09:16:32.881356+0000","last_became_peered":"2024-05-09T09:16:32.881356+0000","last_unstale":"2024-05-09T09:18:30.512597+0000","last_undegraded":"2024-05-09T09:18:30.512597+0000","last_fullsized":"2024-05-09T09:18:30.512597+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:31:26.175610+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226905+0000","last_change":"2024-05-09T09:17:30.205057+0000","last_active":"2024-05-09T09:18:30.226905+0000","last_peered":"2024-05-09T09:18:30.226905+0000","last_clean":"2024-05-09T09:18:30.226905+0000","last_became_active":"2024-05-09T09:17:30.204736+0000","last_became_peered":"2024-05-09T09:17:30.204736+0000","last_unstale":"2024-05-09T09:18:30.226905+0000","last_undegraded":"2024-05-09T09:18:30.226905+0000","last_fullsized":"2024-05-09T09:18:30.226905+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:35:00.112812+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492692+0000","last_change":"2024-05-09T09:16:32.885153+0000","last_active":"2024-05-09T09:18:30.492692+0000","last_peered":"2024-05-09T09:18:30.492692+0000","last_clean":"2024-05-09T09:18:30.492692+0000","last_became_active":"2024-05-09T09:16:32.884780+0000","last_became_peered":"2024-05-09T09:16:32.884780+0000","last_unstale":"2024-05-09T09:18:30.492692+0000","last_undegraded":"2024-05-09T09:18:30.492692+0000","last_fullsized":"2024-05-09T09:18:30.492692+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:35.855640+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980080+0000","last_change":"2024-05-09T09:16:32.881672+0000","last_active":"2024-05-09T09:18:30.980080+0000","last_peered":"2024-05-09T09:18:30.980080+0000","last_clean":"2024-05-09T09:18:30.980080+0000","last_became_active":"2024-05-09T09:16:32.881340+0000","last_became_peered":"2024-05-09T09:16:32.881340+0000","last_unstale":"2024-05-09T09:18:30.980080+0000","last_undegraded":"2024-05-09T09:18:30.980080+0000","last_fullsized":"2024-05-09T09:18:30.980080+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:30:30.374218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980964+0000","last_change":"2024-05-09T09:16:32.870640+0000","last_active":"2024-05-09T09:18:30.980964+0000","last_peered":"2024-05-09T09:18:30.980964+0000","last_clean":"2024-05-09T09:18:30.980964+0000","last_became_active":"2024-05-09T09:16:32.870314+0000","last_became_peered":"2024-05-09T09:16:32.870314+0000","last_unstale":"2024-05-09T09:18:30.980964+0000","last_undegraded":"2024-05-09T09:18:30.980964+0000","last_fullsized":"2024-05-09T09:18:30.980964+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:14:26.296623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227506+0000","last_change":"2024-05-09T09:16:32.880115+0000","last_active":"2024-05-09T09:18:30.227506+0000","last_peered":"2024-05-09T09:18:30.227506+0000","last_clean":"2024-05-09T09:18:30.227506+0000","last_became_active":"2024-05-09T09:16:32.879788+0000","last_became_peered":"2024-05-09T09:16:32.879788+0000","last_unstale":"2024-05-09T09:18:30.227506+0000","last_undegraded":"2024-05-09T09:18:30.227506+0000","last_fullsized":"2024-05-09T09:18:30.227506+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:36:36.244523+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493143+0000","last_change":"2024-05-09T09:16:32.891209+0000","last_active":"2024-05-09T09:18:30.493143+0000","last_peered":"2024-05-09T09:18:30.493143+0000","last_clean":"2024-05-09T09:18:30.493143+0000","last_became_active":"2024-05-09T09:16:32.888139+0000","last_became_peered":"2024-05-09T09:16:32.888139+0000","last_unstale":"2024-05-09T09:18:30.493143+0000","last_undegraded":"2024-05-09T09:18:30.493143+0000","last_fullsized":"2024-05-09T09:18:30.493143+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:20:42.203139+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529489+0000","last_change":"2024-05-09T09:17:30.205228+0000","last_active":"2024-05-09T09:18:30.529489+0000","last_peered":"2024-05-09T09:18:30.529489+0000","last_clean":"2024-05-09T09:18:30.529489+0000","last_became_active":"2024-05-09T09:17:30.204824+0000","last_became_peered":"2024-05-09T09:17:30.204824+0000","last_unstale":"2024-05-09T09:18:30.529489+0000","last_undegraded":"2024-05-09T09:18:30.529489+0000","last_fullsized":"2024-05-09T09:18:30.529489+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:06:28.158394+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,10,8],"acting":[9,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223560+0000","last_change":"2024-05-09T09:16:32.877266+0000","last_active":"2024-05-09T09:18:30.223560+0000","last_peered":"2024-05-09T09:18:30.223560+0000","last_clean":"2024-05-09T09:18:30.223560+0000","last_became_active":"2024-05-09T09:16:32.876946+0000","last_became_peered":"2024-05-09T09:16:32.876946+0000","last_unstale":"2024-05-09T09:18:30.223560+0000","last_undegraded":"2024-05-09T09:18:30.223560+0000","last_fullsized":"2024-05-09T09:18:30.223560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:44:48.970124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980510+0000","last_change":"2024-05-09T09:16:32.892447+0000","last_active":"2024-05-09T09:18:30.980510+0000","last_peered":"2024-05-09T09:18:30.980510+0000","last_clean":"2024-05-09T09:18:30.980510+0000","last_became_active":"2024-05-09T09:16:32.888761+0000","last_became_peered":"2024-05-09T09:16:32.888761+0000","last_unstale":"2024-05-09T09:18:30.980510+0000","last_undegraded":"2024-05-09T09:18:30.980510+0000","last_fullsized":"2024-05-09T09:18:30.980510+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:09:27.242327+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980927+0000","last_change":"2024-05-09T09:16:32.893861+0000","last_active":"2024-05-09T09:18:30.980927+0000","last_peered":"2024-05-09T09:18:30.980927+0000","last_clean":"2024-05-09T09:18:30.980927+0000","last_became_active":"2024-05-09T09:16:32.893511+0000","last_became_peered":"2024-05-09T09:16:32.893511+0000","last_unstale":"2024-05-09T09:18:30.980927+0000","last_undegraded":"2024-05-09T09:18:30.980927+0000","last_fullsized":"2024-05-09T09:18:30.980927+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:40:37.779227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980335+0000","last_change":"2024-05-09T09:16:32.880736+0000","last_active":"2024-05-09T09:18:30.980335+0000","last_peered":"2024-05-09T09:18:30.980335+0000","last_clean":"2024-05-09T09:18:30.980335+0000","last_became_active":"2024-05-09T09:16:32.880118+0000","last_became_peered":"2024-05-09T09:16:32.880118+0000","last_unstale":"2024-05-09T09:18:30.980335+0000","last_undegraded":"2024-05-09T09:18:30.980335+0000","last_fullsized":"2024-05-09T09:18:30.980335+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:42:28.672974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164757+0000","last_change":"2024-05-09T09:16:32.887328+0000","last_active":"2024-05-09T09:18:31.164757+0000","last_peered":"2024-05-09T09:18:31.164757+0000","last_clean":"2024-05-09T09:18:31.164757+0000","last_became_active":"2024-05-09T09:16:32.883580+0000","last_became_peered":"2024-05-09T09:16:32.883580+0000","last_unstale":"2024-05-09T09:18:31.164757+0000","last_undegraded":"2024-05-09T09:18:31.164757+0000","last_fullsized":"2024-05-09T09:18:31.164757+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:43:59.715282+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529708+0000","last_change":"2024-05-09T09:16:32.885739+0000","last_active":"2024-05-09T09:18:30.529708+0000","last_peered":"2024-05-09T09:18:30.529708+0000","last_clean":"2024-05-09T09:18:30.529708+0000","last_became_active":"2024-05-09T09:16:32.885345+0000","last_became_peered":"2024-05-09T09:16:32.885345+0000","last_unstale":"2024-05-09T09:18:30.529708+0000","last_undegraded":"2024-05-09T09:18:30.529708+0000","last_fullsized":"2024-05-09T09:18:30.529708+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:47:28.397954+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226663+0000","last_change":"2024-05-09T09:16:32.885592+0000","last_active":"2024-05-09T09:18:30.226663+0000","last_peered":"2024-05-09T09:18:30.226663+0000","last_clean":"2024-05-09T09:18:30.226663+0000","last_became_active":"2024-05-09T09:16:32.882748+0000","last_became_peered":"2024-05-09T09:16:32.882748+0000","last_unstale":"2024-05-09T09:18:30.226663+0000","last_undegraded":"2024-05-09T09:18:30.226663+0000","last_fullsized":"2024-05-09T09:18:30.226663+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:19:08.484352+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226902+0000","last_change":"2024-05-09T09:16:36.921344+0000","last_active":"2024-05-09T09:18:30.226902+0000","last_peered":"2024-05-09T09:18:30.226902+0000","last_clean":"2024-05-09T09:18:30.226902+0000","last_became_active":"2024-05-09T09:16:36.920987+0000","last_became_peered":"2024-05-09T09:16:36.920987+0000","last_unstale":"2024-05-09T09:18:30.226902+0000","last_undegraded":"2024-05-09T09:18:30.226902+0000","last_fullsized":"2024-05-09T09:18:30.226902+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:23:55.719198+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"89'9","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529614+0000","last_change":"2024-05-09T09:16:31.870813+0000","last_active":"2024-05-09T09:18:30.529614+0000","last_peered":"2024-05-09T09:18:30.529614+0000","last_clean":"2024-05-09T09:18:30.529614+0000","last_became_active":"2024-05-09T09:16:31.870270+0000","last_became_peered":"2024-05-09T09:16:31.870270+0000","last_unstale":"2024-05-09T09:18:30.529614+0000","last_undegraded":"2024-05-09T09:18:30.529614+0000","last_fullsized":"2024-05-09T09:18:30.529614+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:43:45.862519+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980873+0000","last_change":"2024-05-09T09:16:32.884427+0000","last_active":"2024-05-09T09:18:30.980873+0000","last_peered":"2024-05-09T09:18:30.980873+0000","last_clean":"2024-05-09T09:18:30.980873+0000","last_became_active":"2024-05-09T09:16:32.884127+0000","last_became_peered":"2024-05-09T09:16:32.884127+0000","last_unstale":"2024-05-09T09:18:30.980873+0000","last_undegraded":"2024-05-09T09:18:30.980873+0000","last_fullsized":"2024-05-09T09:18:30.980873+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:01:34.109389+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228966+0000","last_change":"2024-05-09T09:16:36.916715+0000","last_active":"2024-05-09T09:18:30.228966+0000","last_peered":"2024-05-09T09:18:30.228966+0000","last_clean":"2024-05-09T09:18:30.228966+0000","last_became_active":"2024-05-09T09:16:36.916374+0000","last_became_peered":"2024-05-09T09:16:36.916374+0000","last_unstale":"2024-05-09T09:18:30.228966+0000","last_undegraded":"2024-05-09T09:18:30.228966+0000","last_fullsized":"2024-05-09T09:18:30.228966+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:31:16.532151+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981126+0000","last_change":"2024-05-09T09:16:31.866578+0000","last_active":"2024-05-09T09:18:30.981126+0000","last_peered":"2024-05-09T09:18:30.981126+0000","last_clean":"2024-05-09T09:18:30.981126+0000","last_became_active":"2024-05-09T09:16:31.865712+0000","last_became_peered":"2024-05-09T09:16:31.865712+0000","last_unstale":"2024-05-09T09:18:30.981126+0000","last_undegraded":"2024-05-09T09:18:30.981126+0000","last_fullsized":"2024-05-09T09:18:30.981126+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:22:32.516916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980521+0000","last_change":"2024-05-09T09:16:32.888282+0000","last_active":"2024-05-09T09:18:30.980521+0000","last_peered":"2024-05-09T09:18:30.980521+0000","last_clean":"2024-05-09T09:18:30.980521+0000","last_became_active":"2024-05-09T09:16:32.884829+0000","last_became_peered":"2024-05-09T09:16:32.884829+0000","last_unstale":"2024-05-09T09:18:30.980521+0000","last_undegraded":"2024-05-09T09:18:30.980521+0000","last_fullsized":"2024-05-09T09:18:30.980521+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:53:51.226796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229455+0000","last_change":"2024-05-09T09:16:36.912916+0000","last_active":"2024-05-09T09:18:30.229455+0000","last_peered":"2024-05-09T09:18:30.229455+0000","last_clean":"2024-05-09T09:18:30.229455+0000","last_became_active":"2024-05-09T09:16:36.912465+0000","last_became_peered":"2024-05-09T09:16:36.912465+0000","last_unstale":"2024-05-09T09:18:30.229455+0000","last_undegraded":"2024-05-09T09:18:30.229455+0000","last_fullsized":"2024-05-09T09:18:30.229455+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:53:37.777339+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"89'4","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164671+0000","last_change":"2024-05-09T09:17:30.213383+0000","last_active":"2024-05-09T09:18:31.164671+0000","last_peered":"2024-05-09T09:18:31.164671+0000","last_clean":"2024-05-09T09:18:31.164671+0000","last_became_active":"2024-05-09T09:17:30.205007+0000","last_became_peered":"2024-05-09T09:17:30.205007+0000","last_unstale":"2024-05-09T09:18:31.164671+0000","last_undegraded":"2024-05-09T09:18:31.164671+0000","last_fullsized":"2024-05-09T09:18:31.164671+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:55:36.820105+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,7],"acting":[11,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980640+0000","last_change":"2024-05-09T09:16:32.876717+0000","last_active":"2024-05-09T09:18:30.980640+0000","last_peered":"2024-05-09T09:18:30.980640+0000","last_clean":"2024-05-09T09:18:30.980640+0000","last_became_active":"2024-05-09T09:16:32.876395+0000","last_became_peered":"2024-05-09T09:16:32.876395+0000","last_unstale":"2024-05-09T09:18:30.980640+0000","last_undegraded":"2024-05-09T09:18:30.980640+0000","last_fullsized":"2024-05-09T09:18:30.980640+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:15:22.965592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229018+0000","last_change":"2024-05-09T09:16:36.916904+0000","last_active":"2024-05-09T09:18:30.229018+0000","last_peered":"2024-05-09T09:18:30.229018+0000","last_clean":"2024-05-09T09:18:30.229018+0000","last_became_active":"2024-05-09T09:16:36.916440+0000","last_became_peered":"2024-05-09T09:16:36.916440+0000","last_unstale":"2024-05-09T09:18:30.229018+0000","last_undegraded":"2024-05-09T09:18:30.229018+0000","last_fullsized":"2024-05-09T09:18:30.229018+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:11:58.567336+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512765+0000","last_change":"2024-05-09T09:16:31.875916+0000","last_active":"2024-05-09T09:18:30.512765+0000","last_peered":"2024-05-09T09:18:30.512765+0000","last_clean":"2024-05-09T09:18:30.512765+0000","last_became_active":"2024-05-09T09:16:31.875480+0000","last_became_peered":"2024-05-09T09:16:31.875480+0000","last_unstale":"2024-05-09T09:18:30.512765+0000","last_undegraded":"2024-05-09T09:18:30.512765+0000","last_fullsized":"2024-05-09T09:18:30.512765+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:38:18.070101+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227762+0000","last_change":"2024-05-09T09:16:32.892057+0000","last_active":"2024-05-09T09:18:30.227762+0000","last_peered":"2024-05-09T09:18:30.227762+0000","last_clean":"2024-05-09T09:18:30.227762+0000","last_became_active":"2024-05-09T09:16:32.891717+0000","last_became_peered":"2024-05-09T09:16:32.891717+0000","last_unstale":"2024-05-09T09:18:30.227762+0000","last_undegraded":"2024-05-09T09:18:30.227762+0000","last_fullsized":"2024-05-09T09:18:30.227762+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:15:28.553018+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226533+0000","last_change":"2024-05-09T09:16:36.910321+0000","last_active":"2024-05-09T09:18:30.226533+0000","last_peered":"2024-05-09T09:18:30.226533+0000","last_clean":"2024-05-09T09:18:30.226533+0000","last_became_active":"2024-05-09T09:16:36.909817+0000","last_became_peered":"2024-05-09T09:16:36.909817+0000","last_unstale":"2024-05-09T09:18:30.226533+0000","last_undegraded":"2024-05-09T09:18:30.226533+0000","last_fullsized":"2024-05-09T09:18:30.226533+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:17:49.622144+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"93'8","reported_seq":169,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389904+0000","last_change":"2024-05-09T09:17:30.281412+0000","last_active":"2024-05-09T09:18:45.389904+0000","last_peered":"2024-05-09T09:18:45.389904+0000","last_clean":"2024-05-09T09:18:45.389904+0000","last_became_active":"2024-05-09T09:17:30.277253+0000","last_became_peered":"2024-05-09T09:17:30.277253+0000","last_unstale":"2024-05-09T09:18:45.389904+0000","last_undegraded":"2024-05-09T09:18:45.389904+0000","last_fullsized":"2024-05-09T09:18:45.389904+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:18:48.491205+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1462,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":117,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":1013,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,7],"acting":[5,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512581+0000","last_change":"2024-05-09T09:16:32.879099+0000","last_active":"2024-05-09T09:18:30.512581+0000","last_peered":"2024-05-09T09:18:30.512581+0000","last_clean":"2024-05-09T09:18:30.512581+0000","last_became_active":"2024-05-09T09:16:32.878759+0000","last_became_peered":"2024-05-09T09:16:32.878759+0000","last_unstale":"2024-05-09T09:18:30.512581+0000","last_undegraded":"2024-05-09T09:18:30.512581+0000","last_fullsized":"2024-05-09T09:18:30.512581+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:33:05.310416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229789+0000","last_change":"2024-05-09T09:18:30.229789+0000","last_active":"2024-05-09T09:18:30.229789+0000","last_peered":"2024-05-09T09:18:30.229789+0000","last_clean":"2024-05-09T09:18:30.229789+0000","last_became_active":"2024-05-09T09:18:30.229281+0000","last_became_peered":"2024-05-09T09:18:30.229281+0000","last_unstale":"2024-05-09T09:18:30.229789+0000","last_undegraded":"2024-05-09T09:18:30.229789+0000","last_fullsized":"2024-05-09T09:18:30.229789+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:55.816495+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,10,5],"acting":[2,6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226891+0000","last_change":"2024-05-09T09:16:31.874483+0000","last_active":"2024-05-09T09:18:30.226891+0000","last_peered":"2024-05-09T09:18:30.226891+0000","last_clean":"2024-05-09T09:18:30.226891+0000","last_became_active":"2024-05-09T09:16:31.874163+0000","last_became_peered":"2024-05-09T09:16:31.874163+0000","last_unstale":"2024-05-09T09:18:30.226891+0000","last_undegraded":"2024-05-09T09:18:30.226891+0000","last_fullsized":"2024-05-09T09:18:30.226891+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:33:45.259106+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227826+0000","last_change":"2024-05-09T09:16:32.891834+0000","last_active":"2024-05-09T09:18:30.227826+0000","last_peered":"2024-05-09T09:18:30.227826+0000","last_clean":"2024-05-09T09:18:30.227826+0000","last_became_active":"2024-05-09T09:16:32.890511+0000","last_became_peered":"2024-05-09T09:16:32.890511+0000","last_unstale":"2024-05-09T09:18:30.227826+0000","last_undegraded":"2024-05-09T09:18:30.227826+0000","last_fullsized":"2024-05-09T09:18:30.227826+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:09:24.453770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164767+0000","last_change":"2024-05-09T09:17:30.277465+0000","last_active":"2024-05-09T09:18:31.164767+0000","last_peered":"2024-05-09T09:18:31.164767+0000","last_clean":"2024-05-09T09:18:31.164767+0000","last_became_active":"2024-05-09T09:17:30.277014+0000","last_became_peered":"2024-05-09T09:17:30.277014+0000","last_unstale":"2024-05-09T09:18:31.164767+0000","last_undegraded":"2024-05-09T09:18:31.164767+0000","last_fullsized":"2024-05-09T09:18:31.164767+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:36:04.346440+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8,0],"acting":[11,4,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223244+0000","last_change":"2024-05-09T09:16:31.871876+0000","last_active":"2024-05-09T09:18:30.223244+0000","last_peered":"2024-05-09T09:18:30.223244+0000","last_clean":"2024-05-09T09:18:30.223244+0000","last_became_active":"2024-05-09T09:16:31.868685+0000","last_became_peered":"2024-05-09T09:16:31.868685+0000","last_unstale":"2024-05-09T09:18:30.223244+0000","last_undegraded":"2024-05-09T09:18:30.223244+0000","last_fullsized":"2024-05-09T09:18:30.223244+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:32:56.269766+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980233+0000","last_change":"2024-05-09T09:16:32.890687+0000","last_active":"2024-05-09T09:18:30.980233+0000","last_peered":"2024-05-09T09:18:30.980233+0000","last_clean":"2024-05-09T09:18:30.980233+0000","last_became_active":"2024-05-09T09:16:32.890363+0000","last_became_peered":"2024-05-09T09:16:32.890363+0000","last_unstale":"2024-05-09T09:18:30.980233+0000","last_undegraded":"2024-05-09T09:18:30.980233+0000","last_fullsized":"2024-05-09T09:18:30.980233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:30:47.649632+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981359+0000","last_change":"2024-05-09T09:16:36.918617+0000","last_active":"2024-05-09T09:18:30.981359+0000","last_peered":"2024-05-09T09:18:30.981359+0000","last_clean":"2024-05-09T09:18:30.981359+0000","last_became_active":"2024-05-09T09:16:36.918191+0000","last_became_peered":"2024-05-09T09:16:36.918191+0000","last_unstale":"2024-05-09T09:18:30.981359+0000","last_undegraded":"2024-05-09T09:18:30.981359+0000","last_fullsized":"2024-05-09T09:18:30.981359+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:13:46.117884+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227863+0000","last_change":"2024-05-09T09:16:31.869873+0000","last_active":"2024-05-09T09:18:30.227863+0000","last_peered":"2024-05-09T09:18:30.227863+0000","last_clean":"2024-05-09T09:18:30.227863+0000","last_became_active":"2024-05-09T09:16:31.869175+0000","last_became_peered":"2024-05-09T09:16:31.869175+0000","last_unstale":"2024-05-09T09:18:30.227863+0000","last_undegraded":"2024-05-09T09:18:30.227863+0000","last_fullsized":"2024-05-09T09:18:30.227863+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:56:58.065024+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529239+0000","last_change":"2024-05-09T09:16:32.878761+0000","last_active":"2024-05-09T09:18:30.529239+0000","last_peered":"2024-05-09T09:18:30.529239+0000","last_clean":"2024-05-09T09:18:30.529239+0000","last_became_active":"2024-05-09T09:16:32.878402+0000","last_became_peered":"2024-05-09T09:16:32.878402+0000","last_unstale":"2024-05-09T09:18:30.529239+0000","last_undegraded":"2024-05-09T09:18:30.529239+0000","last_fullsized":"2024-05-09T09:18:30.529239+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:39:24.167326+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980320+0000","last_change":"2024-05-09T09:16:36.908423+0000","last_active":"2024-05-09T09:18:30.980320+0000","last_peered":"2024-05-09T09:18:30.980320+0000","last_clean":"2024-05-09T09:18:30.980320+0000","last_became_active":"2024-05-09T09:16:36.908022+0000","last_became_peered":"2024-05-09T09:16:36.908022+0000","last_unstale":"2024-05-09T09:18:30.980320+0000","last_undegraded":"2024-05-09T09:18:30.980320+0000","last_fullsized":"2024-05-09T09:18:30.980320+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:15.104068+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"93'5","reported_seq":149,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:45.389402+0000","last_change":"2024-05-09T09:16:31.869808+0000","last_active":"2024-05-09T09:18:45.389402+0000","last_peered":"2024-05-09T09:18:45.389402+0000","last_clean":"2024-05-09T09:18:45.389402+0000","last_became_active":"2024-05-09T09:16:31.869481+0000","last_became_peered":"2024-05-09T09:16:31.869481+0000","last_unstale":"2024-05-09T09:18:45.389402+0000","last_undegraded":"2024-05-09T09:18:45.389402+0000","last_fullsized":"2024-05-09T09:18:45.389402+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:38:35.200221+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":113,"num_read_kb":113,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980894+0000","last_change":"2024-05-09T09:16:32.881570+0000","last_active":"2024-05-09T09:18:30.980894+0000","last_peered":"2024-05-09T09:18:30.980894+0000","last_clean":"2024-05-09T09:18:30.980894+0000","last_became_active":"2024-05-09T09:16:32.881251+0000","last_became_peered":"2024-05-09T09:16:32.881251+0000","last_unstale":"2024-05-09T09:18:30.980894+0000","last_undegraded":"2024-05-09T09:18:30.980894+0000","last_fullsized":"2024-05-09T09:18:30.980894+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:55:38.603733+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529090+0000","last_change":"2024-05-09T09:16:36.914185+0000","last_active":"2024-05-09T09:18:30.529090+0000","last_peered":"2024-05-09T09:18:30.529090+0000","last_clean":"2024-05-09T09:18:30.529090+0000","last_became_active":"2024-05-09T09:16:36.913831+0000","last_became_peered":"2024-05-09T09:16:36.913831+0000","last_unstale":"2024-05-09T09:18:30.529090+0000","last_undegraded":"2024-05-09T09:18:30.529090+0000","last_fullsized":"2024-05-09T09:18:30.529090+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:18:19.729833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"89'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493132+0000","last_change":"2024-05-09T09:16:31.872498+0000","last_active":"2024-05-09T09:18:30.493132+0000","last_peered":"2024-05-09T09:18:30.493132+0000","last_clean":"2024-05-09T09:18:30.493132+0000","last_became_active":"2024-05-09T09:16:31.871887+0000","last_became_peered":"2024-05-09T09:16:31.871887+0000","last_unstale":"2024-05-09T09:18:30.493132+0000","last_undegraded":"2024-05-09T09:18:30.493132+0000","last_fullsized":"2024-05-09T09:18:30.493132+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:28:17.085494+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980583+0000","last_change":"2024-05-09T09:16:32.895205+0000","last_active":"2024-05-09T09:18:30.980583+0000","last_peered":"2024-05-09T09:18:30.980583+0000","last_clean":"2024-05-09T09:18:30.980583+0000","last_became_active":"2024-05-09T09:16:32.893613+0000","last_became_peered":"2024-05-09T09:16:32.893613+0000","last_unstale":"2024-05-09T09:18:30.980583+0000","last_undegraded":"2024-05-09T09:18:30.980583+0000","last_fullsized":"2024-05-09T09:18:30.980583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:24:01.174012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229610+0000","last_change":"2024-05-09T09:16:36.912847+0000","last_active":"2024-05-09T09:18:30.229610+0000","last_peered":"2024-05-09T09:18:30.229610+0000","last_clean":"2024-05-09T09:18:30.229610+0000","last_became_active":"2024-05-09T09:16:36.912375+0000","last_became_peered":"2024-05-09T09:16:36.912375+0000","last_unstale":"2024-05-09T09:18:30.229610+0000","last_undegraded":"2024-05-09T09:18:30.229610+0000","last_fullsized":"2024-05-09T09:18:30.229610+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:56:58.707697+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"89'4","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227664+0000","last_change":"2024-05-09T09:16:31.871149+0000","last_active":"2024-05-09T09:18:30.227664+0000","last_peered":"2024-05-09T09:18:30.227664+0000","last_clean":"2024-05-09T09:18:30.227664+0000","last_became_active":"2024-05-09T09:16:31.870813+0000","last_became_peered":"2024-05-09T09:16:31.870813+0000","last_unstale":"2024-05-09T09:18:30.227664+0000","last_undegraded":"2024-05-09T09:18:30.227664+0000","last_fullsized":"2024-05-09T09:18:30.227664+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:45:17.408734+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980767+0000","last_change":"2024-05-09T09:16:32.892632+0000","last_active":"2024-05-09T09:18:30.980767+0000","last_peered":"2024-05-09T09:18:30.980767+0000","last_clean":"2024-05-09T09:18:30.980767+0000","last_became_active":"2024-05-09T09:16:32.892233+0000","last_became_peered":"2024-05-09T09:16:32.892233+0000","last_unstale":"2024-05-09T09:18:30.980767+0000","last_undegraded":"2024-05-09T09:18:30.980767+0000","last_fullsized":"2024-05-09T09:18:30.980767+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:32:10.406209+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529402+0000","last_change":"2024-05-09T09:16:36.917590+0000","last_active":"2024-05-09T09:18:30.529402+0000","last_peered":"2024-05-09T09:18:30.529402+0000","last_clean":"2024-05-09T09:18:30.529402+0000","last_became_active":"2024-05-09T09:16:36.916323+0000","last_became_peered":"2024-05-09T09:16:36.916323+0000","last_unstale":"2024-05-09T09:18:30.529402+0000","last_undegraded":"2024-05-09T09:18:30.529402+0000","last_fullsized":"2024-05-09T09:18:30.529402+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:16:04.824468+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"93'37","reported_seq":304,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:46.390433+0000","last_change":"2024-05-09T09:16:31.872230+0000","last_active":"2024-05-09T09:18:46.390433+0000","last_peered":"2024-05-09T09:18:46.390433+0000","last_clean":"2024-05-09T09:18:46.390433+0000","last_became_active":"2024-05-09T09:16:31.871872+0000","last_became_peered":"2024-05-09T09:16:31.871872+0000","last_unstale":"2024-05-09T09:18:46.390433+0000","last_undegraded":"2024-05-09T09:18:46.390433+0000","last_fullsized":"2024-05-09T09:18:46.390433+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:54:46.265386+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63913,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":236,"num_read_kb":181,"num_write":38,"num_write_kb":81,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512855+0000","last_change":"2024-05-09T09:16:32.884279+0000","last_active":"2024-05-09T09:18:30.512855+0000","last_peered":"2024-05-09T09:18:30.512855+0000","last_clean":"2024-05-09T09:18:30.512855+0000","last_became_active":"2024-05-09T09:16:32.883965+0000","last_became_peered":"2024-05-09T09:16:32.883965+0000","last_unstale":"2024-05-09T09:18:30.512855+0000","last_undegraded":"2024-05-09T09:18:30.512855+0000","last_fullsized":"2024-05-09T09:18:30.512855+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:07:27.728361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228102+0000","last_change":"2024-05-09T09:16:36.913155+0000","last_active":"2024-05-09T09:18:30.228102+0000","last_peered":"2024-05-09T09:18:30.228102+0000","last_clean":"2024-05-09T09:18:30.228102+0000","last_became_active":"2024-05-09T09:16:36.912738+0000","last_became_peered":"2024-05-09T09:16:36.912738+0000","last_unstale":"2024-05-09T09:18:30.228102+0000","last_undegraded":"2024-05-09T09:18:30.228102+0000","last_fullsized":"2024-05-09T09:18:30.228102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:23:14.451446+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"89'6","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226401+0000","last_change":"2024-05-09T09:16:31.863982+0000","last_active":"2024-05-09T09:18:30.226401+0000","last_peered":"2024-05-09T09:18:30.226401+0000","last_clean":"2024-05-09T09:18:30.226401+0000","last_became_active":"2024-05-09T09:16:31.862488+0000","last_became_peered":"2024-05-09T09:16:31.862488+0000","last_unstale":"2024-05-09T09:18:30.226401+0000","last_undegraded":"2024-05-09T09:18:30.226401+0000","last_fullsized":"2024-05-09T09:18:30.226401+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:05:57.837296+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223357+0000","last_change":"2024-05-09T09:16:32.885309+0000","last_active":"2024-05-09T09:18:30.223357+0000","last_peered":"2024-05-09T09:18:30.223357+0000","last_clean":"2024-05-09T09:18:30.223357+0000","last_became_active":"2024-05-09T09:16:32.884438+0000","last_became_peered":"2024-05-09T09:16:32.884438+0000","last_unstale":"2024-05-09T09:18:30.223357+0000","last_undegraded":"2024-05-09T09:18:30.223357+0000","last_fullsized":"2024-05-09T09:18:30.223357+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:12:43.967784+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164421+0000","last_change":"2024-05-09T09:16:36.911334+0000","last_active":"2024-05-09T09:18:31.164421+0000","last_peered":"2024-05-09T09:18:31.164421+0000","last_clean":"2024-05-09T09:18:31.164421+0000","last_became_active":"2024-05-09T09:16:36.910787+0000","last_became_peered":"2024-05-09T09:16:36.910787+0000","last_unstale":"2024-05-09T09:18:31.164421+0000","last_undegraded":"2024-05-09T09:18:31.164421+0000","last_fullsized":"2024-05-09T09:18:31.164421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:44:23.369240+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227484+0000","last_change":"2024-05-09T09:16:31.870090+0000","last_active":"2024-05-09T09:18:30.227484+0000","last_peered":"2024-05-09T09:18:30.227484+0000","last_clean":"2024-05-09T09:18:30.227484+0000","last_became_active":"2024-05-09T09:16:31.869688+0000","last_became_peered":"2024-05-09T09:16:31.869688+0000","last_unstale":"2024-05-09T09:18:30.227484+0000","last_undegraded":"2024-05-09T09:18:30.227484+0000","last_fullsized":"2024-05-09T09:18:30.227484+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:12:10.168643+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492851+0000","last_change":"2024-05-09T09:16:32.890677+0000","last_active":"2024-05-09T09:18:30.492851+0000","last_peered":"2024-05-09T09:18:30.492851+0000","last_clean":"2024-05-09T09:18:30.492851+0000","last_became_active":"2024-05-09T09:16:32.888948+0000","last_became_peered":"2024-05-09T09:16:32.888948+0000","last_unstale":"2024-05-09T09:18:30.492851+0000","last_undegraded":"2024-05-09T09:18:30.492851+0000","last_fullsized":"2024-05-09T09:18:30.492851+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:03:36.691690+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513032+0000","last_change":"2024-05-09T09:17:30.465425+0000","last_active":"2024-05-09T09:18:30.513032+0000","last_peered":"2024-05-09T09:18:30.513032+0000","last_clean":"2024-05-09T09:18:30.513032+0000","last_became_active":"2024-05-09T09:17:30.465043+0000","last_became_peered":"2024-05-09T09:17:30.465043+0000","last_unstale":"2024-05-09T09:18:30.513032+0000","last_undegraded":"2024-05-09T09:18:30.513032+0000","last_fullsized":"2024-05-09T09:18:30.513032+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:16:54.706845+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,7,0,3],"acting":[8,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.11","version":"89'3","reported_seq":144,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:40.381710+0000","last_change":"2024-05-09T09:16:31.870178+0000","last_active":"2024-05-09T09:18:40.381710+0000","last_peered":"2024-05-09T09:18:40.381710+0000","last_clean":"2024-05-09T09:18:40.381710+0000","last_became_active":"2024-05-09T09:16:31.869835+0000","last_became_peered":"2024-05-09T09:16:31.869835+0000","last_unstale":"2024-05-09T09:18:40.381710+0000","last_undegraded":"2024-05-09T09:18:40.381710+0000","last_fullsized":"2024-05-09T09:18:40.381710+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:47:48.169662+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":112,"num_read_kb":3,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492685+0000","last_change":"2024-05-09T09:16:32.885315+0000","last_active":"2024-05-09T09:18:30.492685+0000","last_peered":"2024-05-09T09:18:30.492685+0000","last_clean":"2024-05-09T09:18:30.492685+0000","last_became_active":"2024-05-09T09:16:32.884939+0000","last_became_peered":"2024-05-09T09:16:32.884939+0000","last_unstale":"2024-05-09T09:18:30.492685+0000","last_undegraded":"2024-05-09T09:18:30.492685+0000","last_fullsized":"2024-05-09T09:18:30.492685+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:10:44.086031+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223712+0000","last_change":"2024-05-09T09:16:36.917591+0000","last_active":"2024-05-09T09:18:30.223712+0000","last_peered":"2024-05-09T09:18:30.223712+0000","last_clean":"2024-05-09T09:18:30.223712+0000","last_became_active":"2024-05-09T09:16:36.916024+0000","last_became_peered":"2024-05-09T09:16:36.916024+0000","last_unstale":"2024-05-09T09:18:30.223712+0000","last_undegraded":"2024-05-09T09:18:30.223712+0000","last_fullsized":"2024-05-09T09:18:30.223712+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:15:46.376373+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"89'15","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512424+0000","last_change":"2024-05-09T09:16:31.871305+0000","last_active":"2024-05-09T09:18:30.512424+0000","last_peered":"2024-05-09T09:18:30.512424+0000","last_clean":"2024-05-09T09:18:30.512424+0000","last_became_active":"2024-05-09T09:16:31.870909+0000","last_became_peered":"2024-05-09T09:16:31.870909+0000","last_unstale":"2024-05-09T09:18:30.512424+0000","last_undegraded":"2024-05-09T09:18:30.512424+0000","last_fullsized":"2024-05-09T09:18:30.512424+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:58:03.763013+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227187+0000","last_change":"2024-05-09T09:16:32.891121+0000","last_active":"2024-05-09T09:18:30.227187+0000","last_peered":"2024-05-09T09:18:30.227187+0000","last_clean":"2024-05-09T09:18:30.227187+0000","last_became_active":"2024-05-09T09:16:32.890754+0000","last_became_peered":"2024-05-09T09:16:32.890754+0000","last_unstale":"2024-05-09T09:18:30.227187+0000","last_undegraded":"2024-05-09T09:18:30.227187+0000","last_fullsized":"2024-05-09T09:18:30.227187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:41:11.029801+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227206+0000","last_change":"2024-05-09T09:16:36.912115+0000","last_active":"2024-05-09T09:18:30.227206+0000","last_peered":"2024-05-09T09:18:30.227206+0000","last_clean":"2024-05-09T09:18:30.227206+0000","last_became_active":"2024-05-09T09:16:36.909714+0000","last_became_peered":"2024-05-09T09:16:36.909714+0000","last_unstale":"2024-05-09T09:18:30.227206+0000","last_undegraded":"2024-05-09T09:18:30.227206+0000","last_fullsized":"2024-05-09T09:18:30.227206+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:08:15.145628+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"89'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.513190+0000","last_change":"2024-05-09T09:16:31.875485+0000","last_active":"2024-05-09T09:18:30.513190+0000","last_peered":"2024-05-09T09:18:30.513190+0000","last_clean":"2024-05-09T09:18:30.513190+0000","last_became_active":"2024-05-09T09:16:31.875123+0000","last_became_peered":"2024-05-09T09:16:31.875123+0000","last_unstale":"2024-05-09T09:18:30.513190+0000","last_undegraded":"2024-05-09T09:18:30.513190+0000","last_fullsized":"2024-05-09T09:18:30.513190+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:37:25.609557+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164659+0000","last_change":"2024-05-09T09:16:32.874549+0000","last_active":"2024-05-09T09:18:31.164659+0000","last_peered":"2024-05-09T09:18:31.164659+0000","last_clean":"2024-05-09T09:18:31.164659+0000","last_became_active":"2024-05-09T09:16:32.874175+0000","last_became_peered":"2024-05-09T09:16:32.874175+0000","last_unstale":"2024-05-09T09:18:31.164659+0000","last_undegraded":"2024-05-09T09:18:31.164659+0000","last_fullsized":"2024-05-09T09:18:31.164659+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:20.875004+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512820+0000","last_change":"2024-05-09T09:16:36.967278+0000","last_active":"2024-05-09T09:18:30.512820+0000","last_peered":"2024-05-09T09:18:30.512820+0000","last_clean":"2024-05-09T09:18:30.512820+0000","last_became_active":"2024-05-09T09:16:36.966700+0000","last_became_peered":"2024-05-09T09:16:36.966700+0000","last_unstale":"2024-05-09T09:18:30.512820+0000","last_undegraded":"2024-05-09T09:18:30.512820+0000","last_fullsized":"2024-05-09T09:18:30.512820+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:23:03.269303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529852+0000","last_change":"2024-05-09T09:16:31.868348+0000","last_active":"2024-05-09T09:18:30.529852+0000","last_peered":"2024-05-09T09:18:30.529852+0000","last_clean":"2024-05-09T09:18:30.529852+0000","last_became_active":"2024-05-09T09:16:31.868031+0000","last_became_peered":"2024-05-09T09:16:31.868031+0000","last_unstale":"2024-05-09T09:18:30.529852+0000","last_undegraded":"2024-05-09T09:18:30.529852+0000","last_fullsized":"2024-05-09T09:18:30.529852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:10:41.939998+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227403+0000","last_change":"2024-05-09T09:16:32.882152+0000","last_active":"2024-05-09T09:18:30.227403+0000","last_peered":"2024-05-09T09:18:30.227403+0000","last_clean":"2024-05-09T09:18:30.227403+0000","last_became_active":"2024-05-09T09:16:32.881783+0000","last_became_peered":"2024-05-09T09:16:32.881783+0000","last_unstale":"2024-05-09T09:18:30.227403+0000","last_undegraded":"2024-05-09T09:18:30.227403+0000","last_fullsized":"2024-05-09T09:18:30.227403+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:02:05.894158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223570+0000","last_change":"2024-05-09T09:16:36.917360+0000","last_active":"2024-05-09T09:18:30.223570+0000","last_peered":"2024-05-09T09:18:30.223570+0000","last_clean":"2024-05-09T09:18:30.223570+0000","last_became_active":"2024-05-09T09:16:36.916974+0000","last_became_peered":"2024-05-09T09:16:36.916974+0000","last_unstale":"2024-05-09T09:18:30.223570+0000","last_undegraded":"2024-05-09T09:18:30.223570+0000","last_fullsized":"2024-05-09T09:18:30.223570+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:57:44.651896+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:31.876467+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:31.876122+0000","last_became_peered":"2024-05-09T09:16:31.876122+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:06:07.171214+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226975+0000","last_change":"2024-05-09T09:17:30.203699+0000","last_active":"2024-05-09T09:18:30.226975+0000","last_peered":"2024-05-09T09:18:30.226975+0000","last_clean":"2024-05-09T09:18:30.226975+0000","last_became_active":"2024-05-09T09:17:30.203312+0000","last_became_peered":"2024-05-09T09:17:30.203312+0000","last_unstale":"2024-05-09T09:18:30.226975+0000","last_undegraded":"2024-05-09T09:18:30.226975+0000","last_fullsized":"2024-05-09T09:18:30.226975+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:49:23.784938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,0],"acting":[10,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:36.967202+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:36.966742+0000","last_became_peered":"2024-05-09T09:16:36.966742+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:14:01.018943+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529073+0000","last_change":"2024-05-09T09:17:30.276150+0000","last_active":"2024-05-09T09:18:30.529073+0000","last_peered":"2024-05-09T09:18:30.529073+0000","last_clean":"2024-05-09T09:18:30.529073+0000","last_became_active":"2024-05-09T09:17:30.275868+0000","last_became_peered":"2024-05-09T09:17:30.275868+0000","last_unstale":"2024-05-09T09:18:30.529073+0000","last_undegraded":"2024-05-09T09:18:30.529073+0000","last_fullsized":"2024-05-09T09:18:30.529073+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:18:08.892380+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,4],"acting":[9,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.1","version":"89'4","reported_seq":145,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:41.382493+0000","last_change":"2024-05-09T09:16:31.873150+0000","last_active":"2024-05-09T09:18:41.382493+0000","last_peered":"2024-05-09T09:18:41.382493+0000","last_clean":"2024-05-09T09:18:41.382493+0000","last_became_active":"2024-05-09T09:16:31.872832+0000","last_became_peered":"2024-05-09T09:16:31.872832+0000","last_unstale":"2024-05-09T09:18:41.382493+0000","last_undegraded":"2024-05-09T09:18:41.382493+0000","last_fullsized":"2024-05-09T09:18:41.382493+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:11:04.543810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":110,"num_read_kb":110,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493069+0000","last_change":"2024-05-09T09:16:36.905790+0000","last_active":"2024-05-09T09:18:30.493069+0000","last_peered":"2024-05-09T09:18:30.493069+0000","last_clean":"2024-05-09T09:18:30.493069+0000","last_became_active":"2024-05-09T09:16:36.905164+0000","last_became_peered":"2024-05-09T09:16:36.905164+0000","last_unstale":"2024-05-09T09:18:30.493069+0000","last_undegraded":"2024-05-09T09:18:30.493069+0000","last_fullsized":"2024-05-09T09:18:30.493069+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:49:03.094629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980587+0000","last_change":"2024-05-09T09:16:32.886232+0000","last_active":"2024-05-09T09:18:30.980587+0000","last_peered":"2024-05-09T09:18:30.980587+0000","last_clean":"2024-05-09T09:18:30.980587+0000","last_became_active":"2024-05-09T09:16:32.885712+0000","last_became_peered":"2024-05-09T09:16:32.885712+0000","last_unstale":"2024-05-09T09:18:30.980587+0000","last_undegraded":"2024-05-09T09:18:30.980587+0000","last_fullsized":"2024-05-09T09:18:30.980587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:12:37.951351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"89'3","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229495+0000","last_change":"2024-05-09T09:16:31.862691+0000","last_active":"2024-05-09T09:18:30.229495+0000","last_peered":"2024-05-09T09:18:30.229495+0000","last_clean":"2024-05-09T09:18:30.229495+0000","last_became_active":"2024-05-09T09:16:31.862314+0000","last_became_peered":"2024-05-09T09:16:31.862314+0000","last_unstale":"2024-05-09T09:18:30.229495+0000","last_undegraded":"2024-05-09T09:18:30.229495+0000","last_fullsized":"2024-05-09T09:18:30.229495+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:05:54.041611+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":4,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":356,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492528+0000","last_change":"2024-05-09T09:15:19.634765+0000","last_active":"2024-05-09T09:18:30.492528+0000","last_peered":"2024-05-09T09:18:30.492528+0000","last_clean":"2024-05-09T09:18:30.492528+0000","last_became_active":"2024-05-09T09:15:19.634302+0000","last_became_peered":"2024-05-09T09:15:19.634302+0000","last_unstale":"2024-05-09T09:18:30.492528+0000","last_undegraded":"2024-05-09T09:18:30.492528+0000","last_fullsized":"2024-05-09T09:18:30.492528+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:11:51.782220+0000","last_clean_scrub_stamp":"2024-05-09T09:11:51.782220+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:16:16.332309+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227346+0000","last_change":"2024-05-09T09:16:36.967330+0000","last_active":"2024-05-09T09:18:30.227346+0000","last_peered":"2024-05-09T09:18:30.227346+0000","last_clean":"2024-05-09T09:18:30.227346+0000","last_became_active":"2024-05-09T09:16:36.966762+0000","last_became_peered":"2024-05-09T09:16:36.966762+0000","last_unstale":"2024-05-09T09:18:30.227346+0000","last_undegraded":"2024-05-09T09:18:30.227346+0000","last_fullsized":"2024-05-09T09:18:30.227346+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:45:44.912691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223208+0000","last_change":"2024-05-09T09:16:32.888648+0000","last_active":"2024-05-09T09:18:30.223208+0000","last_peered":"2024-05-09T09:18:30.223208+0000","last_clean":"2024-05-09T09:18:30.223208+0000","last_became_active":"2024-05-09T09:16:32.888288+0000","last_became_peered":"2024-05-09T09:16:32.888288+0000","last_unstale":"2024-05-09T09:18:30.223208+0000","last_undegraded":"2024-05-09T09:18:30.223208+0000","last_fullsized":"2024-05-09T09:18:30.223208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:56:32.201060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"89'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492715+0000","last_change":"2024-05-09T09:16:31.875218+0000","last_active":"2024-05-09T09:18:30.492715+0000","last_peered":"2024-05-09T09:18:30.492715+0000","last_clean":"2024-05-09T09:18:30.492715+0000","last_became_active":"2024-05-09T09:16:31.874900+0000","last_became_peered":"2024-05-09T09:16:31.874900+0000","last_unstale":"2024-05-09T09:18:30.492715+0000","last_undegraded":"2024-05-09T09:18:30.492715+0000","last_fullsized":"2024-05-09T09:18:30.492715+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T10:28:32.789279+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164409+0000","last_change":"2024-05-09T09:16:36.911409+0000","last_active":"2024-05-09T09:18:31.164409+0000","last_peered":"2024-05-09T09:18:31.164409+0000","last_clean":"2024-05-09T09:18:31.164409+0000","last_became_active":"2024-05-09T09:16:36.910858+0000","last_became_peered":"2024-05-09T09:16:36.910858+0000","last_unstale":"2024-05-09T09:18:31.164409+0000","last_undegraded":"2024-05-09T09:18:31.164409+0000","last_fullsized":"2024-05-09T09:18:31.164409+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:33:04.281914+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"89'1","reported_seq":19,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164541+0000","last_change":"2024-05-09T09:17:30.208053+0000","last_active":"2024-05-09T09:18:31.164541+0000","last_peered":"2024-05-09T09:18:31.164541+0000","last_clean":"2024-05-09T09:18:31.164541+0000","last_became_active":"2024-05-09T09:17:30.203628+0000","last_became_peered":"2024-05-09T09:17:30.203628+0000","last_unstale":"2024-05-09T09:18:31.164541+0000","last_undegraded":"2024-05-09T09:18:31.164541+0000","last_fullsized":"2024-05-09T09:18:31.164541+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:37:37.871338+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":22,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223788+0000","last_change":"2024-05-09T09:16:32.874055+0000","last_active":"2024-05-09T09:18:30.223788+0000","last_peered":"2024-05-09T09:18:30.223788+0000","last_clean":"2024-05-09T09:18:30.223788+0000","last_became_active":"2024-05-09T09:16:32.873740+0000","last_became_peered":"2024-05-09T09:16:32.873740+0000","last_unstale":"2024-05-09T09:18:30.223788+0000","last_undegraded":"2024-05-09T09:18:30.223788+0000","last_fullsized":"2024-05-09T09:18:30.223788+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:35:59.934671+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980275+0000","last_change":"2024-05-09T09:16:36.967224+0000","last_active":"2024-05-09T09:18:30.980275+0000","last_peered":"2024-05-09T09:18:30.980275+0000","last_clean":"2024-05-09T09:18:30.980275+0000","last_became_active":"2024-05-09T09:16:36.966815+0000","last_became_peered":"2024-05-09T09:16:36.966815+0000","last_unstale":"2024-05-09T09:18:30.980275+0000","last_undegraded":"2024-05-09T09:18:30.980275+0000","last_fullsized":"2024-05-09T09:18:30.980275+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:42:55.770054+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'10","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:43.881851+0000","last_change":"2024-05-09T09:16:31.868376+0000","last_active":"2024-05-09T09:18:43.881851+0000","last_peered":"2024-05-09T09:18:43.881851+0000","last_clean":"2024-05-09T09:18:43.881851+0000","last_became_active":"2024-05-09T09:16:31.868055+0000","last_became_peered":"2024-05-09T09:16:31.868055+0000","last_unstale":"2024-05-09T09:18:43.881851+0000","last_undegraded":"2024-05-09T09:18:43.881851+0000","last_fullsized":"2024-05-09T09:18:43.881851+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:30:36.279379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":18,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981095+0000","last_change":"2024-05-09T09:16:32.890067+0000","last_active":"2024-05-09T09:18:30.981095+0000","last_peered":"2024-05-09T09:18:30.981095+0000","last_clean":"2024-05-09T09:18:30.981095+0000","last_became_active":"2024-05-09T09:16:32.889724+0000","last_became_peered":"2024-05-09T09:16:32.889724+0000","last_unstale":"2024-05-09T09:18:30.981095+0000","last_undegraded":"2024-05-09T09:18:30.981095+0000","last_fullsized":"2024-05-09T09:18:30.981095+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:14:07.911696+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226060+0000","last_change":"2024-05-09T09:16:36.907472+0000","last_active":"2024-05-09T09:18:30.226060+0000","last_peered":"2024-05-09T09:18:30.226060+0000","last_clean":"2024-05-09T09:18:30.226060+0000","last_became_active":"2024-05-09T09:16:36.907054+0000","last_became_peered":"2024-05-09T09:16:36.907054+0000","last_unstale":"2024-05-09T09:18:30.226060+0000","last_undegraded":"2024-05-09T09:18:30.226060+0000","last_fullsized":"2024-05-09T09:18:30.226060+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:11:38.387277+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229540+0000","last_change":"2024-05-09T09:16:31.878276+0000","last_active":"2024-05-09T09:18:30.229540+0000","last_peered":"2024-05-09T09:18:30.229540+0000","last_clean":"2024-05-09T09:18:30.229540+0000","last_became_active":"2024-05-09T09:16:31.878017+0000","last_became_peered":"2024-05-09T09:16:31.878017+0000","last_unstale":"2024-05-09T09:18:30.229540+0000","last_undegraded":"2024-05-09T09:18:30.229540+0000","last_fullsized":"2024-05-09T09:18:30.229540+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:36:00.670951+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981427+0000","last_change":"2024-05-09T09:16:32.893217+0000","last_active":"2024-05-09T09:18:30.981427+0000","last_peered":"2024-05-09T09:18:30.981427+0000","last_clean":"2024-05-09T09:18:30.981427+0000","last_became_active":"2024-05-09T09:16:32.892887+0000","last_became_peered":"2024-05-09T09:16:32.892887+0000","last_unstale":"2024-05-09T09:18:30.981427+0000","last_undegraded":"2024-05-09T09:18:30.981427+0000","last_fullsized":"2024-05-09T09:18:30.981427+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:56:20.713685+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.530637+0000","last_change":"2024-05-09T09:18:30.530637+0000","last_active":"2024-05-09T09:18:30.530637+0000","last_peered":"2024-05-09T09:18:30.530637+0000","last_clean":"2024-05-09T09:18:30.530637+0000","last_became_active":"2024-05-09T09:18:30.530213+0000","last_became_peered":"2024-05-09T09:18:30.530213+0000","last_unstale":"2024-05-09T09:18:30.530637+0000","last_undegraded":"2024-05-09T09:18:30.530637+0000","last_fullsized":"2024-05-09T09:18:30.530637+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:58:48.951306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,10,9],"acting":[4,8,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"89'2","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980453+0000","last_change":"2024-05-09T09:16:31.871362+0000","last_active":"2024-05-09T09:18:30.980453+0000","last_peered":"2024-05-09T09:18:30.980453+0000","last_clean":"2024-05-09T09:18:30.980453+0000","last_became_active":"2024-05-09T09:16:31.871049+0000","last_became_peered":"2024-05-09T09:16:31.871049+0000","last_unstale":"2024-05-09T09:18:30.980453+0000","last_undegraded":"2024-05-09T09:18:30.980453+0000","last_fullsized":"2024-05-09T09:18:30.980453+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:00:55.442882+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512512+0000","last_change":"2024-05-09T09:16:32.881760+0000","last_active":"2024-05-09T09:18:30.512512+0000","last_peered":"2024-05-09T09:18:30.512512+0000","last_clean":"2024-05-09T09:18:30.512512+0000","last_became_active":"2024-05-09T09:16:32.881418+0000","last_became_peered":"2024-05-09T09:16:32.881418+0000","last_unstale":"2024-05-09T09:18:30.512512+0000","last_undegraded":"2024-05-09T09:18:30.512512+0000","last_fullsized":"2024-05-09T09:18:30.512512+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:08:19.217847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.529569+0000","last_change":"2024-05-09T09:16:36.918895+0000","last_active":"2024-05-09T09:18:30.529569+0000","last_peered":"2024-05-09T09:18:30.529569+0000","last_clean":"2024-05-09T09:18:30.529569+0000","last_became_active":"2024-05-09T09:16:36.918355+0000","last_became_peered":"2024-05-09T09:16:36.918355+0000","last_unstale":"2024-05-09T09:18:30.529569+0000","last_undegraded":"2024-05-09T09:18:30.529569+0000","last_fullsized":"2024-05-09T09:18:30.529569+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:22:38.677636+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980366+0000","last_change":"2024-05-09T09:16:31.866291+0000","last_active":"2024-05-09T09:18:30.980366+0000","last_peered":"2024-05-09T09:18:30.980366+0000","last_clean":"2024-05-09T09:18:30.980366+0000","last_became_active":"2024-05-09T09:16:31.865932+0000","last_became_peered":"2024-05-09T09:16:31.865932+0000","last_unstale":"2024-05-09T09:18:30.980366+0000","last_undegraded":"2024-05-09T09:18:30.980366+0000","last_fullsized":"2024-05-09T09:18:30.980366+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:39:49.830288+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164377+0000","last_change":"2024-05-09T09:16:32.883256+0000","last_active":"2024-05-09T09:18:31.164377+0000","last_peered":"2024-05-09T09:18:31.164377+0000","last_clean":"2024-05-09T09:18:31.164377+0000","last_became_active":"2024-05-09T09:16:32.882497+0000","last_became_peered":"2024-05-09T09:16:32.882497+0000","last_unstale":"2024-05-09T09:18:31.164377+0000","last_undegraded":"2024-05-09T09:18:31.164377+0000","last_fullsized":"2024-05-09T09:18:31.164377+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:43:44.094620+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229286+0000","last_change":"2024-05-09T09:16:36.912998+0000","last_active":"2024-05-09T09:18:30.229286+0000","last_peered":"2024-05-09T09:18:30.229286+0000","last_clean":"2024-05-09T09:18:30.229286+0000","last_became_active":"2024-05-09T09:16:36.912521+0000","last_became_peered":"2024-05-09T09:16:36.912521+0000","last_unstale":"2024-05-09T09:18:30.229286+0000","last_undegraded":"2024-05-09T09:18:30.229286+0000","last_fullsized":"2024-05-09T09:18:30.229286+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:03:46.239276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"89'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226564+0000","last_change":"2024-05-09T09:16:31.865935+0000","last_active":"2024-05-09T09:18:30.226564+0000","last_peered":"2024-05-09T09:18:30.226564+0000","last_clean":"2024-05-09T09:18:30.226564+0000","last_became_active":"2024-05-09T09:16:31.865339+0000","last_became_peered":"2024-05-09T09:16:31.865339+0000","last_unstale":"2024-05-09T09:18:30.226564+0000","last_undegraded":"2024-05-09T09:18:30.226564+0000","last_fullsized":"2024-05-09T09:18:30.226564+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T15:24:16.386266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980483+0000","last_change":"2024-05-09T09:16:32.880764+0000","last_active":"2024-05-09T09:18:30.980483+0000","last_peered":"2024-05-09T09:18:30.980483+0000","last_clean":"2024-05-09T09:18:30.980483+0000","last_became_active":"2024-05-09T09:16:32.880395+0000","last_became_peered":"2024-05-09T09:16:32.880395+0000","last_unstale":"2024-05-09T09:18:30.980483+0000","last_undegraded":"2024-05-09T09:18:30.980483+0000","last_fullsized":"2024-05-09T09:18:30.980483+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:18:16.668596+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.229268+0000","last_change":"2024-05-09T09:16:36.966314+0000","last_active":"2024-05-09T09:18:30.229268+0000","last_peered":"2024-05-09T09:18:30.229268+0000","last_clean":"2024-05-09T09:18:30.229268+0000","last_became_active":"2024-05-09T09:16:36.965872+0000","last_became_peered":"2024-05-09T09:16:36.965872+0000","last_unstale":"2024-05-09T09:18:30.229268+0000","last_undegraded":"2024-05-09T09:18:30.229268+0000","last_fullsized":"2024-05-09T09:18:30.229268+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:17:55.674996+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"89'4","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.512609+0000","last_change":"2024-05-09T09:16:31.874556+0000","last_active":"2024-05-09T09:18:30.512609+0000","last_peered":"2024-05-09T09:18:30.512609+0000","last_clean":"2024-05-09T09:18:30.512609+0000","last_became_active":"2024-05-09T09:16:31.874206+0000","last_became_peered":"2024-05-09T09:16:31.874206+0000","last_unstale":"2024-05-09T09:18:30.512609+0000","last_undegraded":"2024-05-09T09:18:30.512609+0000","last_fullsized":"2024-05-09T09:18:30.512609+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:54:31.176075+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492902+0000","last_change":"2024-05-09T09:16:32.890148+0000","last_active":"2024-05-09T09:18:30.492902+0000","last_peered":"2024-05-09T09:18:30.492902+0000","last_clean":"2024-05-09T09:18:30.492902+0000","last_became_active":"2024-05-09T09:16:32.888418+0000","last_became_peered":"2024-05-09T09:16:32.888418+0000","last_unstale":"2024-05-09T09:18:30.492902+0000","last_undegraded":"2024-05-09T09:18:30.492902+0000","last_fullsized":"2024-05-09T09:18:30.492902+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:06:54.790480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980379+0000","last_change":"2024-05-09T09:16:36.913654+0000","last_active":"2024-05-09T09:18:30.980379+0000","last_peered":"2024-05-09T09:18:30.980379+0000","last_clean":"2024-05-09T09:18:30.980379+0000","last_became_active":"2024-05-09T09:16:36.913180+0000","last_became_peered":"2024-05-09T09:16:36.913180+0000","last_unstale":"2024-05-09T09:18:30.980379+0000","last_undegraded":"2024-05-09T09:18:30.980379+0000","last_fullsized":"2024-05-09T09:18:30.980379+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T09:34:31.897197+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"89'3","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227058+0000","last_change":"2024-05-09T09:16:31.877533+0000","last_active":"2024-05-09T09:18:30.227058+0000","last_peered":"2024-05-09T09:18:30.227058+0000","last_clean":"2024-05-09T09:18:30.227058+0000","last_became_active":"2024-05-09T09:16:31.877216+0000","last_became_peered":"2024-05-09T09:16:31.877216+0000","last_unstale":"2024-05-09T09:18:30.227058+0000","last_undegraded":"2024-05-09T09:18:30.227058+0000","last_fullsized":"2024-05-09T09:18:30.227058+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T20:52:38.534248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493221+0000","last_change":"2024-05-09T09:16:32.882332+0000","last_active":"2024-05-09T09:18:30.493221+0000","last_peered":"2024-05-09T09:18:30.493221+0000","last_clean":"2024-05-09T09:18:30.493221+0000","last_became_active":"2024-05-09T09:16:32.881980+0000","last_became_peered":"2024-05-09T09:16:32.881980+0000","last_unstale":"2024-05-09T09:18:30.493221+0000","last_undegraded":"2024-05-09T09:18:30.493221+0000","last_fullsized":"2024-05-09T09:18:30.493221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:40:37.494225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980643+0000","last_change":"2024-05-09T09:16:36.922576+0000","last_active":"2024-05-09T09:18:30.980643+0000","last_peered":"2024-05-09T09:18:30.980643+0000","last_clean":"2024-05-09T09:18:30.980643+0000","last_became_active":"2024-05-09T09:16:36.922112+0000","last_became_peered":"2024-05-09T09:16:36.922112+0000","last_unstale":"2024-05-09T09:18:30.980643+0000","last_undegraded":"2024-05-09T09:18:30.980643+0000","last_fullsized":"2024-05-09T09:18:30.980643+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:48:39.141157+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227912+0000","last_change":"2024-05-09T09:16:31.868381+0000","last_active":"2024-05-09T09:18:30.227912+0000","last_peered":"2024-05-09T09:18:30.227912+0000","last_clean":"2024-05-09T09:18:30.227912+0000","last_became_active":"2024-05-09T09:16:31.866293+0000","last_became_peered":"2024-05-09T09:16:31.866293+0000","last_unstale":"2024-05-09T09:18:30.227912+0000","last_undegraded":"2024-05-09T09:18:30.227912+0000","last_fullsized":"2024-05-09T09:18:30.227912+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T11:24:12.723227+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.492929+0000","last_change":"2024-05-09T09:16:32.883750+0000","last_active":"2024-05-09T09:18:30.492929+0000","last_peered":"2024-05-09T09:18:30.492929+0000","last_clean":"2024-05-09T09:18:30.492929+0000","last_became_active":"2024-05-09T09:16:32.883223+0000","last_became_peered":"2024-05-09T09:16:32.883223+0000","last_unstale":"2024-05-09T09:18:30.492929+0000","last_undegraded":"2024-05-09T09:18:30.492929+0000","last_fullsized":"2024-05-09T09:18:30.492929+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:46:44.028600+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227073+0000","last_change":"2024-05-09T09:16:36.912086+0000","last_active":"2024-05-09T09:18:30.227073+0000","last_peered":"2024-05-09T09:18:30.227073+0000","last_clean":"2024-05-09T09:18:30.227073+0000","last_became_active":"2024-05-09T09:16:36.909589+0000","last_became_peered":"2024-05-09T09:16:36.909589+0000","last_unstale":"2024-05-09T09:18:30.227073+0000","last_undegraded":"2024-05-09T09:18:30.227073+0000","last_fullsized":"2024-05-09T09:18:30.227073+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:35.885310+0000","last_clean_scrub_stamp":"2024-05-09T09:16:35.885310+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:51:27.191293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223662+0000","last_change":"2024-05-09T09:16:31.869409+0000","last_active":"2024-05-09T09:18:30.223662+0000","last_peered":"2024-05-09T09:18:30.223662+0000","last_clean":"2024-05-09T09:18:30.223662+0000","last_became_active":"2024-05-09T09:16:31.869109+0000","last_became_peered":"2024-05-09T09:16:31.869109+0000","last_unstale":"2024-05-09T09:18:30.223662+0000","last_undegraded":"2024-05-09T09:18:30.223662+0000","last_fullsized":"2024-05-09T09:18:30.223662+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:30.841245+0000","last_clean_scrub_stamp":"2024-05-09T09:16:30.841245+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T21:08:54.749372+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.493231+0000","last_change":"2024-05-09T09:16:32.879703+0000","last_active":"2024-05-09T09:18:30.493231+0000","last_peered":"2024-05-09T09:18:30.493231+0000","last_clean":"2024-05-09T09:18:30.493231+0000","last_became_active":"2024-05-09T09:16:32.879325+0000","last_became_peered":"2024-05-09T09:16:32.879325+0000","last_unstale":"2024-05-09T09:18:30.493231+0000","last_undegraded":"2024-05-09T09:18:30.493231+0000","last_fullsized":"2024-05-09T09:18:30.493231+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:47:04.535218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164450+0000","last_change":"2024-05-09T09:16:32.875405+0000","last_active":"2024-05-09T09:18:31.164450+0000","last_peered":"2024-05-09T09:18:31.164450+0000","last_clean":"2024-05-09T09:18:31.164450+0000","last_became_active":"2024-05-09T09:16:32.874949+0000","last_became_peered":"2024-05-09T09:16:32.874949+0000","last_unstale":"2024-05-09T09:18:31.164450+0000","last_undegraded":"2024-05-09T09:18:31.164450+0000","last_fullsized":"2024-05-09T09:18:31.164450+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T19:22:35.723508+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227152+0000","last_change":"2024-05-09T09:16:32.893309+0000","last_active":"2024-05-09T09:18:30.227152+0000","last_peered":"2024-05-09T09:18:30.227152+0000","last_clean":"2024-05-09T09:18:30.227152+0000","last_became_active":"2024-05-09T09:16:32.892985+0000","last_became_peered":"2024-05-09T09:16:32.892985+0000","last_unstale":"2024-05-09T09:18:30.227152+0000","last_undegraded":"2024-05-09T09:18:30.227152+0000","last_fullsized":"2024-05-09T09:18:30.227152+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:46:00.132670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.226406+0000","last_change":"2024-05-09T09:16:32.885972+0000","last_active":"2024-05-09T09:18:30.226406+0000","last_peered":"2024-05-09T09:18:30.226406+0000","last_clean":"2024-05-09T09:18:30.226406+0000","last_became_active":"2024-05-09T09:16:32.885591+0000","last_became_peered":"2024-05-09T09:16:32.885591+0000","last_unstale":"2024-05-09T09:18:30.226406+0000","last_undegraded":"2024-05-09T09:18:30.226406+0000","last_fullsized":"2024-05-09T09:18:30.226406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:04:53.320321+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980983+0000","last_change":"2024-05-09T09:16:32.892965+0000","last_active":"2024-05-09T09:18:30.980983+0000","last_peered":"2024-05-09T09:18:30.980983+0000","last_clean":"2024-05-09T09:18:30.980983+0000","last_became_active":"2024-05-09T09:16:32.892615+0000","last_became_peered":"2024-05-09T09:16:32.892615+0000","last_unstale":"2024-05-09T09:18:30.980983+0000","last_undegraded":"2024-05-09T09:18:30.980983+0000","last_fullsized":"2024-05-09T09:18:30.980983+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:05:01.259124+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223725+0000","last_change":"2024-05-09T09:16:32.893972+0000","last_active":"2024-05-09T09:18:30.223725+0000","last_peered":"2024-05-09T09:18:30.223725+0000","last_clean":"2024-05-09T09:18:30.223725+0000","last_became_active":"2024-05-09T09:16:32.893639+0000","last_became_peered":"2024-05-09T09:16:32.893639+0000","last_unstale":"2024-05-09T09:18:30.223725+0000","last_undegraded":"2024-05-09T09:18:30.223725+0000","last_fullsized":"2024-05-09T09:18:30.223725+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T13:01:15.167478+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164624+0000","last_change":"2024-05-09T09:16:32.887677+0000","last_active":"2024-05-09T09:18:31.164624+0000","last_peered":"2024-05-09T09:18:31.164624+0000","last_clean":"2024-05-09T09:18:31.164624+0000","last_became_active":"2024-05-09T09:16:32.887165+0000","last_became_peered":"2024-05-09T09:16:32.887165+0000","last_unstale":"2024-05-09T09:18:31.164624+0000","last_undegraded":"2024-05-09T09:18:31.164624+0000","last_fullsized":"2024-05-09T09:18:31.164624+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:49:38.247135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":44,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.227381+0000","last_change":"2024-05-09T09:16:32.891070+0000","last_active":"2024-05-09T09:18:30.227381+0000","last_peered":"2024-05-09T09:18:30.227381+0000","last_clean":"2024-05-09T09:18:30.227381+0000","last_became_active":"2024-05-09T09:16:32.890717+0000","last_became_peered":"2024-05-09T09:16:32.890717+0000","last_unstale":"2024-05-09T09:18:30.227381+0000","last_undegraded":"2024-05-09T09:18:30.227381+0000","last_fullsized":"2024-05-09T09:18:30.227381+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:08:14.102475+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.981199+0000","last_change":"2024-05-09T09:16:32.893007+0000","last_active":"2024-05-09T09:18:30.981199+0000","last_peered":"2024-05-09T09:18:30.981199+0000","last_clean":"2024-05-09T09:18:30.981199+0000","last_became_active":"2024-05-09T09:16:32.892710+0000","last_became_peered":"2024-05-09T09:16:32.892710+0000","last_unstale":"2024-05-09T09:18:30.981199+0000","last_undegraded":"2024-05-09T09:18:30.981199+0000","last_fullsized":"2024-05-09T09:18:30.981199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T16:02:56.480412+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.228022+0000","last_change":"2024-05-09T09:16:32.873815+0000","last_active":"2024-05-09T09:18:30.228022+0000","last_peered":"2024-05-09T09:18:30.228022+0000","last_clean":"2024-05-09T09:18:30.228022+0000","last_became_active":"2024-05-09T09:16:32.873274+0000","last_became_peered":"2024-05-09T09:16:32.873274+0000","last_unstale":"2024-05-09T09:18:30.228022+0000","last_undegraded":"2024-05-09T09:18:30.228022+0000","last_fullsized":"2024-05-09T09:18:30.228022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T14:17:44.654958+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980884+0000","last_change":"2024-05-09T09:16:32.890257+0000","last_active":"2024-05-09T09:18:30.980884+0000","last_peered":"2024-05-09T09:18:30.980884+0000","last_clean":"2024-05-09T09:18:30.980884+0000","last_became_active":"2024-05-09T09:16:32.889929+0000","last_became_peered":"2024-05-09T09:16:32.889929+0000","last_unstale":"2024-05-09T09:18:30.980884+0000","last_undegraded":"2024-05-09T09:18:30.980884+0000","last_fullsized":"2024-05-09T09:18:30.980884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:09:40.697699+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.980558+0000","last_change":"2024-05-09T09:16:32.880476+0000","last_active":"2024-05-09T09:18:30.980558+0000","last_peered":"2024-05-09T09:18:30.980558+0000","last_clean":"2024-05-09T09:18:30.980558+0000","last_became_active":"2024-05-09T09:16:32.880161+0000","last_became_peered":"2024-05-09T09:16:32.880161+0000","last_unstale":"2024-05-09T09:18:30.980558+0000","last_undegraded":"2024-05-09T09:18:30.980558+0000","last_fullsized":"2024-05-09T09:18:30.980558+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T12:03:16.215194+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:30.223438+0000","last_change":"2024-05-09T09:16:32.883509+0000","last_active":"2024-05-09T09:18:30.223438+0000","last_peered":"2024-05-09T09:18:30.223438+0000","last_clean":"2024-05-09T09:18:30.223438+0000","last_became_active":"2024-05-09T09:16:32.883152+0000","last_became_peered":"2024-05-09T09:16:32.883152+0000","last_unstale":"2024-05-09T09:18:30.223438+0000","last_undegraded":"2024-05-09T09:18:30.223438+0000","last_fullsized":"2024-05-09T09:18:30.223438+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T17:38:36.372576+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-05-09T09:18:31.164557+0000","last_change":"2024-05-09T09:16:32.893308+0000","last_active":"2024-05-09T09:18:31.164557+0000","last_peered":"2024-05-09T09:18:31.164557+0000","last_clean":"2024-05-09T09:18:31.164557+0000","last_became_active":"2024-05-09T09:16:32.892986+0000","last_became_peered":"2024-05-09T09:16:32.892986+0000","last_unstale":"2024-05-09T09:18:31.164557+0000","last_undegraded":"2024-05-09T09:18:31.164557+0000","last_fullsized":"2024-05-09T09:18:31.164557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-05-09T09:16:31.850383+0000","last_clean_scrub_stamp":"2024-05-09T09:16:31.850383+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-05-10T18:10:10.841259+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":393216,"data_stored":4132576,"data_compressed":245085,"data_compressed_allocated":315392,"data_compressed_original":4079616,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":68363,"num_objects":60,"num_object_clones":0,"num_object_copies":180,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":60,"num_whiteouts":0,"num_read":700,"num_read_kb":424,"num_write":173,"num_write_kb":125,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1035,"num_keys_recovered":10,"num_objects_omap":40,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":421888,"data_stored":209538,"data_compressed":4887,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":0,"internal_metadata":0},"log_size":150,"ondisk_log_size":150,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383720,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32180,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712972,"statfs":{"total":95995035648,"available":95962083328,"internally_reserved":0,"allocated":1024000,"data_stored":519498,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645358,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28152,"kb_used_data":1136,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717000,"statfs":{"total":95995035648,"available":95966208000,"internally_reserved":0,"allocated":1163264,"data_stored":1170657,"data_compressed":63914,"data_compressed_allocated":86016,"data_compressed_original":671744,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808883,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28032,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717120,"statfs":{"total":95995035648,"available":95966330880,"internally_reserved":0,"allocated":1040384,"data_stored":523808,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1600,"internal_metadata":27654592},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070520,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1032192,"data_stored":519610,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332157,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1060,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1085440,"data_stored":1115682,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495682,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32176,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712976,"statfs":{"total":95995035648,"available":95962087424,"internally_reserved":0,"allocated":1015808,"data_stored":514888,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":34,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28076,"kb_used_data":1056,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717076,"statfs":{"total":95995035648,"available":95966285824,"internally_reserved":0,"allocated":1081344,"data_stored":578403,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1016,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1040384,"data_stored":516276,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411415,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28088,"kb_used_data":1072,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717064,"statfs":{"total":95995035648,"available":95966273536,"internally_reserved":0,"allocated":1097728,"data_stored":579439,"data_compressed":28199,"data_compressed_allocated":40960,"data_compressed_original":81920,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182481,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1028096,"data_stored":515082,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920844,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28028,"kb_used_data":1012,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717124,"statfs":{"total":95995035648,"available":95966334976,"internally_reserved":0,"allocated":1036288,"data_stored":520692,"data_compressed":26570,"data_compressed_allocated":36864,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757319,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1093632,"data_stored":1111578,"data_compressed":62285,"data_compressed_allocated":81920,"data_compressed_original":663552,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34075,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":35715,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":1478,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":69632,"data_stored":63605,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":70014,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":284,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":1798,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":2316,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":2324,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":716,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":102400,"data_stored":65491,"data_compressed":1629,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-05-09T09:18:49.116 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/159002233' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:48 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2696353069' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:49.256 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/446806713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 msgr2=0x7f5fc4100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/446806713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc4100ba0 secure :-1 s=READY pgs=490 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8009900 tx=0x7f5fb802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/446806713 shutdown_connections 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/446806713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 0x7f5fc4108bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/446806713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 0x7f5fc4101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/446806713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc4100ba0 unknown :-1 s=CLOSED pgs=490 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/446806713 >> 172.21.15.69:0/446806713 conn(0x7f5fc40fbd30 msgr2=0x7f5fc40fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/446806713 shutdown_connections 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/446806713 wait complete. 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 Processor -- start 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- start start 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 0x7f5fc4109740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.257 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 0x7f5fc4109c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410d580 con 0x7f5fc4100780 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410d400 con 0x7f5fc4102330 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fcc1f4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5fc410d700 con 0x7f5fc4101970 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.254+0000 7f5fc9f90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40036/0 (socket says 172.21.15.69:40036) 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 -- 172.21.15.69:0/1148519024 learned_addr learned my addr 172.21.15.69:0/1148519024 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fca791640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 0x7f5fc4109c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc978f640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 0x7f5fc4109740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 -- 172.21.15.69:0/1148519024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 msgr2=0x7f5fc4109740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 0x7f5fc4109740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 -- 172.21.15.69:0/1148519024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 msgr2=0x7f5fc4109c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 0x7f5fc4109c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 -- 172.21.15.69:0/1148519024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5fc410a510 con 0x7f5fc4100780 2024-05-09T09:18:49.258 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fc9f90640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8031610 tx=0x7f5fb80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb803f030 con 0x7f5fc4100780 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5fb8003ce0 con 0x7f5fc4100780 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5fc41a1cf0 con 0x7f5fc4100780 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5fb8035070 con 0x7f5fc4100780 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.255+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5fc41a2190 con 0x7f5fc4100780 2024-05-09T09:18:49.259 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.256+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5fc4107aa0 con 0x7f5fc4100780 2024-05-09T09:18:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.257+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5fb8042030 con 0x7f5fc4100780 2024-05-09T09:18:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.258+0000 7f5fb6ffd640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 0x7f5fb00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.258+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f5fb80efa10 con 0x7f5fc4100780 2024-05-09T09:18:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.258+0000 7f5fc978f640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 0x7f5fb00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.260 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.258+0000 7f5fc978f640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 0x7f5fb00a5050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0005fd0 tx=0x7f5fc0005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.262+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5fb80bc5f0 con 0x7f5fc4100780 2024-05-09T09:18:49.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.440+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5fc4062460 con 0x7f5fc4100780 2024-05-09T09:18:49.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.441+0000 7f5fb6ffd640 1 -- 172.21.15.69:0/1148519024 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7f5fb80bbef0 con 0x7f5fc4100780 2024-05-09T09:18:49.442 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:49.442 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:49.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 msgr2=0x7f5fb00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 0x7f5fb00a5050 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f5fc0005fd0 tx=0x7f5fc0005850 comp rx=0 tx=0).stop 2024-05-09T09:18:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 msgr2=0x7f5fc410b100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 crypto rx=0x7f5fb8031610 tx=0x7f5fb80043b0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 shutdown_connections 2024-05-09T09:18:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5fb00a2b90 0x7f5fb00a5050 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5fc4102330 0x7f5fc4109c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5fc4101970 0x7f5fc4109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 --2- 172.21.15.69:0/1148519024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5fc4100780 0x7f5fc410b100 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.446 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 >> 172.21.15.69:0/1148519024 conn(0x7f5fc40fbd30 msgr2=0x7f5fc4104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 shutdown_connections 2024-05-09T09:18:49.447 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.443+0000 7f5fcc1f4640 1 -- 172.21.15.69:0/1148519024 wait complete. 2024-05-09T09:18:49.455 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:18:49.456 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:18:49.561 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 -- 172.21.15.69:0/4112994763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14100770 msgr2=0x7f9d14100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 --2- 172.21.15.69:0/4112994763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14100770 0x7f9d14100b90 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f9d04009900 tx=0x7f9d0402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 -- 172.21.15.69:0/4112994763 shutdown_connections 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 --2- 172.21.15.69:0/4112994763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14102320 0x7f9d14108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 --2- 172.21.15.69:0/4112994763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14101960 0x7f9d14101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 --2- 172.21.15.69:0/4112994763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14100770 0x7f9d14100b90 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 -- 172.21.15.69:0/4112994763 >> 172.21.15.69:0/4112994763 conn(0x7f9d140fbce0 msgr2=0x7f9d140fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 -- 172.21.15.69:0/4112994763 shutdown_connections 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.559+0000 7f9d19d42640 1 -- 172.21.15.69:0/4112994763 wait complete. 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 Processor -- start 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 -- start start 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14100770 0x7f9d1410b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14101960 0x7f9d14109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.562 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1410d5d0 con 0x7f9d14102320 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1410d450 con 0x7f9d14101960 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.560+0000 7f9d19d42640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9d1410d750 con 0x7f9d14100770 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40052/0 (socket says 172.21.15.69:40052) 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 -- 172.21.15.69:0/3804140123 learned_addr learned my addr 172.21.15.69:0/3804140123 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d12ffd640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14101960 0x7f9d14109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d137fe640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14100770 0x7f9d1410b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 -- 172.21.15.69:0/3804140123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14100770 msgr2=0x7f9d1410b0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14100770 0x7f9d1410b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 -- 172.21.15.69:0/3804140123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14101960 msgr2=0x7f9d14109730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14101960 0x7f9d14109730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.563 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 -- 172.21.15.69:0/3804140123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9d1410a500 con 0x7f9d14102320 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d13fff640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009fd0 tx=0x7f9d0c00b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c011830 con 0x7f9d14102320 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9d0c016070 con 0x7f9d14102320 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9d141a1e80 con 0x7f9d14102320 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9d0c00f2e0 con 0x7f9d14102320 2024-05-09T09:18:49.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.561+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9d141a2310 con 0x7f9d14102320 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.562+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9d14107c50 con 0x7f9d14102320 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.563+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9d0c0119d0 con 0x7f9d14102320 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.564+0000 7f9d10ff9640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 0x7f9cfc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.564+0000 7f9d137fe640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 0x7f9cfc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.564+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f9d0c0cead0 con 0x7f9d14102320 2024-05-09T09:18:49.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.564+0000 7f9d137fe640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 0x7f9cfc0a5050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9d04031610 tx=0x7f9d04000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.568+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9d0c09b6b0 con 0x7f9d14102320 2024-05-09T09:18:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.745+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7f9d140624d0 con 0x7f9d14102320 2024-05-09T09:18:49.745 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.745+0000 7f9d10ff9640 1 -- 172.21.15.69:0/3804140123 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) ==== 93+0+10 (secure 0 0 0) 0x7f9d0c09afb0 con 0x7f9d14102320 2024-05-09T09:18:49.746 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:18:49.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 msgr2=0x7f9cfc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.747 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 0x7f9cfc0a5050 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f9d04031610 tx=0x7f9d04000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 msgr2=0x7f9d14109c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c009fd0 tx=0x7f9d0c00b8e0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 shutdown_connections 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9cfc0a2b90 0x7f9cfc0a5050 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9d14102320 0x7f9d14109c70 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9d14101960 0x7f9d14109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 --2- 172.21.15.69:0/3804140123 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9d14100770 0x7f9d1410b0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.747+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 >> 172.21.15.69:0/3804140123 conn(0x7f9d140fbce0 msgr2=0x7f9d14104e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.748+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 shutdown_connections 2024-05-09T09:18:49.748 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.748+0000 7f9d19d42640 1 -- 172.21.15.69:0/3804140123 wait complete. 2024-05-09T09:18:49.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:49 smithi069 ceph-mon[30123]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:49.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:49 smithi069 ceph-mon[30123]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:49.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:49 smithi069 ceph-mon[30123]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:49.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:49 smithi069 ceph-mon[30123]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:49.758 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:49 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1148519024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:49.759 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:18:49.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/1334841554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 msgr2=0x7f8bf8109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/1334841554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf8109a00 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0009ca0 tx=0x7f8bf002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/1334841554 shutdown_connections 2024-05-09T09:18:49.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/1334841554 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf8109a00 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/1334841554 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 0x7f8bf81070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/1334841554 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf81047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/1334841554 >> 172.21.15.69:0/1334841554 conn(0x7f8bf80fbd20 msgr2=0x7f8bf80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/1334841554 shutdown_connections 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.862+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/1334841554 wait complete. 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 Processor -- start 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 -- start start 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf8197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 0x7f8bf81980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d530 con 0x7f8bf8107610 2024-05-09T09:18:49.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d3b0 con 0x7f8bf8104ce0 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bfd0ae640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8bf810d6b0 con 0x7f8bf81023b0 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40078/0 (socket says 172.21.15.69:40078) 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 -- 172.21.15.69:0/4200170739 learned_addr learned my addr 172.21.15.69:0/4200170739 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf6d9d640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf8197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf659c640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 0x7f8bf81980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 -- 172.21.15.69:0/4200170739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 msgr2=0x7f8bf8197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf8197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 -- 172.21.15.69:0/4200170739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 msgr2=0x7f8bf81980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:49.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 0x7f8bf81980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf759e640 1 -- 172.21.15.69:0/4200170739 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8bf819f790 con 0x7f8bf8107610 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.863+0000 7f8bf6d9d640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bf759e640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0005e30 tx=0x7f8bf0031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf0003c30 con 0x7f8bf8107610 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8bf819fa20 con 0x7f8bf8107610 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8bf0003dd0 con 0x7f8bf8107610 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8bf002fde0 con 0x7f8bf8107610 2024-05-09T09:18:49.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.864+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8bf819ffc0 con 0x7f8bf8107610 2024-05-09T09:18:49.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.865+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8bf80fd810 con 0x7f8bf8107610 2024-05-09T09:18:49.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.866+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8bf0004050 con 0x7f8bf8107610 2024-05-09T09:18:49.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.867+0000 7f8bdffff640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 0x7f8be00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:49.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.867+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f8bf00f07e0 con 0x7f8bf8107610 2024-05-09T09:18:49.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.867+0000 7f8bf6d9d640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 0x7f8be00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:49.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.867+0000 7f8bf6d9d640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 0x7f8be00a5050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8be80058c0 tx=0x7f8be8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:49.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:49.871+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8bf00bd3c0 con 0x7f8bf8107610 2024-05-09T09:18:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.051+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f8bf8062430 con 0x7f8bf8107610 2024-05-09T09:18:50.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.051+0000 7f8bdffff640 1 -- 172.21.15.69:0/4200170739 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) ==== 104+0+11 (secure 0 0 0) 0x7f8bf00bccc0 con 0x7f8bf8107610 2024-05-09T09:18:50.051 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 msgr2=0x7f8be00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 0x7f8be00a5050 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f8be80058c0 tx=0x7f8be8005850 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 msgr2=0x7f8bf819f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f8bf0005e30 tx=0x7f8bf0031ee0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 shutdown_connections 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8be00a2b90 0x7f8be00a5050 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8bf8107610 0x7f8bf819f120 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8bf8104ce0 0x7f8bf81980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.053+0000 7f8bfd0ae640 1 --2- 172.21.15.69:0/4200170739 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8bf81023b0 0x7f8bf8197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.054+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 >> 172.21.15.69:0/4200170739 conn(0x7f8bf80fbd20 msgr2=0x7f8bf8102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.054+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 shutdown_connections 2024-05-09T09:18:50.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.054+0000 7f8bfd0ae640 1 -- 172.21.15.69:0/4200170739 wait complete. 2024-05-09T09:18:50.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:49 smithi163 ceph-mon[34354]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:49 smithi163 ceph-mon[34354]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:49 smithi163 ceph-mon[34354]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:49 smithi163 ceph-mon[34354]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:50.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:49 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1148519024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:50.066 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:18:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:49 smithi103 ceph-mon[34421]: from='client.15522 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:49 smithi103 ceph-mon[34421]: from='client.15528 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:49 smithi103 ceph-mon[34421]: from='client.15534 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-05-09T09:18:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:49 smithi103 ceph-mon[34421]: pgmap v363: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 426 B/s wr, 6 op/s 2024-05-09T09:18:50.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:49 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1148519024' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:50.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- 172.21.15.69:0/3776120272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624102340 msgr2=0x7f9624108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- 172.21.15.69:0/3776120272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624102340 0x7f9624108bd0 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f961c009ca0 tx=0x7f961c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- 172.21.15.69:0/3776120272 shutdown_connections 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- 172.21.15.69:0/3776120272 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624102340 0x7f9624108bd0 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- 172.21.15.69:0/3776120272 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624101980 0x7f9624101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- 172.21.15.69:0/3776120272 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624100790 0x7f9624100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- 172.21.15.69:0/3776120272 >> 172.21.15.69:0/3776120272 conn(0x7f96240fbd20 msgr2=0x7f96240fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- 172.21.15.69:0/3776120272 shutdown_connections 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- 172.21.15.69:0/3776120272 wait complete. 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 Processor -- start 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- start start 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 0x7f962406a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624102340 0x7f962419ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f962410c410 con 0x7f9624101980 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f962410c290 con 0x7f9624100790 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f96297f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f962410c590 con 0x7f9624102340 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.173+0000 7f961bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40092/0 (socket says 172.21.15.69:40092) 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 -- 172.21.15.69:0/1986966151 learned_addr learned my addr 172.21.15.69:0/1986966151 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f96237fe640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624102340 0x7f962419ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f9622ffd640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 0x7f962406a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 -- 172.21.15.69:0/1986966151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624102340 msgr2=0x7f962419ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624102340 0x7f962419ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 -- 172.21.15.69:0/1986966151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 msgr2=0x7f962406a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 0x7f962406a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 -- 172.21.15.69:0/1986966151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f962419f7e0 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f9622ffd640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 0x7f962406a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f961bfff640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e9f0 tx=0x7f960c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c00cd80 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f960c010910 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f962419fad0 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f960c018990 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.174+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f96241a0020 con 0x7f9624101980 2024-05-09T09:18:50.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.175+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9624107a50 con 0x7f9624101980 2024-05-09T09:18:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.177+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f960c010ac0 con 0x7f9624101980 2024-05-09T09:18:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.178+0000 7f9620ff9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 0x7f96100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.178+0000 7f9622ffd640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 0x7f96100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.178+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f960c0cf760 con 0x7f9624101980 2024-05-09T09:18:50.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.178+0000 7f9622ffd640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 0x7f96100a5050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f96241017e0 tx=0x7f9614005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.182+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f960c09c340 con 0x7f9624101980 2024-05-09T09:18:50.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.365+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f96240624d0 con 0x7f9624101980 2024-05-09T09:18:50.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.366+0000 7f9620ff9640 1 -- 172.21.15.69:0/1986966151 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) ==== 100+0+11 (secure 0 0 0) 0x7f960c09bc40 con 0x7f9624101980 2024-05-09T09:18:50.366 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 msgr2=0x7f96100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 0x7f96100a5050 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f96241017e0 tx=0x7f9614005f90 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 msgr2=0x7f962419c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f960c00e9f0 tx=0x7f960c00eeb0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 shutdown_connections 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96100a2b90 0x7f96100a5050 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9624102340 0x7f962419ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9624101980 0x7f962419c8b0 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 --2- 172.21.15.69:0/1986966151 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9624100790 0x7f962406a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 >> 172.21.15.69:0/1986966151 conn(0x7f96240fbd20 msgr2=0x7f9624104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 shutdown_connections 2024-05-09T09:18:50.368 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.368+0000 7f96297f9640 1 -- 172.21.15.69:0/1986966151 wait complete. 2024-05-09T09:18:50.380 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.488+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3963653266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 msgr2=0x7f9b5c108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.488+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3963653266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c108bb0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 crypto rx=0x7f9b58009ca0 tx=0x7f9b5802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3963653266 shutdown_connections 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3963653266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c108bb0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3963653266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c101960 0x7f9b5c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3963653266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c100770 0x7f9b5c100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3963653266 >> 172.21.15.69:0/3963653266 conn(0x7f9b5c0fbce0 msgr2=0x7f9b5c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3963653266 shutdown_connections 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3963653266 wait complete. 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 Processor -- start 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- start start 2024-05-09T09:18:50.491 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c100770 0x7f9b5c069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c101960 0x7f9b5c19ecc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c10c3d0 con 0x7f9b5c102320 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c10c250 con 0x7f9b5c101960 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b63bd5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5c10c550 con 0x7f9b5c100770 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.489+0000 7f9b62172640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40124/0 (socket says 172.21.15.69:40124) 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 -- 172.21.15.69:0/3846671297 learned_addr learned my addr 172.21.15.69:0/3846671297 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b61170640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c101960 0x7f9b5c19ecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b61971640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c100770 0x7f9b5c069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 -- 172.21.15.69:0/3846671297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c100770 msgr2=0x7f9b5c069530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c100770 0x7f9b5c069530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 -- 172.21.15.69:0/3846671297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c101960 msgr2=0x7f9b5c19ecc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c101960 0x7f9b5c19ecc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.492 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 -- 172.21.15.69:0/3846671297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5c1a1a00 con 0x7f9b5c102320 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b62172640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9b58005e30 tx=0x7f9b58031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b58003c30 con 0x7f9b5c102320 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5c1a1c30 con 0x7f9b5c102320 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b58003dd0 con 0x7f9b5c102320 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b5802fde0 con 0x7f9b5c102320 2024-05-09T09:18:50.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.490+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5c1a2180 con 0x7f9b5c102320 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.491+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f9b5c107a90 con 0x7f9b5c102320 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.492+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9b58004050 con 0x7f9b5c102320 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.493+0000 7f9b52ffd640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 0x7f9b480a5080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.493+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f9b580f06d0 con 0x7f9b5c102320 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.493+0000 7f9b61971640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 0x7f9b480a5080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.494+0000 7f9b61971640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 0x7f9b480a5080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c1017c0 tx=0x7f9b5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.497+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f9b580bd2b0 con 0x7f9b5c102320 2024-05-09T09:18:50.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.678+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f9b5c0624d0 con 0x7f9b5c102320 2024-05-09T09:18:50.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.679+0000 7f9b52ffd640 1 -- 172.21.15.69:0/3846671297 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) ==== 103+0+11 (secure 0 0 0) 0x7f9b580bcbb0 con 0x7f9b5c102320 2024-05-09T09:18:50.679 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 msgr2=0x7f9b480a5080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 0x7f9b480a5080 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f9b5c1017c0 tx=0x7f9b5400b390 comp rx=0 tx=0).stop 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 msgr2=0x7f9b5c1a1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 crypto rx=0x7f9b58005e30 tx=0x7f9b58031ee0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 shutdown_connections 2024-05-09T09:18:50.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b480a2bc0 0x7f9b480a5080 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b5c102320 0x7f9b5c1a1210 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b5c101960 0x7f9b5c19ecc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 --2- 172.21.15.69:0/3846671297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b5c100770 0x7f9b5c069530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 >> 172.21.15.69:0/3846671297 conn(0x7f9b5c0fbce0 msgr2=0x7f9b5c104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 shutdown_connections 2024-05-09T09:18:50.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.681+0000 7f9b63bd5640 1 -- 172.21.15.69:0/3846671297 wait complete. 2024-05-09T09:18:50.692 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 -- 172.21.15.69:0/3373711638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c1022f0 msgr2=0x7f8a9c1046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 --2- 172.21.15.69:0/3373711638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c1046e0 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c009900 tx=0x7f8a8c02a410 comp rx=0 tx=0).stop 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 -- 172.21.15.69:0/3373711638 shutdown_connections 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 --2- 172.21.15.69:0/3373711638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c107550 0x7f8a9c1099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 --2- 172.21.15.69:0/3373711638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c107010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 --2- 172.21.15.69:0/3373711638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c1046e0 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 -- 172.21.15.69:0/3373711638 >> 172.21.15.69:0/3373711638 conn(0x7f8a9c0fbce0 msgr2=0x7f8a9c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 -- 172.21.15.69:0/3373711638 shutdown_connections 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.796+0000 7f8aa2122640 1 -- 172.21.15.69:0/3373711638 wait complete. 2024-05-09T09:18:50.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 Processor -- start 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 -- start start 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c1980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c10d570 con 0x7f8a9c107550 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c10d3f0 con 0x7f8a9c1022f0 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8aa2122640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a9c10d6f0 con 0x7f8a9c104c20 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8a9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40152/0 (socket says 172.21.15.69:40152) 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.797+0000 7f8a9bfff640 1 -- 172.21.15.69:0/1530968294 learned_addr learned my addr 172.21.15.69:0/1530968294 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9b7fe640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a92ffd640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c1980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 -- 172.21.15.69:0/1530968294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 msgr2=0x7f8a9c1980f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c1980f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 -- 172.21.15.69:0/1530968294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c1022f0 msgr2=0x7f8a9c197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 -- 172.21.15.69:0/1530968294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a9c19f7d0 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a92ffd640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a9bfff640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400cce0 tx=0x7f8a9400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a9400f450 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a9400fad0 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a9c19fac0 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a94018c10 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.798+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a9c1a0040 con 0x7f8a9c107550 2024-05-09T09:18:50.801 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.799+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8a9c0fd7d0 con 0x7f8a9c107550 2024-05-09T09:18:50.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.800+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8a9400f5f0 con 0x7f8a9c107550 2024-05-09T09:18:50.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.801+0000 7f8a997fa640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 0x7f8a880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:50.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.801+0000 7f8a9b7fe640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 0x7f8a880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:50.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.801+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f8a940cfb30 con 0x7f8a9c107550 2024-05-09T09:18:50.802 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.801+0000 7f8a9b7fe640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 0x7f8a880a5050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c000c00 tx=0x7f8a8c031400 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:50.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:50.805+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8a94014030 con 0x7f8a9c107550 2024-05-09T09:18:51.009 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3804140123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4200170739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:50 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1986966151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.023+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f8a9c02d1c0 con 0x7f8a9c107550 2024-05-09T09:18:51.023 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.024+0000 7f8a997fa640 1 -- 172.21.15.69:0/1530968294 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11879 (secure 0 0 0) 0x7f8a9409c710 con 0x7f8a9c107550 2024-05-09T09:18:51.024 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:51.024 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:18:44.567983+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24491,24511]},"id":1}]} 2024-05-09T09:18:51.024 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 msgr2=0x7f8a880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 0x7f8a880a5050 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f8a8c000c00 tx=0x7f8a8c031400 comp rx=0 tx=0).stop 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 msgr2=0x7f8a9c19f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f8a9400cce0 tx=0x7f8a9400a620 comp rx=0 tx=0).stop 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 shutdown_connections 2024-05-09T09:18:51.026 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a880a2b90 0x7f8a880a5050 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a9c107550 0x7f8a9c19f160 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a9c104c20 0x7f8a9c1980f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 --2- 172.21.15.69:0/1530968294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a9c1022f0 0x7f8a9c197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 >> 172.21.15.69:0/1530968294 conn(0x7f8a9c0fbce0 msgr2=0x7f8a9c102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 shutdown_connections 2024-05-09T09:18:51.027 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.026+0000 7f8aa2122640 1 -- 172.21.15.69:0/1530968294 wait complete. 2024-05-09T09:18:51.037 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:18:51.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3804140123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4200170739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:50 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1986966151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3804140123' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4200170739' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:50 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1986966151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:18:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 -- 172.21.15.69:0/4139182778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc101960 msgr2=0x7f4afc101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/4139182778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc101960 0x7f4afc101de0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 crypto rx=0x7f4ae4009900 tx=0x7f4ae402a410 comp rx=0 tx=0).stop 2024-05-09T09:18:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 -- 172.21.15.69:0/4139182778 shutdown_connections 2024-05-09T09:18:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/4139182778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc102320 0x7f4afc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.144 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/4139182778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc101960 0x7f4afc101de0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.145 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/4139182778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 0x7f4afc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 -- 172.21.15.69:0/4139182778 >> 172.21.15.69:0/4139182778 conn(0x7f4afc0fbce0 msgr2=0x7f4afc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 -- 172.21.15.69:0/4139182778 shutdown_connections 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.141+0000 7f4b02a4d640 1 -- 172.21.15.69:0/4139182778 wait complete. 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 Processor -- start 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 -- start start 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 0x7f4afc197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc101960 0x7f4afc198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.146 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.147 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10c3d0 con 0x7f4afc102320 2024-05-09T09:18:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10c250 con 0x7f4afc101960 2024-05-09T09:18:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b02a4d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4afc10c550 con 0x7f4afc100770 2024-05-09T09:18:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b00fea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.148 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b00fea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40170/0 (socket says 172.21.15.69:40170) 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4b00fea640 1 -- 172.21.15.69:0/1229222662 learned_addr learned my addr 172.21.15.69:0/1229222662 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.142+0000 7f4afbfff640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 0x7f4afc197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4af3fff640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc101960 0x7f4afc198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 -- 172.21.15.69:0/1229222662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 msgr2=0x7f4afc197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 0x7f4afc197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 -- 172.21.15.69:0/1229222662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc101960 msgr2=0x7f4afc198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc101960 0x7f4afc198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 -- 172.21.15.69:0/1229222662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4afc19f760 con 0x7f4afc102320 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b00fea640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4af400cce0 tx=0x7f4af400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af400f450 con 0x7f4afc102320 2024-05-09T09:18:51.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4afc19fa50 con 0x7f4afc102320 2024-05-09T09:18:51.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4af400fad0 con 0x7f4afc102320 2024-05-09T09:18:51.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4af4018c10 con 0x7f4afc102320 2024-05-09T09:18:51.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.143+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4afc19ff20 con 0x7f4afc102320 2024-05-09T09:18:51.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.144+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4afc1a0210 con 0x7f4afc102320 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.145+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4af400f5f0 con 0x7f4afc102320 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.146+0000 7f4af9ffb640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 0x7f4ae80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.146+0000 7f4afbfff640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 0x7f4ae80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.146+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f4af40cfa00 con 0x7f4afc102320 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.147+0000 7f4afbfff640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 0x7f4ae80a5050 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4afc1017c0 tx=0x7f4aec00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.152 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.151+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f4af4014030 con 0x7f4afc102320 2024-05-09T09:18:51.335 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.335+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f4afc1a03e0 con 0x7f4afc102320 2024-05-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.337+0000 7f4af9ffb640 1 -- 172.21.15.69:0/1229222662 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) ==== 74+0+24708 (secure 0 0 0) 0x7f4af409c5e0 con 0x7f4afc102320 2024-05-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:18:51.337 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":93,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:18:30.220267+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 msgr2=0x7f4ae80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 0x7f4ae80a5050 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f4afc1017c0 tx=0x7f4aec00b390 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 msgr2=0x7f4afc19f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 crypto rx=0x7f4af400cce0 tx=0x7f4af400a620 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 shutdown_connections 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ae80a2b90 0x7f4ae80a5050 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4afc102320 0x7f4afc19f0f0 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4afc101960 0x7f4afc198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 --2- 172.21.15.69:0/1229222662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4afc100770 0x7f4afc197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.339+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 >> 172.21.15.69:0/1229222662 conn(0x7f4afc0fbce0 msgr2=0x7f4afc0689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.340+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 shutdown_connections 2024-05-09T09:18:51.343 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.340+0000 7f4b02a4d640 1 -- 172.21.15.69:0/1229222662 wait complete. 2024-05-09T09:18:51.352 DEBUG:tasks.fwd_scrub:Yielding 2024-05-09T09:18:51.352 INFO:teuthology.run_tasks:Running task background_exec... 2024-05-09T09:18:51.377 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:18:51.377 DEBUG:teuthology.orchestra.run.smithi069:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-05-09T09:18:51.378 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-05-09T09:18:51.378 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:18:51.378 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:18:51.378 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:18:51.380 INFO:teuthology.run_tasks:Running task workunit... 2024-05-09T09:18:51.398 INFO:tasks.workunit:Pulling workunits from ref be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T09:18:51.398 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-05-09T09:18:51.398 INFO:tasks.workunit:timeout=3h 2024-05-09T09:18:51.398 INFO:tasks.workunit:cleanup=True 2024-05-09T09:18:51.399 DEBUG:teuthology.orchestra.run.smithi069:> stat -- /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout:Device: 72h/114d Inode: 1099511628283 Links: 2 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-05-09T09:18:51.422 INFO:teuthology.orchestra.run.smithi069.stdout:Access: 2024-05-09 09:16:54.628907604 +0000 2024-05-09T09:18:51.423 INFO:teuthology.orchestra.run.smithi069.stdout:Modify: 2024-05-09 09:16:54.628907604 +0000 2024-05-09T09:18:51.423 INFO:teuthology.orchestra.run.smithi069.stdout:Change: 2024-05-09 09:18:43.886087785 +0000 2024-05-09T09:18:51.423 INFO:teuthology.orchestra.run.smithi069.stdout: Birth: 2024-05-09 09:16:54.628907604 +0000 2024-05-09T09:18:51.423 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-05-09T09:18:51.423 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-05-09T09:18:51.488 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.488+0000 7fbfdebaf640 1 -- 172.21.15.69:0/356335634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd8104c90 msgr2=0x7fbfd8107080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.488+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/356335634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd8104c90 0x7fbfd8107080 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009900 tx=0x7fbfc802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- 172.21.15.69:0/356335634 shutdown_connections 2024-05-09T09:18:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/356335634 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd81075c0 0x7fbfd81099b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.489 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/356335634 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd8104c90 0x7fbfd8107080 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.491 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout be85c202969d25af6e130678e7d73a20e9c4ca11 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/356335634 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 0x7fbfd8104750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- 172.21.15.69:0/356335634 >> 172.21.15.69:0/356335634 conn(0x7fbfd80fbcf0 msgr2=0x7fbfd80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- 172.21.15.69:0/356335634 shutdown_connections 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- 172.21.15.69:0/356335634 wait complete. 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 Processor -- start 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- start start 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 0x7fbfd8101cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd8104c90 0x7fbfd8100310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd810d500 con 0x7fbfd81075c0 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd810d380 con 0x7fbfd8102360 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdebaf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd810d680 con 0x7fbfd8104c90 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdd14c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdd14c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40190/0 (socket says 172.21.15.69:40190) 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.489+0000 7fbfdd14c640 1 -- 172.21.15.69:0/3779817817 learned_addr learned my addr 172.21.15.69:0/3779817817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfcffff640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd8104c90 0x7fbfd8100310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 -- 172.21.15.69:0/3779817817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd8104c90 msgr2=0x7fbfd8100310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 0x7fbfd8101cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd8104c90 0x7fbfd8100310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 -- 172.21.15.69:0/3779817817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 msgr2=0x7fbfd8101cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 0x7fbfd8101cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 -- 172.21.15.69:0/3779817817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbfd81010e0 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdd14c640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e970 tx=0x7fbfd000ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfcdffb640 1 -- 172.21.15.69:0/3779817817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd000ccc0 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbfd81a1cf0 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfcdffb640 1 -- 172.21.15.69:0/3779817817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfd000ce60 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfcdffb640 1 -- 172.21.15.69:0/3779817817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd0010450 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.490+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbfd81a21b0 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.492+0000 7fbfcdffb640 1 -- 172.21.15.69:0/3779817817 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbfd0005020 con 0x7fbfd81075c0 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 Processor -- start 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 -- start start 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c97246a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c9724724b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c9723ce990 con 0x55c97246aeb0 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c9723765b0 con 0x55c97246a4c0 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfe1995740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c9723ca950 con 0x55c972470020 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40198/0 (socket says 172.21.15.69:40198) 2024-05-09T09:18:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 -- 172.21.15.69:0/1584432448 learned_addr learned my addr 172.21.15.69:0/1584432448 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfcdffb640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 0x7fbfc00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c97246a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c9724724b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 -- 172.21.15.69:0/1584432448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 msgr2=0x55c9724724b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdd14c640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 0x7fbfc00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c9724724b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 -- 172.21.15.69:0/1584432448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 msgr2=0x55c97246a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c97246a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 -- 172.21.15.69:0/1584432448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c9721aa7e0 con 0x55c97246aeb0 2024-05-09T09:18:51.500 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c9724724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:51.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfcdffb640 1 -- 172.21.15.69:0/3779817817 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbfd00cf990 con 0x7fbfd81075c0 2024-05-09T09:18:51.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.493+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009f30 tx=0x7fbfd400f7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98db7361fb7ce8aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfbeffd640 1 -- 172.21.15.69:0/1584432448 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd4011640 con 0x55c97246aeb0 2024-05-09T09:18:51.501 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfdd14c640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 0x7fbfc00a5050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00cd160 tx=0x7fbfd00c6a00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfbeffd640 1 -- 172.21.15.69:0/1584432448 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfd4011cc0 con 0x55c97246aeb0 2024-05-09T09:18:51.502 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfbeffd640 1 -- 172.21.15.69:0/1584432448 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfd401bd40 con 0x55c97246aeb0 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfe1995740 1 -- 172.21.15.69:0/1584432448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 msgr2=0x55c97246f9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.494+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 crypto rx=0x7fbfd4009f30 tx=0x7fbfd400f7f0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- 172.21.15.69:0/1584432448 shutdown_connections 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c9724724b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246aeb0 0x55c97246f9b0 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1584432448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c97246a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- 172.21.15.69:0/1584432448 >> 172.21.15.69:0/1584432448 conn(0x55c9724320a0 msgr2=0x55c972465f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- 172.21.15.69:0/1584432448 shutdown_connections 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- 172.21.15.69:0/1584432448 wait complete. 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- start start 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 Processor -- start 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c972474e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c972470020 0x55c97247be60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c972472fd0 con 0x55c97246a4c0 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c972389b40 con 0x55c972470020 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c972473150 con 0x55c97246aeb0 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfe1995740 10 client.0 _ll_register_callbacks cb 0x55c97237e0f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:51.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.495+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40204/0 (socket says 172.21.15.69:40204) 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 -- 172.21.15.69:0/1265466199 learned_addr learned my addr 172.21.15.69:0/1265466199 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c972474e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c972470020 0x55c97247be60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 -- 172.21.15.69:0/1265466199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 msgr2=0x55c972474e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c972474e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 -- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c972470020 msgr2=0x55c97247be60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c972470020 0x55c97247be60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 -- 172.21.15.69:0/1265466199 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97220af70 con 0x55c97246a4c0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c972474e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbfd806a8d0 tx=0x7fbfc802a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3764fe02312d7058 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfb7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035040 con 0x55c97246a4c0 2024-05-09T09:18:51.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbfc80035f0 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfd4011cc0 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbfc8050050 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfe1995740 10 client.15588 fetch_fsmap learned FSMap version 25 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.496+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c9720a2170 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfd4011cc0 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbfc803a040 con 0x55c97246a4c0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 10 client.15588 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 10 client.15588 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 20 client.15588 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c972480270 0x55c9724826e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 4 client.15588 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c9723dc0c0 con 0x55c972480270 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c972480270 0x55c9724826e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c972480270 0x55c9724826e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49b71bf6023e496 server_cookie=a85a2e66cfd29701 in_seq=0 out_seq=0 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.497+0000 7fbfb7fff640 10 client.15588 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1265466199 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c9723dc0c0 con 0x55c972480270 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfb7fff640 10 client.15588 handle_command_reply: tid=0 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 client.15588 shutdown 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 msgr2=0x55c9724748e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 crypto rx=0x7fbfd806a8d0 tx=0x7fbfc802a410 comp rx=0 tx=0).stop 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 shutdown_connections 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c972470020 0x55c97247be60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c972480270 0x55c9724826e0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c972474e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfb7fff640 0 client.15588 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.511 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1265466199 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c9724748e0 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 >> 172.21.15.69:0/1265466199 conn(0x55c9724320a0 msgr2=0x55c972465080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 shutdown_connections 2024-05-09T09:18:51.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.498+0000 7fbfe1995740 1 -- 172.21.15.69:0/1265466199 wait complete. 2024-05-09T09:18:51.512 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.499+0000 7fbfe1995740 20 client.15588 trim_cache size 0 max 16384 2024-05-09T09:18:51.512 INFO:tasks.workunit.client.0.smithi069.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-05-09T09:18:51.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 Processor -- start 2024-05-09T09:18:51.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 -- start start 2024-05-09T09:18:51.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c972469620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c97247be20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81aeef0 con 0x55c972470020 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81ac7c0 con 0x55c97246a4c0 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.516+0000 7fbfe1995740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81aa610 con 0x55c97246aeb0 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40218/0 (socket says 172.21.15.69:40218) 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 -- 172.21.15.69:0/3245080560 learned_addr learned my addr 172.21.15.69:0/3245080560 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c97247be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c972469620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 -- 172.21.15.69:0/3245080560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 msgr2=0x55c97247be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c97247be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 -- 172.21.15.69:0/3245080560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 msgr2=0x55c972469620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c972469620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 -- 172.21.15.69:0/3245080560 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c972466190 con 0x55c972470020 2024-05-09T09:18:51.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfcffff640 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009900 tx=0x7fbfc8000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edea0a2b88eedf8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfbe7fc640 1 -- 172.21.15.69:0/3245080560 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035070 con 0x55c972470020 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfbe7fc640 1 -- 172.21.15.69:0/3245080560 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfc803dba0 con 0x55c972470020 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.517+0000 7fbfbe7fc640 1 -- 172.21.15.69:0/3245080560 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8003de0 con 0x55c972470020 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- 172.21.15.69:0/3245080560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 msgr2=0x55c972486fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 crypto rx=0x7fbfc8009900 tx=0x7fbfc8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- 172.21.15.69:0/3245080560 shutdown_connections 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c972470020 0x55c972486fa0 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c97246aeb0 0x55c97247be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 --2- 172.21.15.69:0/3245080560 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246a4c0 0x55c972469620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- 172.21.15.69:0/3245080560 >> 172.21.15.69:0/3245080560 conn(0x55c9724320a0 msgr2=0x55c972468f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- 172.21.15.69:0/3245080560 shutdown_connections 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- 172.21.15.69:0/3245080560 wait complete. 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 -- start start 2024-05-09T09:18:51.520 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.518+0000 7fbfe1995740 1 Processor -- start 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246aeb0 0x55c97247bcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c972479d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81aac80 con 0x55c97246a4c0 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81ac0f0 con 0x55c97246aeb0 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbfd81ab350 con 0x55c972470020 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 10 client.0 _ll_register_callbacks cb 0x7fbfd40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40222/0 (socket says 172.21.15.69:40222) 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 -- 172.21.15.69:0/1762703731 learned_addr learned my addr 172.21.15.69:0/1762703731 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c972479d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246aeb0 0x55c97247bcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 -- 172.21.15.69:0/1762703731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 msgr2=0x55c972479d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c972479d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 -- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246aeb0 msgr2=0x55c97247bcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246aeb0 0x55c97247bcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 -- 172.21.15.69:0/1762703731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c97247a5b0 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfcffff640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbfc802ab60 tx=0x7fbfc805a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=45d981e37e584482 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfb7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbfc8035040 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbfc8003b60 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.519+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbfc803dba0 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbfc8003d60 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 10 client.15594 fetch_fsmap learned FSMap version 25 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55c9724e7300 con 0x55c97246a4c0 2024-05-09T09:18:51.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbfc803dba0 con 0x55c97246a4c0 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbfc8054730 con 0x55c97246a4c0 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 10 client.15594 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 10 client.15594 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 20 client.15594 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c9724e7c40 0x55c9724ea060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 4 client.15594 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55c9723d6410 con 0x55c9724e7c40 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.520+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c9724e7c40 0x55c9724ea060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.521+0000 7fbfb7fff640 10 client.15594 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.521+0000 7fbfdc94b640 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c9724e7c40 0x55c9724ea060 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95cd30a24d359bad server_cookie=7b0d801c7a776434 in_seq=0 out_seq=0 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfb7fff640 1 -- 172.21.15.69:0/1762703731 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55c9723d6410 con 0x55c9724e7c40 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfb7fff640 10 client.15594 handle_command_reply: tid=0 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 client.15594 shutdown 2024-05-09T09:18:51.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 msgr2=0x55c97247b770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7fbfc802ab60 tx=0x7fbfc805a040 comp rx=0 tx=0).stop 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 shutdown_connections 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c9724e7c40 0x55c9724ea060 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfb7fff640 0 client.15594 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c972470020 0x55c972479d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c97246aeb0 0x55c97247bcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 --2- 172.21.15.69:0/1762703731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c97246a4c0 0x55c97247b770 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 >> 172.21.15.69:0/1762703731 conn(0x55c9724320a0 msgr2=0x55c9724697a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 shutdown_connections 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 1 -- 172.21.15.69:0/1762703731 wait complete. 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.522+0000 7fbfe1995740 20 client.15594 trim_cache size 0 max 16384 2024-05-09T09:18:51.527 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:51.528 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-05-09T09:18:51.528 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "9e61dd6f-327b-4bda-a963-bba023a27344", 2024-05-09T09:18:51.528 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-05-09T09:18:51.528 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 msgr2=0x7fbfc00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 0x7fbfc00a5050 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fbfd00cd160 tx=0x7fbfd00c6a00 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 msgr2=0x7fbfd8100850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 crypto rx=0x7fbfd000e970 tx=0x7fbfd000ed70 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 shutdown_connections 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbfc00a2b90 0x7fbfc00a5050 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbfd81075c0 0x7fbfd8100850 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbfd8104c90 0x7fbfd8100310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 --2- 172.21.15.69:0/3779817817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbfd8102360 0x7fbfd8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 >> 172.21.15.69:0/3779817817 conn(0x7fbfd80fbcf0 msgr2=0x7fbfd8102e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 shutdown_connections 2024-05-09T09:18:51.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.523+0000 7fbfdebaf640 1 -- 172.21.15.69:0/3779817817 wait complete. 2024-05-09T09:18:51.534 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:18:51.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:18:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 -- 172.21.15.69:0/2498505453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0101880 msgr2=0x7f24c0101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 --2- 172.21.15.69:0/2498505453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0101880 0x7f24c0101ca0 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f24b0009900 tx=0x7f24b002a410 comp rx=0 tx=0).stop 2024-05-09T09:18:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 -- 172.21.15.69:0/2498505453 shutdown_connections 2024-05-09T09:18:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 --2- 172.21.15.69:0/2498505453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0103430 0x7f24c0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.683 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 --2- 172.21.15.69:0/2498505453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0102a70 0x7f24c0102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 --2- 172.21.15.69:0/2498505453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0101880 0x7f24c0101ca0 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 -- 172.21.15.69:0/2498505453 >> 172.21.15.69:0/2498505453 conn(0x7f24c00fce10 msgr2=0x7f24c00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 -- 172.21.15.69:0/2498505453 shutdown_connections 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 -- 172.21.15.69:0/2498505453 wait complete. 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.681+0000 7f24c69c8640 1 Processor -- start 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 -- start start 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0101880 0x7f24c0197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0102a70 0x7f24c01981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d570 con 0x7f24c0103430 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d3f0 con 0x7f24c0102a70 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c69c8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24c010d6f0 con 0x7f24c0101880 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.684 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40248/0 (socket says 172.21.15.69:40248) 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 -- 172.21.15.69:0/1130388227 learned_addr learned my addr 172.21.15.69:0/1130388227 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24bffff640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0101880 0x7f24c0197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0102a70 0x7f24c01981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 -- 172.21.15.69:0/1130388227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0101880 msgr2=0x7f24c0197c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0101880 0x7f24c0197c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 -- 172.21.15.69:0/1130388227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0102a70 msgr2=0x7f24c01981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0102a70 0x7f24c01981c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 -- 172.21.15.69:0/1130388227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f24c019f8c0 con 0x7f24c0103430 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.682+0000 7f24c4f65640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f24b800b970 tx=0x7f24b800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.683+0000 7f24bd7fa640 1 -- 172.21.15.69:0/1130388227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8019070 con 0x7f24c0103430 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.683+0000 7f24bd7fa640 1 -- 172.21.15.69:0/1130388227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b800cb40 con 0x7f24c0103430 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.683+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f24c019fbb0 con 0x7f24c0103430 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.683+0000 7f24bd7fa640 1 -- 172.21.15.69:0/1130388227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b8007650 con 0x7f24c0103430 2024-05-09T09:18:51.685 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.683+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f24c01a0100 con 0x7f24c0103430 2024-05-09T09:18:51.693 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.685+0000 7f24bd7fa640 1 -- 172.21.15.69:0/1130388227 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f24b801c070 con 0x7f24c0103430 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 Processor -- start 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 -- start start 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193340 0x564930193760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193d30 0x564930198830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5649300f3d00 con 0x564930198ea0 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56493009b4c0 con 0x564930193d30 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c97b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5649300efcc0 con 0x564930193340 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.694 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40262/0 (socket says 172.21.15.69:40262) 2024-05-09T09:18:51.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 -- 172.21.15.69:0/3447874362 learned_addr learned my addr 172.21.15.69:0/3447874362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bffff640 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193d30 0x564930198830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193340 0x564930193760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bd7fa640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 0x7f24a80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.695 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 -- 172.21.15.69:0/3447874362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193340 msgr2=0x564930193760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193340 0x564930193760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 -- 172.21.15.69:0/3447874362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193d30 msgr2=0x564930198830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193d30 0x564930198830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24bf7fe640 1 -- 172.21.15.69:0/3447874362 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56492fed17d0 con 0x564930198ea0 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.686+0000 7f24c4f65640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 0x7f24a80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f24b4005d00 tx=0x7f24b4009590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc9c67950e6f140 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.696 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24bd7fa640 1 -- 172.21.15.69:0/1130388227 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f24b80ce9a0 con 0x7f24c0103430 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24a67fc640 1 -- 172.21.15.69:0/3447874362 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b401b070 con 0x564930198ea0 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24a67fc640 1 -- 172.21.15.69:0/3447874362 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b40139a0 con 0x564930198ea0 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24a67fc640 1 -- 172.21.15.69:0/3447874362 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b4010a40 con 0x564930198ea0 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 -- 172.21.15.69:0/3447874362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 msgr2=0x56493019b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c4f65640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 0x7f24a80a5050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f24b8097900 tx=0x7f24b80bb040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f24b4005d00 tx=0x7f24b4009590 comp rx=0 tx=0).stop 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 -- 172.21.15.69:0/3447874362 shutdown_connections 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x56493019b330 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193d30 0x564930198830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3447874362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193340 0x564930193760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.697 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 -- 172.21.15.69:0/3447874362 >> 172.21.15.69:0/3447874362 conn(0x56493018a5a0 msgr2=0x56493018a9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.687+0000 7f24c97b3740 1 -- 172.21.15.69:0/3447874362 shutdown_connections 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 -- 172.21.15.69:0/3447874362 wait complete. 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 -- start start 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 Processor -- start 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x56493019d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193d30 0x56493019dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56493019be50 con 0x564930198ea0 2024-05-09T09:18:51.698 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5649300aeb90 con 0x564930193340 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56493019bfd0 con 0x564930193d30 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24c97b3740 10 client.0 _ll_register_callbacks cb 0x5649300eada0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40278/0 (socket says 172.21.15.69:40278) 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24bf7fe640 1 -- 172.21.15.69:0/2271210534 learned_addr learned my addr 172.21.15.69:0/2271210534 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.699 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x56493019d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.688+0000 7f24bffff640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193d30 0x56493019dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 -- 172.21.15.69:0/2271210534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193d30 msgr2=0x56493019dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193d30 0x56493019dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 -- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 msgr2=0x56493019d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x56493019d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 -- 172.21.15.69:0/2271210534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56492fe073f0 con 0x564930198ea0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f24b400a010 tx=0x7f24b4009f00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a37a03bc3130fd38 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b401b040 con 0x564930198ea0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f24b40136c0 con 0x564930198ea0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24b40139a0 con 0x564930198ea0 2024-05-09T09:18:51.700 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24b4033050 con 0x564930198ea0 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24c97b3740 10 client.15606 fetch_fsmap learned FSMap version 25 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56492fdc70f0 con 0x564930198ea0 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b40139a0 con 0x564930198ea0 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.689+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f24b4021070 con 0x564930198ea0 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 10 client.15606 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 10 client.15606 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 20 client.15606 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5649301a9300 0x5649301ab6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 4 client.15606 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564930101430 con 0x5649301a9300 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24bffff640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5649301a9300 0x5649301ab6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f249f7fe640 10 client.15606 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.701 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.690+0000 7f24bffff640 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5649301a9300 0x5649301ab6f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ee5edc257611b1 server_cookie=33aa21d57e7ec082 in_seq=0 out_seq=0 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f249f7fe640 1 -- 172.21.15.69:0/2271210534 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564930101430 con 0x5649301a9300 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f249f7fe640 10 client.15606 handle_command_reply: tid=0 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 client.15606 shutdown 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 msgr2=0x5649301a4d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7f24b400a010 tx=0x7f24b4009f00 comp rx=0 tx=0).stop 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 shutdown_connections 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930198ea0 0x5649301a4d00 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5649301a9300 0x5649301ab6f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930193d30 0x56493019dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f249f7fe640 0 client.15606 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.702 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 --2- 172.21.15.69:0/2271210534 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x56493019d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 >> 172.21.15.69:0/2271210534 conn(0x56493018a5a0 msgr2=0x5649301897e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 shutdown_connections 2024-05-09T09:18:51.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 1 -- 172.21.15.69:0/2271210534 wait complete. 2024-05-09T09:18:51.703 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.691+0000 7f24c97b3740 20 client.15606 trim_cache size 0 max 16384 2024-05-09T09:18:51.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.708+0000 7f24c97b3740 1 Processor -- start 2024-05-09T09:18:51.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.708+0000 7f24c97b3740 1 -- start start 2024-05-09T09:18:51.716 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x564930185670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x564930189660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01aac50 con 0x564930193d30 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01a8520 con 0x564930193340 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24c97b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01a6370 con 0x564930198ea0 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.717 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40290/0 (socket says 172.21.15.69:40290) 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 -- 172.21.15.69:0/3875376530 learned_addr learned my addr 172.21.15.69:0/3875376530 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x564930189660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x564930185670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 -- 172.21.15.69:0/3875376530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 msgr2=0x564930189660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x564930189660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 -- 172.21.15.69:0/3875376530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 msgr2=0x564930185670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x564930185670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 -- 172.21.15.69:0/3875376530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56493018a3d0 con 0x564930193d30 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x564930185670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24bffff640 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f24b00314e0 tx=0x7f24b0005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9451e1c8c63782c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.718 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24a5ffb640 1 -- 172.21.15.69:0/3875376530 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0035040 con 0x564930193d30 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24a5ffb640 1 -- 172.21.15.69:0/3875376530 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0045bc0 con 0x564930193d30 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.709+0000 7f24a5ffb640 1 -- 172.21.15.69:0/3875376530 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0005040 con 0x564930193d30 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- 172.21.15.69:0/3875376530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 msgr2=0x5649301a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7f24b00314e0 tx=0x7f24b0005fd0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- 172.21.15.69:0/3875376530 shutdown_connections 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x564930189660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a4cc0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3875376530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x564930185670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- 172.21.15.69:0/3875376530 >> 172.21.15.69:0/3875376530 conn(0x56493018a5a0 msgr2=0x564930188f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- 172.21.15.69:0/3875376530 shutdown_connections 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- 172.21.15.69:0/3875376530 wait complete. 2024-05-09T09:18:51.719 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 -- start start 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.710+0000 7f24c97b3740 1 Processor -- start 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x5649301a22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x5649301a3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01a69e0 con 0x564930193d30 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01a7e50 con 0x564930193340 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f24c01a70b0 con 0x564930198ea0 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 10 client.0 _ll_register_callbacks cb 0x7f24b4019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40296/0 (socket says 172.21.15.69:40296) 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 -- 172.21.15.69:0/3767811727 learned_addr learned my addr 172.21.15.69:0/3767811727 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:18:51.720 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x5649301a3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x5649301a22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 -- 172.21.15.69:0/3767811727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 msgr2=0x5649301a3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x5649301a3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 -- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 msgr2=0x5649301a22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x5649301a22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 -- 172.21.15.69:0/3767811727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5649301a3e00 con 0x564930193d30 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x5649301a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24bffff640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f24b0031440 tx=0x7f24b0032ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e385677b8887d7b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f249f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f24b0051020 con 0x564930193d30 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f24b0032250 con 0x564930193d30 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f24b0045bc0 con 0x564930193d30 2024-05-09T09:18:51.721 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f24b005c050 con 0x564930193d30 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 10 client.15612 fetch_fsmap learned FSMap version 25 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.711+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56493020f740 con 0x564930193d30 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f24b0045bc0 con 0x564930193d30 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f24b0004920 con 0x564930193d30 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 10 client.15612 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 10 client.15612 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 20 client.15612 populate_metadata read hostname 'smithi069' 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564930210050 0x564930212470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 4 client.15612 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:18:51.722 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x5649300fb780 con 0x564930210050 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564930210050 0x564930212470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f249f7fe640 10 client.15612 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.712+0000 7f24bf7fe640 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564930210050 0x564930212470 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=18cfdff50bb176ec server_cookie=e152092d1c6c6744 in_seq=0 out_seq=0 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f249f7fe640 1 -- 172.21.15.69:0/3767811727 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x5649300fb780 con 0x564930210050 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f249f7fe640 10 client.15612 handle_command_reply: tid=0 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 client.15612 shutdown 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 msgr2=0x5649301a2800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 crypto rx=0x7f24b0031440 tx=0x7f24b0032ad0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 shutdown_connections 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564930210050 0x564930212470 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564930198ea0 0x5649301a3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564930193d30 0x5649301a2800 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.723 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 --2- 172.21.15.69:0/3767811727 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564930193340 0x5649301a22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 >> 172.21.15.69:0/3767811727 conn(0x56493018a5a0 msgr2=0x56493018db10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f249f7fe640 0 client.15612 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 shutdown_connections 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 1 -- 172.21.15.69:0/3767811727 wait complete. 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c97b3740 20 client.15612 trim_cache size 0 max 16384 2024-05-09T09:18:51.724 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "9e61dd6f-327b-4bda-a963-bba023a27344": { 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "9e61dd6f-327b-4bda-a963-bba023a27344", 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,force" 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:18:51.725 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 msgr2=0x7f24a80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 0x7f24a80a5050 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f24b8097900 tx=0x7f24b80bb040 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 msgr2=0x7f24c019f220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.713+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f24b800b970 tx=0x7f24b800be30 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 shutdown_connections 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f24a80a2b90 0x7f24a80a5050 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f24c0103430 0x7f24c019f220 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f24c0102a70 0x7f24c01981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 --2- 172.21.15.69:0/1130388227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f24c0101880 0x7f24c0197c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:18:51.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 >> 172.21.15.69:0/1130388227 conn(0x7f24c00fce10 msgr2=0x7f24c0105dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:18:51.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 shutdown_connections 2024-05-09T09:18:51.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:18:51.714+0000 7f24c69c8640 1 -- 172.21.15.69:0/1130388227 wait complete. 2024-05-09T09:18:51.728 INFO:tasks.cephfs.filesystem:scrub status for tag:9e61dd6f-327b-4bda-a963-bba023a27344 - {'path': '/', 'tag': '9e61dd6f-327b-4bda-a963-bba023a27344', 'options': 'recursive,force'} 2024-05-09T09:18:52.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3846671297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:52.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1530968294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:51 smithi069 ceph-mon[30123]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:52.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1229222662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3846671297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1530968294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:51 smithi163 ceph-mon[34354]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1229222662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3846671297' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:18:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1530968294' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:51 smithi103 ceph-mon[34421]: pgmap v364: 129 pgs: 129 active+clean; 644 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 426 B/s wr, 5 op/s 2024-05-09T09:18:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1229222662' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:18:52.944 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:52 smithi069 ceph-mon[30123]: scrub queued for path: / 2024-05-09T09:18:52.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:52 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:52.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:52 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:18:52.945 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:52 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:52 smithi163 ceph-mon[34354]: scrub queued for path: / 2024-05-09T09:18:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:52 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:52 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:18:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:52 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:52 smithi103 ceph-mon[34421]: scrub queued for path: / 2024-05-09T09:18:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:52 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:52 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:18:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:52 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:18:54.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:53 smithi069 ceph-mon[30123]: pgmap v365: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-05-09T09:18:54.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:53 smithi163 ceph-mon[34354]: pgmap v365: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-05-09T09:18:54.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:53 smithi103 ceph-mon[34421]: pgmap v365: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.5 KiB/s wr, 8 op/s 2024-05-09T09:18:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:56 smithi069 ceph-mon[30123]: pgmap v366: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-05-09T09:18:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:56 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:18:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:56 smithi163 ceph-mon[34354]: pgmap v366: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-05-09T09:18:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:56 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:18:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:56 smithi103 ceph-mon[34421]: pgmap v366: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.6 KiB/s wr, 6 op/s 2024-05-09T09:18:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:56 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: pgmap v367: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:58 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: pgmap v367: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:58 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: pgmap v367: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.7 KiB/s wr, 8 op/s 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:58 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:59.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:18:59.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:18:59.513 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:18:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:18:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:18:59.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:18:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:18:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:18:59.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:18:59.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:18:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:00 smithi069 ceph-mon[30123]: pgmap v368: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:00 smithi163 ceph-mon[34354]: pgmap v368: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:00 smithi103 ceph-mon[34421]: pgmap v368: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:02 smithi069 ceph-mon[30123]: pgmap v369: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:02 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:19:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:02 smithi163 ceph-mon[34354]: pgmap v369: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:02 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:19:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:02 smithi103 ceph-mon[34421]: pgmap v369: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.3 KiB/s wr, 6 op/s 2024-05-09T09:19:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:02 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:19:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:04 smithi069 ceph-mon[30123]: pgmap v370: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-09T09:19:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 -- 172.21.15.69:0/3260956956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe680ffba0 msgr2=0x7fbe680fffc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 --2- 172.21.15.69:0/3260956956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe680ffba0 0x7fbe680fffc0 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 crypto rx=0x7fbe58009930 tx=0x7fbe5802a420 comp rx=0 tx=0).stop 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 -- 172.21.15.69:0/3260956956 shutdown_connections 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 --2- 172.21.15.69:0/3260956956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe681035d0 0x7fbe68109e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 --2- 172.21.15.69:0/3260956956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe68102c10 0x7fbe68103090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 --2- 172.21.15.69:0/3260956956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe680ffba0 0x7fbe680fffc0 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 -- 172.21.15.69:0/3260956956 >> 172.21.15.69:0/3260956956 conn(0x7fbe680fb4b0 msgr2=0x7fbe680fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 -- 172.21.15.69:0/3260956956 shutdown_connections 2024-05-09T09:19:04.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.520+0000 7fbe6d177640 1 -- 172.21.15.69:0/3260956956 wait complete. 2024-05-09T09:19:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 Processor -- start 2024-05-09T09:19:04.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 -- start start 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 0x7fbe68197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe681035d0 0x7fbe6819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810d530 con 0x7fbe68102c10 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810d3b0 con 0x7fbe681035d0 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.521+0000 7fbe6d177640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe6810d6b0 con 0x7fbe680ffba0 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42806/0 (socket says 172.21.15.69:42806) 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 -- 172.21.15.69:0/2294446615 learned_addr learned my addr 172.21.15.69:0/2294446615 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe681035d0 0x7fbe6819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 0x7fbe68197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 -- 172.21.15.69:0/2294446615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 msgr2=0x7fbe68197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 0x7fbe68197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 -- 172.21.15.69:0/2294446615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe681035d0 msgr2=0x7fbe6819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe681035d0 0x7fbe6819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 -- 172.21.15.69:0/2294446615 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbe6819f800 con 0x7fbe68102c10 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 0x7fbe68197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6659c640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00d950 tx=0x7fbe5c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe4ffff640 1 -- 172.21.15.69:0/2294446615 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c017070 con 0x7fbe68102c10 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbe6819faf0 con 0x7fbe68102c10 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe4ffff640 1 -- 172.21.15.69:0/2294446615 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe5c009c30 con 0x7fbe68102c10 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe4ffff640 1 -- 172.21.15.69:0/2294446615 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe5c00f6b0 con 0x7fbe68102c10 2024-05-09T09:19:04.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.522+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbe681a0090 con 0x7fbe68102c10 2024-05-09T09:19:04.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe4ffff640 1 -- 172.21.15.69:0/2294446615 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbe5c0109d0 con 0x7fbe68102c10 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 Processor -- start 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 -- start start 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408630b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 0x55b408633650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b40858b340 con 0x55b40862b660 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b408533290 con 0x55b40862c050 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b408587300 con 0x55b4086311c0 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42810/0 (socket says 172.21.15.69:42810) 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe6759e640 1 -- 172.21.15.69:0/2958872408 learned_addr learned my addr 172.21.15.69:0/2958872408 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:04.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.525+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408630b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6659c640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 0x55b408633650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 -- 172.21.15.69:0/2958872408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 msgr2=0x55b408633650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 0x55b408633650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 -- 172.21.15.69:0/2958872408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 msgr2=0x55b408630b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe4ffff640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 0x7fbe500a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408630b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 -- 172.21.15.69:0/2958872408 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b4085578e0 con 0x55b40862b660 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6659c640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 0x55b408633650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe4ffff640 1 -- 172.21.15.69:0/2294446615 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbe5c0cefd0 con 0x7fbe68102c10 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbe60010370 tx=0x7fbe60010830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=54c4c3c7b1114d5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 0x7fbe500a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe4d7fa640 1 -- 172.21.15.69:0/2958872408 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60016070 con 0x55b40862b660 2024-05-09T09:19:04.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe4d7fa640 1 -- 172.21.15.69:0/2958872408 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6001b460 con 0x55b40862b660 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.526+0000 7fbe4d7fa640 1 -- 172.21.15.69:0/2958872408 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe600117e0 con 0x55b40862b660 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- 172.21.15.69:0/2958872408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 msgr2=0x55b40862ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6759e640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 0x7fbe500a50f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbe6001a000 tx=0x7fbe60011eb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 crypto rx=0x7fbe60010370 tx=0x7fbe60010830 comp rx=0 tx=0).stop 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- 172.21.15.69:0/2958872408 shutdown_connections 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b4086311c0 0x55b408633650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408630b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/2958872408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b40862b660 0x55b40862ba80 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- 172.21.15.69:0/2958872408 >> 172.21.15.69:0/2958872408 conn(0x55b4086228c0 msgr2=0x55b408622cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- 172.21.15.69:0/2958872408 shutdown_connections 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- 172.21.15.69:0/2958872408 wait complete. 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 -- start start 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.527+0000 7fbe6ff64740 1 Processor -- start 2024-05-09T09:19:04.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b408634020 con 0x55b4086311c0 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b408546790 con 0x55b40862c050 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b4086341a0 con 0x55b40862b660 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 10 client.0 _ll_register_callbacks cb 0x55b408581a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42824/0 (socket says 172.21.15.69:42824) 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 -- 172.21.15.69:0/3965873970 learned_addr learned my addr 172.21.15.69:0/3965873970 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6659c640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 -- 172.21.15.69:0/3965873970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 msgr2=0x55b4086357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 -- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 msgr2=0x55b408635cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 -- 172.21.15.69:0/3965873970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b40829f210 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbe6001bcb0 tx=0x7fbe6001b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1918159814036154 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60035060 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbe6001b820 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.528+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe6001b460 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe6003c050 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 10 client.15624 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b40836b620 con 0x55b4086311c0 2024-05-09T09:19:04.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe6001b460 con 0x55b4086311c0 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbe6000f040 con 0x55b4086311c0 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 10 client.15624 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 10 client.15624 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 20 client.15624 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b408640f80 0x55b408643370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 4 client.15624 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b408598a70 con 0x55b408640f80 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.529+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b408640f80 0x55b408643370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b408640f80 0x55b408643370 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8608da32c6393f20 server_cookie=ca12143ca9982ec0 in_seq=0 out_seq=0 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe4aff5640 10 client.15624 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe4aff5640 1 -- 172.21.15.69:0/3965873970 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b408598a70 con 0x55b408640f80 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe4aff5640 10 client.15624 handle_command_reply: tid=0 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 client.15624 shutdown 2024-05-09T09:19:04.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 msgr2=0x55b40863cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 crypto rx=0x7fbe6001bcb0 tx=0x7fbe6001b000 comp rx=0 tx=0).stop 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 shutdown_connections 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863cd30 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.530+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b408640f80 0x55b408643370 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3965873970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 >> 172.21.15.69:0/3965873970 conn(0x55b4086228c0 msgr2=0x55b408621a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe4aff5640 0 client.15624 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 shutdown_connections 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3965873970 wait complete. 2024-05-09T09:19:04.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.531+0000 7fbe6ff64740 20 client.15624 trim_cache size 0 max 16384 2024-05-09T09:19:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.545+0000 7fbe6ff64740 1 Processor -- start 2024-05-09T09:19:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.545+0000 7fbe6ff64740 1 -- start start 2024-05-09T09:19:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.545+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.545+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.545+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681aabc0 con 0x55b4086311c0 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681a8490 con 0x55b40862c050 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681a62e0 con 0x55b40862b660 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42834/0 (socket says 172.21.15.69:42834) 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 -- 172.21.15.69:0/3478772273 learned_addr learned my addr 172.21.15.69:0/3478772273 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 -- 172.21.15.69:0/3478772273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 msgr2=0x55b4086357b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 -- 172.21.15.69:0/3478772273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 msgr2=0x55b408635cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 -- 172.21.15.69:0/3478772273 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b408620f00 con 0x55b4086311c0 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe6759e640 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbe600116b0 tx=0x7fbe60009710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d24658dbc7d1abc6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe4cff9640 1 -- 172.21.15.69:0/3478772273 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60045070 con 0x55b4086311c0 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe4cff9640 1 -- 172.21.15.69:0/3478772273 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe60010d30 con 0x55b4086311c0 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.546+0000 7fbe4cff9640 1 -- 172.21.15.69:0/3478772273 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60035070 con 0x55b4086311c0 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3478772273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 msgr2=0x55b40863ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fbe600116b0 tx=0x7fbe60009710 comp rx=0 tx=0).stop 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3478772273 shutdown_connections 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b40863ccf0 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408635cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/3478772273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086357b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3478772273 >> 172.21.15.69:0/3478772273 conn(0x55b4086228c0 msgr2=0x55b408622ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3478772273 shutdown_connections 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- 172.21.15.69:0/3478772273 wait complete. 2024-05-09T09:19:04.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 -- start start 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.547+0000 7fbe6ff64740 1 Processor -- start 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086398a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408639de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681a6950 con 0x55b4086311c0 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681a7dc0 con 0x55b40862c050 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbe681a7020 con 0x55b40862b660 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 10 client.0 _ll_register_callbacks cb 0x7fbe60009e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42836/0 (socket says 172.21.15.69:42836) 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 -- 172.21.15.69:0/1920013242 learned_addr learned my addr 172.21.15.69:0/1920013242 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6659c640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408639de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086398a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 -- 172.21.15.69:0/1920013242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 msgr2=0x55b4086398a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086398a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 -- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 msgr2=0x55b408639de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408639de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 -- 172.21.15.69:0/1920013242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b40863ceb0 con 0x55b4086311c0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6759e640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbe60000f80 tx=0x7fbe6001bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5afedc20bb31f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe4aff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbe60045070 con 0x55b4086311c0 2024-05-09T09:19:04.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbe60011270 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.548+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbe60010d30 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fbe60035070 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 10 client.15630 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b4086a6890 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbe60010d30 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fbe60003b10 con 0x55b4086311c0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 10 client.15630 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 10 client.15630 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 20 client.15630 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b4086a7170 0x55b4086a9590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 4 client.15630 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b408592dc0 con 0x55b4086a7170 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.549+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b4086a7170 0x55b4086a9590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:04.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.550+0000 7fbe4aff5640 10 client.15630 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:04.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.550+0000 7fbe66d9d640 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b4086a7170 0x55b4086a9590 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2359936d8e564db6 server_cookie=3bdab323b6e3f51a in_seq=0 out_seq=0 2024-05-09T09:19:04.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:04 smithi163 ceph-mon[34354]: pgmap v370: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe4aff5640 1 -- 172.21.15.69:0/1920013242 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b408592dc0 con 0x55b4086a7170 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe4aff5640 10 client.15630 handle_command_reply: tid=0 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 client.15630 shutdown 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 msgr2=0x55b4086a5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fbe60000f80 tx=0x7fbe6001bee0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 shutdown_connections 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b4086311c0 0x55b4086a5470 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b40862c050 0x55b408639de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b4086a7170 0x55b4086a9590 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 --2- 172.21.15.69:0/1920013242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b40862b660 0x55b4086398a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe4aff5640 0 client.15630 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.562+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 >> 172.21.15.69:0/1920013242 conn(0x55b4086228c0 msgr2=0x55b4086357b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 shutdown_connections 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6ff64740 1 -- 172.21.15.69:0/1920013242 wait complete. 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6ff64740 20 client.15630 trim_cache size 0 max 16384 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:19:04.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:19:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 msgr2=0x7fbe500a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 0x7fbe500a50f0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fbe6001a000 tx=0x7fbe60011eb0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 msgr2=0x7fbe68198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 crypto rx=0x7fbe5c00d950 tx=0x7fbe5c00de10 comp rx=0 tx=0).stop 2024-05-09T09:19:04.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 shutdown_connections 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbe500a2c30 0x7fbe500a50f0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbe681035d0 0x7fbe6819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbe68102c10 0x7fbe68198120 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 --2- 172.21.15.69:0/2294446615 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbe680ffba0 0x7fbe68197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 >> 172.21.15.69:0/2294446615 conn(0x7fbe680fb4b0 msgr2=0x7fbe68105eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 shutdown_connections 2024-05-09T09:19:04.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:04.563+0000 7fbe6d177640 1 -- 172.21.15.69:0/2294446615 wait complete. 2024-05-09T09:19:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:04 smithi103 ceph-mon[34421]: pgmap v370: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.3 KiB/s wr, 8 op/s 2024-05-09T09:19:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:06 smithi069 ceph-mon[30123]: pgmap v371: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:06 smithi163 ceph-mon[34354]: pgmap v371: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:06 smithi103 ceph-mon[34421]: pgmap v371: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:08 smithi069 ceph-mon[30123]: pgmap v372: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:08 smithi163 ceph-mon[34354]: pgmap v372: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:08 smithi103 ceph-mon[34421]: pgmap v372: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:10 smithi069 ceph-mon[30123]: pgmap v373: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:10 smithi163 ceph-mon[34354]: pgmap v373: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:10 smithi103 ceph-mon[34421]: pgmap v373: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 1.2 KiB/s wr, 10 op/s 2024-05-09T09:19:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:12 smithi069 ceph-mon[30123]: pgmap v374: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T09:19:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:12 smithi163 ceph-mon[34354]: pgmap v374: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T09:19:12.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:12 smithi103 ceph-mon[34421]: pgmap v374: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.5 KiB/s wr, 9 op/s 2024-05-09T09:19:14.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:14 smithi069 ceph-mon[30123]: pgmap v375: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:14 smithi163 ceph-mon[34354]: pgmap v375: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:14 smithi103 ceph-mon[34421]: pgmap v375: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:16 smithi069 ceph-mon[30123]: pgmap v376: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T09:19:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:16 smithi163 ceph-mon[34354]: pgmap v376: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T09:19:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:16 smithi103 ceph-mon[34421]: pgmap v376: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 -- 172.21.15.69:0/110298661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 msgr2=0x7f2348104f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 --2- 172.21.15.69:0/110298661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f2348104f80 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7f2338009900 tx=0x7f233802a410 comp rx=0 tx=0).stop 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 -- 172.21.15.69:0/110298661 shutdown_connections 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 --2- 172.21.15.69:0/110298661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f2348105940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 --2- 172.21.15.69:0/110298661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f2348104f80 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 --2- 172.21.15.69:0/110298661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 0x7f23481025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 -- 172.21.15.69:0/110298661 >> 172.21.15.69:0/110298661 conn(0x7f23480fbda0 msgr2=0x7f23480fe1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 -- 172.21.15.69:0/110298661 shutdown_connections 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.661+0000 7f234fea5640 1 -- 172.21.15.69:0/110298661 wait complete. 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 Processor -- start 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 -- start start 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 0x7f234819e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f23481a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2348113b40 con 0x7f2348102b10 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23481139c0 con 0x7f23481054c0 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234fea5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2348113cc0 con 0x7f23481001a0 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55770/0 (socket says 172.21.15.69:55770) 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.662+0000 7f234d440640 1 -- 172.21.15.69:0/1656515431 learned_addr learned my addr 172.21.15.69:0/1656515431 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:17.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234dc41640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 0x7f234819e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234e442640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f23481a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 -- 172.21.15.69:0/1656515431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 msgr2=0x7f234819e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 0x7f234819e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 -- 172.21.15.69:0/1656515431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 msgr2=0x7f23481a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f23481a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 -- 172.21.15.69:0/1656515431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f23481a5cc0 con 0x7f2348102b10 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234e442640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f23481a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234d440640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2338009230 tx=0x7f2338004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f233effd640 1 -- 172.21.15.69:0/1656515431 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233803f030 con 0x7f2348102b10 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f233effd640 1 -- 172.21.15.69:0/1656515431 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2338003ce0 con 0x7f2348102b10 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f233effd640 1 -- 172.21.15.69:0/1656515431 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338035070 con 0x7f2348102b10 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f23481a5f50 con 0x7f2348102b10 2024-05-09T09:19:17.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.663+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f23481a6360 con 0x7f2348102b10 2024-05-09T09:19:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f233effd640 1 -- 172.21.15.69:0/1656515431 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2338042030 con 0x7f2348102b10 2024-05-09T09:19:17.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 Processor -- start 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 -- start start 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee49555670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee494b9340 con 0x55ee4955adb0 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee49461290 con 0x55ee49555250 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f2352bb2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ee494b5300 con 0x55ee49555c40 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55774/0 (socket says 172.21.15.69:55774) 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.666+0000 7f234d440640 1 -- 172.21.15.69:0/936308563 learned_addr learned my addr 172.21.15.69:0/936308563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234e442640 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee49555670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234dc41640 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f233effd640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 0x7f23340a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 -- 172.21.15.69:0/936308563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 msgr2=0x55ee4955a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 -- 172.21.15.69:0/936308563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 msgr2=0x55ee49555670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee49555670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 -- 172.21.15.69:0/936308563 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee494858e0 con 0x55ee4955adb0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234e442640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 0x7f23340a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f233effd640 1 -- 172.21.15.69:0/1656515431 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f23380efe60 con 0x7f2348102b10 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234d440640 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f233809a290 tx=0x7f2338086f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1b9275e56aaae2c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f2333fff640 1 -- 172.21.15.69:0/936308563 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23380bbca0 con 0x55ee4955adb0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f234e442640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 0x7f23340a50f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55ee495564a0 tx=0x7f2340004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f2333fff640 1 -- 172.21.15.69:0/936308563 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f233809bab0 con 0x55ee4955adb0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.667+0000 7f2333fff640 1 -- 172.21.15.69:0/936308563 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f233809b1b0 con 0x55ee4955adb0 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 -- 172.21.15.69:0/936308563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 msgr2=0x55ee4955d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7f233809a290 tx=0x7f2338086f10 comp rx=0 tx=0).stop 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 -- 172.21.15.69:0/936308563 shutdown_connections 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee4955adb0 0x55ee4955d240 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 --2- 172.21.15.69:0/936308563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee49555670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 -- 172.21.15.69:0/936308563 >> 172.21.15.69:0/936308563 conn(0x55ee495508c0 msgr2=0x55ee49550cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 -- 172.21.15.69:0/936308563 shutdown_connections 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.668+0000 7f2352bb2740 1 -- 172.21.15.69:0/936308563 wait complete. 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 -- start start 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 Processor -- start 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee495666b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee4955db20 con 0x55ee49555250 2024-05-09T09:19:17.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee49474790 con 0x55ee4955adb0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ee4955dca0 con 0x55ee49555c40 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f2352bb2740 10 client.0 _ll_register_callbacks cb 0x55ee494afa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f234dc41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f234dc41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55778/0 (socket says 172.21.15.69:55778) 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.669+0000 7f234dc41640 1 -- 172.21.15.69:0/3573366072 learned_addr learned my addr 172.21.15.69:0/3573366072 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 -- 172.21.15.69:0/3573366072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 msgr2=0x55ee4955f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234e442640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee495666b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 -- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 msgr2=0x55ee495666b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee495666b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 -- 172.21.15.69:0/3573366072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee491cd210 con 0x55ee49555250 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f234dc41640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f234400be80 tx=0x7f2344012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eee51ca94be8ccae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2344005220 con 0x55ee49555250 2024-05-09T09:19:17.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f23440053c0 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ee494c6a70 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f23440055c0 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 10 client.15642 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee49299620 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ee494c6a70 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f2344019070 con 0x55ee49555250 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 10 client.15642 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 10 client.15642 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.670+0000 7f2352bb2740 20 client.15642 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee4956acf0 0x55ee4956d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f2352bb2740 4 client.15642 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee494c0dc0 con 0x55ee4956acf0 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f234d440640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee4956acf0 0x55ee4956d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f23317fa640 10 client.15642 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.671+0000 7f234d440640 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee4956acf0 0x55ee4956d180 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=629be0ffc53375bf server_cookie=306f86f3124c1039 in_seq=0 out_seq=0 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f23317fa640 1 -- 172.21.15.69:0/3573366072 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee494c0dc0 con 0x55ee4956acf0 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f23317fa640 10 client.15642 handle_command_reply: tid=0 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 client.15642 shutdown 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 msgr2=0x55ee4955f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7f234400be80 tx=0x7f2344012820 comp rx=0 tx=0).stop 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 shutdown_connections 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee495666b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee4956acf0 0x55ee4956d180 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555c40 0x55ee4955f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3573366072 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555250 0x55ee4955f130 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f23317fa640 0 client.15642 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 >> 172.21.15.69:0/3573366072 conn(0x55ee495508c0 msgr2=0x55ee4954fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 shutdown_connections 2024-05-09T09:19:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 1 -- 172.21.15.69:0/3573366072 wait complete. 2024-05-09T09:19:17.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.672+0000 7f2352bb2740 20 client.15642 trim_cache size 0 max 16384 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 Processor -- start 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 -- start start 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee4954fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 0x55ee49572a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23481b31b0 con 0x55ee49555c40 2024-05-09T09:19:17.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23481b0a80 con 0x55ee49555250 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.687+0000 7f2352bb2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f23481ae8d0 con 0x55ee4955adb0 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55794/0 (socket says 172.21.15.69:55794) 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 -- 172.21.15.69:0/1811430085 learned_addr learned my addr 172.21.15.69:0/1811430085 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234dc41640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee4954fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234e442640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 0x55ee49572a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 -- 172.21.15.69:0/1811430085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 msgr2=0x55ee49572a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 0x55ee49572a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 -- 172.21.15.69:0/1811430085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 msgr2=0x55ee4954fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee4954fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 -- 172.21.15.69:0/1811430085 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee49556020 con 0x55ee49555c40 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234e442640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 0x55ee49572a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f234d440640 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23380e8860 tx=0x7f23380f3940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b06de9fd3e99e38 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f23337fe640 1 -- 172.21.15.69:0/1811430085 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338049040 con 0x55ee49555c40 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f23337fe640 1 -- 172.21.15.69:0/1811430085 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23380bca40 con 0x55ee49555c40 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.688+0000 7f23337fe640 1 -- 172.21.15.69:0/1811430085 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f23380c4fb0 con 0x55ee49555c40 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- 172.21.15.69:0/1811430085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 msgr2=0x55ee49566670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f23380e8860 tx=0x7f23380f3940 comp rx=0 tx=0).stop 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- 172.21.15.69:0/1811430085 shutdown_connections 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee4955adb0 0x55ee49572a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49566670 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- 172.21.15.69:0/1811430085 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee49555250 0x55ee4954fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- 172.21.15.69:0/1811430085 >> 172.21.15.69:0/1811430085 conn(0x55ee495508c0 msgr2=0x55ee49553fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- 172.21.15.69:0/1811430085 shutdown_connections 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- 172.21.15.69:0/1811430085 wait complete. 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- start start 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 Processor -- start 2024-05-09T09:19:17.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555250 0x55ee49563a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee49564f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f23481aef40 con 0x55ee49555c40 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f23481b03b0 con 0x55ee4955adb0 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f23481af610 con 0x55ee49555250 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.689+0000 7f2352bb2740 10 client.0 _ll_register_callbacks cb 0x7f234400b730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55800/0 (socket says 172.21.15.69:55800) 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 -- 172.21.15.69:0/3647593064 learned_addr learned my addr 172.21.15.69:0/3647593064 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234e442640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee49564f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234dc41640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555250 0x55ee49563a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 -- 172.21.15.69:0/3647593064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555250 msgr2=0x55ee49563a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555250 0x55ee49563a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 -- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 msgr2=0x55ee49564f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee49564f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 -- 172.21.15.69:0/3647593064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee49565560 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234e442640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee49564f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f234d440640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f23380c4a80 tx=0x7f23380bc340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5a15b5a926ca6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f23317fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2338049050 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f233803bdc0 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f23380bca40 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f233808f900 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f2352bb2740 10 client.15648 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55ee495667e0 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.690+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f23380bca40 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f23380b89b0 con 0x55ee49555c40 2024-05-09T09:19:17.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 10 client.15648 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 10 client.15648 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 20 client.15648 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee495d1730 0x55ee495d3b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 4 client.15648 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee4943e8e0 con 0x55ee495d1730 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f234dc41640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee495d1730 0x55ee495d3b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f23317fa640 10 client.15648 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:17.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.691+0000 7f234dc41640 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee495d1730 0x55ee495d3b20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41d4580d307164e1 server_cookie=197a1223d4920631 in_seq=0 out_seq=0 2024-05-09T09:19:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f23317fa640 1 -- 172.21.15.69:0/3647593064 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee4943e8e0 con 0x55ee495d1730 2024-05-09T09:19:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f23317fa640 10 client.15648 handle_command_reply: tid=0 2024-05-09T09:19:17.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 client.15648 shutdown 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 msgr2=0x55ee49563fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f23380c4a80 tx=0x7f23380bc340 comp rx=0 tx=0).stop 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 shutdown_connections 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee4955adb0 0x55ee49564f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee49555c40 0x55ee49563fb0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee495d1730 0x55ee495d3b20 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.695+0000 7f2352bb2740 1 --2- 172.21.15.69:0/3647593064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee49555250 0x55ee49563a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 >> 172.21.15.69:0/3647593064 conn(0x55ee495508c0 msgr2=0x55ee4954fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f23317fa640 0 client.15648 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 shutdown_connections 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f2352bb2740 1 -- 172.21.15.69:0/3647593064 wait complete. 2024-05-09T09:19:17.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f2352bb2740 20 client.15648 trim_cache size 0 max 16384 2024-05-09T09:19:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:19:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:19:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:19:17.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 msgr2=0x7f23340a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 0x7f23340a50f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x55ee495564a0 tx=0x7f2340004a40 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 msgr2=0x7f234819e630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7f2338009230 tx=0x7f2338004360 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 shutdown_connections 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f23340a2c30 0x7f23340a50f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f23481054c0 0x7f23481a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2348102b10 0x7f234819e630 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 --2- 172.21.15.69:0/1656515431 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f23481001a0 0x7f234819e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 >> 172.21.15.69:0/1656515431 conn(0x7f23480fbda0 msgr2=0x7f2348100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 shutdown_connections 2024-05-09T09:19:17.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:17.696+0000 7f234fea5640 1 -- 172.21.15.69:0/1656515431 wait complete. 2024-05-09T09:19:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:18 smithi069 ceph-mon[30123]: pgmap v377: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:18 smithi163 ceph-mon[34354]: pgmap v377: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:18 smithi103 ceph-mon[34421]: pgmap v377: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.5 KiB/s wr, 11 op/s 2024-05-09T09:19:20.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:20 smithi069 ceph-mon[30123]: pgmap v378: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:20 smithi163 ceph-mon[34354]: pgmap v378: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:20 smithi103 ceph-mon[34421]: pgmap v378: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:21.729 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:19:21.729 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.838+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/1600024746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48102320 msgr2=0x7f3c48108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.838+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/1600024746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48102320 0x7f3c48108bb0 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f3c40009ca0 tx=0x7f3c4002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.838+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/1600024746 shutdown_connections 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/1600024746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48102320 0x7f3c48108bb0 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/1600024746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c48101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/1600024746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48100770 0x7f3c48100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/1600024746 >> 172.21.15.69:0/1600024746 conn(0x7f3c480fbce0 msgr2=0x7f3c480fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/1600024746 shutdown_connections 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/1600024746 wait complete. 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 Processor -- start 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- start start 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c4819c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48102320 0x7f3c4819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810c3d0 con 0x7f3c48100770 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810c250 con 0x7f3c48101960 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ecb3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c4810c550 con 0x7f3c48102320 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ca4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ca4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55822/0 (socket says 172.21.15.69:55822) 2024-05-09T09:19:21.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.839+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/3075452020 learned_addr learned my addr 172.21.15.69:0/3075452020 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c4819c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/3075452020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48102320 msgr2=0x7f3c4819ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4d250640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48102320 0x7f3c4819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48102320 0x7f3c4819ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/3075452020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 msgr2=0x7f3c4819c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c4819c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/3075452020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c4819f710 con 0x7f3c48100770 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c4819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:21.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00c970 tx=0x7f3c3c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c45ffb640 1 -- 172.21.15.69:0/3075452020 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c007c10 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c4819fa00 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c45ffb640 1 -- 172.21.15.69:0/3075452020 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c007db0 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c45ffb640 1 -- 172.21.15.69:0/3075452020 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c00f8a0 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.840+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c4819ff50 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.842+0000 7f3c45ffb640 1 -- 172.21.15.69:0/3075452020 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3c3c00fa60 con 0x7f3c48100770 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 Processor -- start 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 -- start start 2024-05-09T09:19:21.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2edb760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c45ffb640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 0x7f3c380a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2e3bd00 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2de34c0 con 0x55b7c2edbd30 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c519c4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2e37cc0 con 0x55b7c2edb340 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 0x7f3c380a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c4d250640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c4d250640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55830/0 (socket says 172.21.15.69:55830) 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c45ffb640 1 -- 172.21.15.69:0/3075452020 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3c3c0cf7c0 con 0x7f3c48100770 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c4d250640 1 -- 172.21.15.69:0/1535968361 learned_addr learned my addr 172.21.15.69:0/1535968361 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.843+0000 7f3c4d250640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2edb760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/1535968361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 msgr2=0x55b7c2edb760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2edb760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/1535968361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 msgr2=0x55b7c2ee3330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/1535968361 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7c2c197d0 con 0x55b7c2edbd30 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4d250640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 0x7f3c380a5050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c480692c0 tx=0x7f3c34005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee0830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c098700 tx=0x7f3c3c09b750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=52070ebbd295d115 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c32ffd640 1 -- 172.21.15.69:0/1535968361 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a00e0 con 0x55b7c2edbd30 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c32ffd640 1 -- 172.21.15.69:0/1535968361 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c0a4910 con 0x55b7c2edbd30 2024-05-09T09:19:21.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.844+0000 7f3c32ffd640 1 -- 172.21.15.69:0/1535968361 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a4480 con 0x55b7c2edbd30 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- 172.21.15.69:0/1535968361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 msgr2=0x55b7c2ee0830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee0830 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c098700 tx=0x7f3c3c09b750 comp rx=0 tx=0).stop 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- 172.21.15.69:0/1535968361 shutdown_connections 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ee3330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee0830 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1535968361 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2edb760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- 172.21.15.69:0/1535968361 >> 172.21.15.69:0/1535968361 conn(0x55b7c2ed25a0 msgr2=0x55b7c2ed29a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- 172.21.15.69:0/1535968361 shutdown_connections 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- 172.21.15.69:0/1535968361 wait complete. 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 -- start start 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.845+0000 7f3c519c4740 1 Processor -- start 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2ee57c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.858 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2ee3e50 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2df6b90 con 0x55b7c2edbd30 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7c2ee3fd0 con 0x55b7c2edb340 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c519c4740 10 client.0 _ll_register_callbacks cb 0x55b7c2e32da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:21.859 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55838/0 (socket says 172.21.15.69:55838) 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 -- 172.21.15.69:0/3257207638 learned_addr learned my addr 172.21.15.69:0/3257207638 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2ee57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 -- 172.21.15.69:0/3257207638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 msgr2=0x55b7c2ee57c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.860 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2ee57c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 -- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 msgr2=0x55b7c2ee5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 -- 172.21.15.69:0/3257207638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7c2b4f3f0 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c47fff640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c4d250640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3c40031ee0 tx=0x7f3c40031f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fcbc968d8fa4bcd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.861 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.846+0000 7f3c27fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4002fa20 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3c4002fbc0 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c3c0a4910 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3c40052050 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 10 client.15660 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7c2b0f0f0 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.862 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c0a4910 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3c40049070 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 10 client.15660 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 10 client.15660 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 20 client.15660 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2ef1630 0x55b7c2ef3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 4 client.15660 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:21.863 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7c2e49430 con 0x55b7c2ef1630 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.847+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2ef1630 0x55b7c2ef3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.848+0000 7f3c27fff640 10 client.15660 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.848+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2ef1630 0x55b7c2ef3ab0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7b402c197a70e7 server_cookie=9801837436b704a4 in_seq=0 out_seq=0 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c27fff640 1 -- 172.21.15.69:0/3257207638 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7c2e49430 con 0x55b7c2ef1630 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c27fff640 10 client.15660 handle_command_reply: tid=0 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 client.15660 shutdown 2024-05-09T09:19:21.864 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 msgr2=0x55b7c2eecd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f3c40031ee0 tx=0x7f3c40031f10 comp rx=0 tx=0).stop 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 shutdown_connections 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eecd40 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edbd30 0x55b7c2ee5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2ef1630 0x55b7c2ef3ab0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 --2- 172.21.15.69:0/3257207638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edb340 0x55b7c2ee57c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 >> 172.21.15.69:0/3257207638 conn(0x55b7c2ed25a0 msgr2=0x55b7c2ed17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c27fff640 0 client.15660 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 shutdown_connections 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 1 -- 172.21.15.69:0/3257207638 wait complete. 2024-05-09T09:19:21.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.849+0000 7f3c519c4740 20 client.15660 trim_cache size 0 max 16384 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 Processor -- start 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 -- start start 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edbd30 0x55b7c2eecd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ed5560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481aaa20 con 0x55b7c2edb340 2024-05-09T09:19:21.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481a82f0 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c519c4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481a6140 con 0x55b7c2edbd30 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55846/0 (socket says 172.21.15.69:55846) 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/978081338 learned_addr learned my addr 172.21.15.69:0/978081338 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c47fff640 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edbd30 0x55b7c2eecd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/978081338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edbd30 msgr2=0x55b7c2eecd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edbd30 0x55b7c2eecd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/978081338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2ee0ea0 msgr2=0x55b7c2ed5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ed5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 -- 172.21.15.69:0/978081338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7c2ed23d0 con 0x55b7c2edb340 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.867+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0ccf90 tx=0x7f3c3c007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee6fae29126b1990 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c327fc640 1 -- 172.21.15.69:0/978081338 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a4910 con 0x55b7c2edb340 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c327fc640 1 -- 172.21.15.69:0/978081338 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c09a030 con 0x55b7c2edb340 2024-05-09T09:19:21.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c327fc640 1 -- 172.21.15.69:0/978081338 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c3c0a46e0 con 0x55b7c2edb340 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 -- 172.21.15.69:0/978081338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 msgr2=0x55b7c2ecd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c0ccf90 tx=0x7f3c3c007f60 comp rx=0 tx=0).stop 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 -- 172.21.15.69:0/978081338 shutdown_connections 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2ed5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2edbd30 0x55b7c2eecd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 --2- 172.21.15.69:0/978081338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edb340 0x55b7c2ecd670 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 -- 172.21.15.69:0/978081338 >> 172.21.15.69:0/978081338 conn(0x55b7c2ed25a0 msgr2=0x55b7c2ee57c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 -- 172.21.15.69:0/978081338 shutdown_connections 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.868+0000 7f3c519c4740 1 -- 172.21.15.69:0/978081338 wait complete. 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 -- start start 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 Processor -- start 2024-05-09T09:19:21.874 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edb340 0x55b7c2eea5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eebae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481a67b0 con 0x55b7c2edbd30 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481a7c20 con 0x55b7c2edb340 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c481a6e80 con 0x55b7c2ee0ea0 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c519c4740 10 client.0 _ll_register_callbacks cb 0x7f3c40009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c47fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c47fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55862/0 (socket says 172.21.15.69:55862) 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.869+0000 7f3c47fff640 1 -- 172.21.15.69:0/1523790001 learned_addr learned my addr 172.21.15.69:0/1523790001 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.875 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edb340 0x55b7c2eea5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 -- 172.21.15.69:0/1523790001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2ee0ea0 msgr2=0x55b7c2eebae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eebae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 -- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edb340 msgr2=0x55b7c2eea5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edb340 0x55b7c2eea5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 -- 172.21.15.69:0/1523790001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7c2eec0d0 con 0x55b7c2edbd30 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c47fff640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3c34011c60 tx=0x7f3c340046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae7906fddd92bf94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c34004220 con 0x55b7c2edbd30 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f3c340043c0 con 0x55b7c2edbd30 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3c3c09a030 con 0x55b7c2edbd30 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f3c34030050 con 0x55b7c2edbd30 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 10 client.15666 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:21.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55b7c2f57e10 con 0x55b7c2edbd30 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c3c09a030 con 0x55b7c2edbd30 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3c3400f040 con 0x55b7c2edbd30 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 10 client.15666 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 10 client.15666 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 20 client.15666 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2f588c0 0x55b7c2f5ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 4 client.15666 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.870+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55b7c2e43780 con 0x55b7c2f588c0 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2f588c0 0x55b7c2f5ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c4ca4f640 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2f588c0 0x55b7c2f5ace0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cd194badbb0d0b85 server_cookie=cb4052ba2af078bd in_seq=0 out_seq=0 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c27fff640 10 client.15666 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c27fff640 1 -- 172.21.15.69:0/1523790001 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55b7c2e43780 con 0x55b7c2f588c0 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c27fff640 10 client.15666 handle_command_reply: tid=0 2024-05-09T09:19:21.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 client.15666 shutdown 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 msgr2=0x55b7c2eeab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3c34011c60 tx=0x7f3c340046c0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 shutdown_connections 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7c2f588c0 0x55b7c2f5ace0 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7c2ee0ea0 0x55b7c2eebae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c27fff640 0 client.15666 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7c2edbd30 0x55b7c2eeab20 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 --2- 172.21.15.69:0/1523790001 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7c2edb340 0x55b7c2eea5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.871+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 >> 172.21.15.69:0/1523790001 conn(0x55b7c2ed25a0 msgr2=0x55b7c2ed0f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 shutdown_connections 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c519c4740 1 -- 172.21.15.69:0/1523790001 wait complete. 2024-05-09T09:19:21.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c519c4740 20 client.15666 trim_cache size 0 max 16384 2024-05-09T09:19:21.879 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:21.879 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-05-09T09:19:21.879 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-05-09T09:19:21.879 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:21.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 msgr2=0x7f3c380a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 0x7f3c380a5050 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c480692c0 tx=0x7f3c34005e50 comp rx=0 tx=0).stop 2024-05-09T09:19:21.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 msgr2=0x7f3c48069550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f3c3c00c970 tx=0x7f3c3c00ce30 comp rx=0 tx=0).stop 2024-05-09T09:19:21.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 shutdown_connections 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c380a2b90 0x7f3c380a5050 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c48102320 0x7f3c4819ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c48101960 0x7f3c4819c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 --2- 172.21.15.69:0/3075452020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c48100770 0x7f3c48069550 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 >> 172.21.15.69:0/3075452020 conn(0x7f3c480fbce0 msgr2=0x7f3c48104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 shutdown_connections 2024-05-09T09:19:21.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.872+0000 7f3c4ecb3640 1 -- 172.21.15.69:0/3075452020 wait complete. 2024-05-09T09:19:21.882 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:19:21.883 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- 172.21.15.69:0/989949835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98101960 msgr2=0x7fbb98101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/989949835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98101960 0x7fbb98101de0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7fbb88009900 tx=0x7fbb8802a410 comp rx=0 tx=0).stop 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- 172.21.15.69:0/989949835 shutdown_connections 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/989949835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 0x7fbb98108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/989949835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98101960 0x7fbb98101de0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/989949835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98100770 0x7fbb98100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- 172.21.15.69:0/989949835 >> 172.21.15.69:0/989949835 conn(0x7fbb980fbce0 msgr2=0x7fbb980fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- 172.21.15.69:0/989949835 shutdown_connections 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- 172.21.15.69:0/989949835 wait complete. 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 Processor -- start 2024-05-09T09:19:21.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.988+0000 7fbb9fa88640 1 -- start start 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98101960 0x7fbb9819c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 0x7fbb9819eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810c3d0 con 0x7fbb98100770 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810c250 con 0x7fbb98102320 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbb9810c550 con 0x7fbb98101960 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55880/0 (socket says 172.21.15.69:55880) 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 -- 172.21.15.69:0/1547125473 learned_addr learned my addr 172.21.15.69:0/1547125473 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d023640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98101960 0x7fbb9819c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9e025640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 0x7fbb9819eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 -- 172.21.15.69:0/1547125473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98101960 msgr2=0x7fbb9819c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98101960 0x7fbb9819c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 -- 172.21.15.69:0/1547125473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 msgr2=0x7fbb9819eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 0x7fbb9819eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 -- 172.21.15.69:0/1547125473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbb9819f800 con 0x7fbb98100770 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9d824640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400d770 tx=0x7fbb9400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94017070 con 0x7fbb98100770 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbb94009c30 con 0x7fbb98100770 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbb9819fa90 con 0x7fbb98100770 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbb94003dd0 con 0x7fbb98100770 2024-05-09T09:19:21.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.989+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbb9819ffb0 con 0x7fbb98100770 2024-05-09T09:19:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.990+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fbb98107a90 con 0x7fbb98100770 2024-05-09T09:19:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.991+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbb94003f70 con 0x7fbb98100770 2024-05-09T09:19:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.992+0000 7fbb8effd640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 0x7fbb840a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.992+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fbb940cf190 con 0x7fbb98100770 2024-05-09T09:19:21.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.992+0000 7fbb9d023640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 0x7fbb840a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:21.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.993+0000 7fbb9d023640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 0x7fbb840a5050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbb880314e0 tx=0x7fbb88000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:21.994 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:21.996+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fbb94009de0 con 0x7fbb98100770 2024-05-09T09:19:22.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.225+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fbb9802d380 con 0x7fbb98100770 2024-05-09T09:19:22.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.226+0000 7fbb8effd640 1 -- 172.21.15.69:0/1547125473 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11879 (secure 0 0 0) 0x7fbb9409bd70 con 0x7fbb98100770 2024-05-09T09:19:22.225 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:19:22.225 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:18:44.567983+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24491,24511]},"id":1}]} 2024-05-09T09:19:22.226 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 msgr2=0x7fbb840a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 0x7fbb840a5050 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fbb880314e0 tx=0x7fbb88000bc0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 msgr2=0x7fbb98069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7fbb9400d770 tx=0x7fbb9400dc30 comp rx=0 tx=0).stop 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 shutdown_connections 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbb840a2b90 0x7fbb840a5050 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbb98102320 0x7fbb9819eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.227 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbb98101960 0x7fbb9819c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 --2- 172.21.15.69:0/1547125473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbb98100770 0x7fbb98069530 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 >> 172.21.15.69:0/1547125473 conn(0x7fbb980fbce0 msgr2=0x7fbb98104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 shutdown_connections 2024-05-09T09:19:22.228 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.228+0000 7fbb9fa88640 1 -- 172.21.15.69:0/1547125473 wait complete. 2024-05-09T09:19:22.239 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:19:22.348 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 -- 172.21.15.69:0/186399873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8102330 msgr2=0x7f40f8108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 --2- 172.21.15.69:0/186399873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8102330 0x7f40f8108bc0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009ca0 tx=0x7f40f002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 -- 172.21.15.69:0/186399873 shutdown_connections 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 --2- 172.21.15.69:0/186399873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8102330 0x7f40f8108bc0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 --2- 172.21.15.69:0/186399873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8101970 0x7f40f8101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 --2- 172.21.15.69:0/186399873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8100780 0x7f40f8100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 -- 172.21.15.69:0/186399873 >> 172.21.15.69:0/186399873 conn(0x7f40f80fbcf0 msgr2=0x7f40f80fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 -- 172.21.15.69:0/186399873 shutdown_connections 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.347+0000 7f40ffff9640 1 -- 172.21.15.69:0/186399873 wait complete. 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 Processor -- start 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 -- start start 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 0x7f40f81969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.349 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8102330 0x7f40f819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c3e0 con 0x7f40f8101970 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c260 con 0x7f40f8100780 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40ffff9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40f810c560 con 0x7f40f8102330 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55892/0 (socket says 172.21.15.69:55892) 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 -- 172.21.15.69:0/3289407180 learned_addr learned my addr 172.21.15.69:0/3289407180 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fe596640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8102330 0x7f40f819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 0x7f40f81969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 -- 172.21.15.69:0/3289407180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8102330 msgr2=0x7f40f819df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8102330 0x7f40f819df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 -- 172.21.15.69:0/3289407180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 msgr2=0x7f40f81969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.350 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 0x7f40f81969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 -- 172.21.15.69:0/3289407180 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f40f819e5e0 con 0x7f40f8101970 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 0x7f40f81969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.348+0000 7f40fd594640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9f0 tx=0x7f40e800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.349+0000 7f40eeffd640 1 -- 172.21.15.69:0/3289407180 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e800cd80 con 0x7f40f8101970 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.349+0000 7f40eeffd640 1 -- 172.21.15.69:0/3289407180 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e8010910 con 0x7f40f8101970 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.349+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f40f819e8d0 con 0x7f40f8101970 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.349+0000 7f40eeffd640 1 -- 172.21.15.69:0/3289407180 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8018990 con 0x7f40f8101970 2024-05-09T09:19:22.351 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.349+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f40f819ee70 con 0x7f40f8101970 2024-05-09T09:19:22.358 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.351+0000 7f40eeffd640 1 -- 172.21.15.69:0/3289407180 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f40e8010ac0 con 0x7f40f8101970 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f40eeffd640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 0x7f40e40a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 0x7f40e40a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f40eeffd640 1 -- 172.21.15.69:0/3289407180 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f40e80cf6e0 con 0x7f40f8101970 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f4102de6740 1 Processor -- start 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 0x7f40e40a50d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f40f81017d0 tx=0x7f40f4005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f4102de6740 1 -- start start 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.352+0000 7f4102de6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5cc360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f4102de6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5d1430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f4102de6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.359 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f4102de6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56288c530c40 con 0x56288c5d1aa0 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f4102de6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56288c4d8580 con 0x56288c5cc930 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f4102de6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56288c52cc00 con 0x56288c5cbf40 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55906/0 (socket says 172.21.15.69:55906) 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 -- 172.21.15.69:0/3853325600 learned_addr learned my addr 172.21.15.69:0/3853325600 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5cc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fd594640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5d1430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 -- 172.21.15.69:0/3853325600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 msgr2=0x56288c5cc360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5cc360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 -- 172.21.15.69:0/3853325600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 msgr2=0x56288c5d1430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5d1430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 -- 172.21.15.69:0/3853325600 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56288c2443f0 con 0x56288c5d1aa0 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:22.360 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40fe596640 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009ca0 tx=0x7f40f0009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=109734422b7333bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40e3fff640 1 -- 172.21.15.69:0/3853325600 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0031d70 con 0x56288c5d1aa0 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40e3fff640 1 -- 172.21.15.69:0/3853325600 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f0036040 con 0x56288c5d1aa0 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.353+0000 7f40e3fff640 1 -- 172.21.15.69:0/3853325600 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f0004320 con 0x56288c5d1aa0 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 -- 172.21.15.69:0/3853325600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 msgr2=0x56288c5d3f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f40f0009ca0 tx=0x7f40f0009800 comp rx=0 tx=0).stop 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 -- 172.21.15.69:0/3853325600 shutdown_connections 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5d1aa0 0x56288c5d3f30 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5d1430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.361 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 --2- 172.21.15.69:0/3853325600 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5cc360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 -- 172.21.15.69:0/3853325600 >> 172.21.15.69:0/3853325600 conn(0x56288c5c75b0 msgr2=0x56288c5c79b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 -- 172.21.15.69:0/3853325600 shutdown_connections 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.354+0000 7f4102de6740 1 -- 172.21.15.69:0/3853325600 wait complete. 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 -- start start 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 Processor -- start 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d82b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5df830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56288c5d6a20 con 0x56288c5cc930 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56288c4ebc40 con 0x56288c5d1aa0 2024-05-09T09:19:22.362 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56288c5d6ba0 con 0x56288c5cbf40 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f4102de6740 10 client.0 _ll_register_callbacks cb 0x56288c243be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55912/0 (socket says 172.21.15.69:55912) 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 -- 172.21.15.69:0/3147195594 learned_addr learned my addr 172.21.15.69:0/3147195594 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fe596640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5df830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d82b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 -- 172.21.15.69:0/3147195594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 msgr2=0x56288c5d82b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.363 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d82b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.355+0000 7f40fd594640 1 -- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 msgr2=0x56288c5df830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40fd594640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5df830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40fd594640 1 -- 172.21.15.69:0/3147195594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56288c30e7d0 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40fe596640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40fd594640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f40e809bb50 tx=0x7f40e809c400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ac419fc95fe646e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e80cf400 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f40e8018b30 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40f0036040 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f40e8018d30 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 10 client.15684 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56288c5e2c70 con 0x56288c5cc930 2024-05-09T09:19:22.364 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40f0036040 con 0x56288c5cc930 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f40e8097910 con 0x56288c5cc930 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 10 client.15684 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 10 client.15684 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 20 client.15684 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c5e4390 0x56288c5e6810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 4 client.15684 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.356+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56288c53e370 con 0x56288c5e4390 2024-05-09T09:19:22.365 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.357+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c5e4390 0x56288c5e6810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.357+0000 7f40fdd95640 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c5e4390 0x56288c5e6810 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=ab89fce5324f0657 server_cookie=5630e91d716eda09 in_seq=0 out_seq=0 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.357+0000 7f40e17fa640 10 client.15684 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f40e17fa640 1 -- 172.21.15.69:0/3147195594 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56288c53e370 con 0x56288c5e4390 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f40e17fa640 10 client.15684 handle_command_reply: tid=0 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 client.15684 shutdown 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 msgr2=0x56288c5d87f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f40e809bb50 tx=0x7f40e809c400 comp rx=0 tx=0).stop 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 shutdown_connections 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c5e4390 0x56288c5e6810 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.366 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5df830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5d87f0 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f40e17fa640 0 client.15684 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 --2- 172.21.15.69:0/3147195594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d82b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 >> 172.21.15.69:0/3147195594 conn(0x56288c5c75b0 msgr2=0x56288c5c67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 shutdown_connections 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 1 -- 172.21.15.69:0/3147195594 wait complete. 2024-05-09T09:19:22.367 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.358+0000 7f4102de6740 20 client.15684 trim_cache size 0 max 16384 2024-05-09T09:19:22.377 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 Processor -- start 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 -- start start 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d6f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5ca560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81aaa20 con 0x56288c5cc930 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81a82f0 con 0x56288c5d1aa0 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f4102de6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81a6140 con 0x56288c5cbf40 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55928/0 (socket says 172.21.15.69:55928) 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 -- 172.21.15.69:0/604574737 learned_addr learned my addr 172.21.15.69:0/604574737 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.378 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 -- 172.21.15.69:0/604574737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 msgr2=0x56288c5d6f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fdd95640 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d6f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d6f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 -- 172.21.15.69:0/604574737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 msgr2=0x56288c5ca560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5ca560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 -- 172.21.15.69:0/604574737 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56288c5ccd10 con 0x56288c5cc930 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40fd594640 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f40e809b690 tx=0x7f40e809a0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9ff015ce16c5b92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.372+0000 7f40e37fe640 1 -- 172.21.15.69:0/604574737 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e8018b30 con 0x56288c5cc930 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f40e37fe640 1 -- 172.21.15.69:0/604574737 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e80b2020 con 0x56288c5cc930 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f40e37fe640 1 -- 172.21.15.69:0/604574737 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40e809b010 con 0x56288c5cc930 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 -- 172.21.15.69:0/604574737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 msgr2=0x56288c5df7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f40e809b690 tx=0x7f40e809a0d0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 -- 172.21.15.69:0/604574737 shutdown_connections 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5d1aa0 0x56288c5ca560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.379 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cc930 0x56288c5df7f0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 --2- 172.21.15.69:0/604574737 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5cbf40 0x56288c5d6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 -- 172.21.15.69:0/604574737 >> 172.21.15.69:0/604574737 conn(0x56288c5c75b0 msgr2=0x56288c5cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 -- 172.21.15.69:0/604574737 shutdown_connections 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.373+0000 7f4102de6740 1 -- 172.21.15.69:0/604574737 wait complete. 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 -- start start 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 Processor -- start 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5dd510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5d1aa0 0x56288c5de520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81a67b0 con 0x56288c5cbf40 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81a7c20 con 0x56288c5cc930 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f40f81a6e80 con 0x56288c5d1aa0 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f4102de6740 10 client.0 _ll_register_callbacks cb 0x7f40e800e940 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.380 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55934/0 (socket says 172.21.15.69:55934) 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 -- 172.21.15.69:0/351834842 learned_addr learned my addr 172.21.15.69:0/351834842 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fd594640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5dd510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fe596640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5d1aa0 0x56288c5de520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 -- 172.21.15.69:0/351834842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5d1aa0 msgr2=0x56288c5de520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5d1aa0 0x56288c5de520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 -- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 msgr2=0x56288c5dd510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5dd510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.374+0000 7f40fdd95640 1 -- 172.21.15.69:0/351834842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56288c5deb10 con 0x56288c5cbf40 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40fdd95640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x56288c5cd190 tx=0x7f40f4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d4315eff99e59e80 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f40f40278c0 con 0x56288c5cbf40 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f40f4004030 con 0x56288c5cbf40 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f40e80b2020 con 0x56288c5cbf40 2024-05-09T09:19:22.381 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f40f4035050 con 0x56288c5cbf40 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 10 client.15690 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x56288c64aa80 con 0x56288c5cbf40 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f40e80b2020 con 0x56288c5cbf40 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f40f4004900 con 0x56288c5cbf40 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 10 client.15690 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 10 client.15690 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 20 client.15690 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c64b460 0x56288c64d880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 4 client.15690 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.375+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "damage ls"}) -- 0x56288c5386c0 con 0x56288c64b460 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f40fd594640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c64b460 0x56288c64d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.382 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f40e17fa640 10 client.15690 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f40fd594640 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c64b460 0x56288c64d880 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=62ed2210f0fbf7fc server_cookie=a2bba2d78e940ce8 in_seq=0 out_seq=0 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f40e17fa640 1 -- 172.21.15.69:0/351834842 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x56288c5386c0 con 0x56288c64b460 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f40e17fa640 10 client.15690 handle_command_reply: tid=0 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f4102de6740 1 client.15690 shutdown 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 msgr2=0x56288c5dcfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.376+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x56288c5cd190 tx=0x7f40f4004730 comp rx=0 tx=0).stop 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 shutdown_connections 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x56288c64b460 0x56288c64d880 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56288c5d1aa0 0x56288c5de520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56288c5cc930 0x56288c5dd510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 --2- 172.21.15.69:0/351834842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56288c5cbf40 0x56288c5dcfd0 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40e17fa640 0 client.15690 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 >> 172.21.15.69:0/351834842 conn(0x56288c5c75b0 msgr2=0x56288c5c5f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 shutdown_connections 2024-05-09T09:19:22.383 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 1 -- 172.21.15.69:0/351834842 wait complete. 2024-05-09T09:19:22.384 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f4102de6740 20 client.15690 trim_cache size 0 max 16384 2024-05-09T09:19:22.384 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 msgr2=0x7f40e40a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 0x7f40e40a50d0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f40f81017d0 tx=0x7f40f4005db0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 msgr2=0x7f40f8196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f40e800e9f0 tx=0x7f40e800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 shutdown_connections 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f40e40a2c10 0x7f40e40a50d0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f40f8102330 0x7f40f819df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f40f8101970 0x7f40f8196f00 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 --2- 172.21.15.69:0/3289407180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f40f8100780 0x7f40f81969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 >> 172.21.15.69:0/3289407180 conn(0x7f40f80fbcf0 msgr2=0x7f40f8104c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 shutdown_connections 2024-05-09T09:19:22.385 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.377+0000 7f40ffff9640 1 -- 172.21.15.69:0/3289407180 wait complete. 2024-05-09T09:19:22.387 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:19:22.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 -- 172.21.15.69:0/505554725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 msgr2=0x7faa10100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.493 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 --2- 172.21.15.69:0/505554725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa10100b90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7faa00009900 tx=0x7faa0002a410 comp rx=0 tx=0).stop 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 -- 172.21.15.69:0/505554725 shutdown_connections 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 --2- 172.21.15.69:0/505554725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 --2- 172.21.15.69:0/505554725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa10101960 0x7faa10101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 --2- 172.21.15.69:0/505554725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa10100b90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 -- 172.21.15.69:0/505554725 >> 172.21.15.69:0/505554725 conn(0x7faa100fbce0 msgr2=0x7faa100fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 -- 172.21.15.69:0/505554725 shutdown_connections 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.494+0000 7faa1651d640 1 -- 172.21.15.69:0/505554725 wait complete. 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 Processor -- start 2024-05-09T09:19:22.494 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 -- start start 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa10101960 0x7faa10109730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10109c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d570 con 0x7faa10100770 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d3f0 con 0x7faa10102320 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa1651d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa1010d6f0 con 0x7faa10101960 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55962/0 (socket says 172.21.15.69:55962) 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 -- 172.21.15.69:0/2298663327 learned_addr learned my addr 172.21.15.69:0/2298663327 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.496 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa14aba640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10109c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 -- 172.21.15.69:0/2298663327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa10101960 msgr2=0x7faa10109730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa10101960 0x7faa10109730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 -- 172.21.15.69:0/2298663327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 msgr2=0x7faa10109c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10109c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa0ffff640 1 -- 172.21.15.69:0/2298663327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faa1010a500 con 0x7faa10100770 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.495+0000 7faa14aba640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faa00031440 tx=0x7faa00032e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa0d7fa640 1 -- 172.21.15.69:0/2298663327 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa0003f030 con 0x7faa10100770 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa0d7fa640 1 -- 172.21.15.69:0/2298663327 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa00003ce0 con 0x7faa10100770 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faa101a1cc0 con 0x7faa10100770 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa0d7fa640 1 -- 172.21.15.69:0/2298663327 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa00035070 con 0x7faa10100770 2024-05-09T09:19:22.497 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.496+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faa101a21c0 con 0x7faa10100770 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.498+0000 7faa19222740 1 Processor -- start 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.498+0000 7faa19222740 1 -- start start 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 0x559b0d80d430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d80daa0 0x559b0d80ff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559b0d76cc40 con 0x559b0d808930 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559b0d714580 con 0x559b0d807f40 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa19222740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559b0d768c00 con 0x559b0d80daa0 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa0d7fa640 1 -- 172.21.15.69:0/2298663327 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faa00042030 con 0x7faa10100770 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:44772/0 (socket says 172.21.15.69:44772) 2024-05-09T09:19:22.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 -- 172.21.15.69:0/2942977901 learned_addr learned my addr 172.21.15.69:0/2942977901 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d80daa0 0x559b0d80ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 0x559b0d80d430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 -- 172.21.15.69:0/2942977901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d80daa0 msgr2=0x559b0d80ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d80daa0 0x559b0d80ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 -- 172.21.15.69:0/2942977901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 msgr2=0x559b0d80d430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 0x559b0d80d430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 -- 172.21.15.69:0/2942977901 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0d4803f0 con 0x559b0d807f40 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 0x559b0d80d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.499+0000 7faa14aba640 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa08009f10 tx=0x7faa0800e6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d42a39c1efc66232 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa0d7fa640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 0x7fa9f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 0x7fa9f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7fa9f67fc640 1 -- 172.21.15.69:0/2942977901 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08016070 con 0x559b0d807f40 2024-05-09T09:19:22.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7fa9f67fc640 1 -- 172.21.15.69:0/2942977901 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa08004d20 con 0x559b0d807f40 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7fa9f67fc640 1 -- 172.21.15.69:0/2942977901 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08011850 con 0x559b0d807f40 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa0d7fa640 1 -- 172.21.15.69:0/2298663327 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7faa000efad0 con 0x7faa10100770 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa19222740 1 -- 172.21.15.69:0/2942977901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 msgr2=0x559b0d808360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa19222740 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7faa08009f10 tx=0x7faa0800e6c0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.500+0000 7faa0ffff640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 0x7fa9f80a5050 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faa000ed5c0 tx=0x7faa00009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- 172.21.15.69:0/2942977901 shutdown_connections 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d80daa0 0x559b0d80ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d808930 0x559b0d80d430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- 172.21.15.69:0/2942977901 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d808360 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- 172.21.15.69:0/2942977901 >> 172.21.15.69:0/2942977901 conn(0x559b0d8035b0 msgr2=0x559b0d8039b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- 172.21.15.69:0/2942977901 shutdown_connections 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- 172.21.15.69:0/2942977901 wait complete. 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- start start 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 Processor -- start 2024-05-09T09:19:22.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d812330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d812870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559b0d810a50 con 0x559b0d80daa0 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559b0d727c40 con 0x559b0d807f40 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559b0d810bd0 con 0x559b0d808930 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa19222740 10 client.0 _ll_register_callbacks cb 0x559b0d47fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.501+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55984/0 (socket says 172.21.15.69:55984) 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 -- 172.21.15.69:0/3311235914 learned_addr learned my addr 172.21.15.69:0/3311235914 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d812870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d812330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 -- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 msgr2=0x559b0d812870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d812870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 -- 172.21.15.69:0/3311235914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 msgr2=0x559b0d812330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d812330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 -- 172.21.15.69:0/3311235914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0d54a7d0 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa14aba640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7faa08004cf0 tx=0x7faa080094d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f32b97bd9db2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7fa9ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08016040 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7faa08004ed0 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa08004d20 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa080050d0 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa19222740 10 client.15702 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b0d81cc20 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.502+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa08004d20 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faa0800f040 con 0x559b0d80daa0 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 10 client.15702 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 10 client.15702 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:19:22.508 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 20 client.15702 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d81e030 0x559b0d820470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 4 client.15702 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559b0d77a370 con 0x559b0d81e030 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d81e030 0x559b0d820470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7fa9ef7fe640 10 client.15702 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.503+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d81e030 0x559b0d820470 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=12ac0afb6f5a3401 server_cookie=e4a0b96a555a934a in_seq=0 out_seq=0 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3311235914 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559b0d77a370 con 0x559b0d81e030 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7fa9ef7fe640 10 client.15702 handle_command_reply: tid=0 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 client.15702 shutdown 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 msgr2=0x559b0d8198b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7faa08004cf0 tx=0x7faa080094d0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 shutdown_connections 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d81e030 0x559b0d820470 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8198b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.509 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d812870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7fa9ef7fe640 0 client.15702 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 --2- 172.21.15.69:0/3311235914 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d807f40 0x559b0d812330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 >> 172.21.15.69:0/3311235914 conn(0x559b0d8035b0 msgr2=0x559b0d8027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 shutdown_connections 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.504+0000 7faa19222740 1 -- 172.21.15.69:0/3311235914 wait complete. 2024-05-09T09:19:22.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.505+0000 7faa19222740 20 client.15702 trim_cache size 0 max 16384 2024-05-09T09:19:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:22 smithi069 ceph-mon[30123]: pgmap v379: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:22 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1547125473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 Processor -- start 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 -- start start 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d807f40 0x559b0d809cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d808930 0x559b0d80a230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa101af030 con 0x559b0d80daa0 2024-05-09T09:19:22.518 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa101ac900 con 0x559b0d808930 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa19222740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa101aa750 con 0x559b0d807f40 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55992/0 (socket says 172.21.15.69:55992) 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.518+0000 7faa14aba640 1 -- 172.21.15.69:0/848082895 learned_addr learned my addr 172.21.15.69:0/848082895 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 -- 172.21.15.69:0/848082895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d807f40 msgr2=0x559b0d809cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d807f40 0x559b0d809cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 -- 172.21.15.69:0/848082895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d808930 msgr2=0x559b0d80a230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d808930 0x559b0d80a230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 -- 172.21.15.69:0/848082895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559b0d7f4d20 con 0x559b0d80daa0 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7faa14aba640 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa08002230 tx=0x7faa0800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d02785c064e19126 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/848082895 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08039070 con 0x559b0d80daa0 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/848082895 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa08004f20 con 0x559b0d80daa0 2024-05-09T09:19:22.519 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.519+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/848082895 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08016030 con 0x559b0d80daa0 2024-05-09T09:19:22.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- 172.21.15.69:0/848082895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 msgr2=0x559b0d8197d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.523 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7faa08002230 tx=0x7faa0800e6c0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- 172.21.15.69:0/848082895 shutdown_connections 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d80daa0 0x559b0d8197d0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d808930 0x559b0d80a230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 --2- 172.21.15.69:0/848082895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d807f40 0x559b0d809cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- 172.21.15.69:0/848082895 >> 172.21.15.69:0/848082895 conn(0x559b0d8035b0 msgr2=0x559b0d812330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- 172.21.15.69:0/848082895 shutdown_connections 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- 172.21.15.69:0/848082895 wait complete. 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 -- start start 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.520+0000 7faa19222740 1 Processor -- start 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d817280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d80daa0 0x559b0d818290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faa101aadc0 con 0x559b0d807f40 2024-05-09T09:19:22.524 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faa101ac230 con 0x559b0d80daa0 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faa101ab490 con 0x559b0d808930 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 10 client.0 _ll_register_callbacks cb 0x7faa04005e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56006/0 (socket says 172.21.15.69:56006) 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 -- 172.21.15.69:0/3856568300 learned_addr learned my addr 172.21.15.69:0/3856568300 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d817280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 -- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 msgr2=0x559b0d817280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d817280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 -- 172.21.15.69:0/3856568300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d80daa0 msgr2=0x559b0d818290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d80daa0 0x559b0d818290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 -- 172.21.15.69:0/3856568300 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559b0d818880 con 0x559b0d807f40 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d817280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa14aba640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faa08000f80 tx=0x7faa08009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ae530484df6d4c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.525 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7fa9ef7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faa08039070 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7faa08026410 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7faa08004f20 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7faa08016040 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 10 client.15708 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.521+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x559b0d819b80 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faa08004f20 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7faa080041f0 con 0x559b0d807f40 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 10 client.15708 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 10 client.15708 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 20 client.15708 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d884c60 0x559b0d887080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.526 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 4 client.15708 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "damage ls"}) -- 0x559b0d7746c0 con 0x559b0d884c60 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.522+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d884c60 0x559b0d887080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7fa9ef7fe640 10 client.15708 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa0f7fe640 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d884c60 0x559b0d887080 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=b665e6eadea24c9 server_cookie=54528eed9f02363e in_seq=0 out_seq=0 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7fa9ef7fe640 1 -- 172.21.15.69:0/3856568300 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x559b0d7746c0 con 0x559b0d884c60 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7fa9ef7fe640 10 client.15708 handle_command_reply: tid=0 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 client.15708 shutdown 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 msgr2=0x559b0d816d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7faa08000f80 tx=0x7faa08009580 comp rx=0 tx=0).stop 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 shutdown_connections 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x559b0d884c60 0x559b0d887080 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559b0d80daa0 0x559b0d818290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7fa9ef7fe640 0 client.15708 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559b0d808930 0x559b0d817280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 --2- 172.21.15.69:0/3856568300 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559b0d807f40 0x559b0d816d40 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.527 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 >> 172.21.15.69:0/3856568300 conn(0x559b0d8035b0 msgr2=0x559b0d812310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 shutdown_connections 2024-05-09T09:19:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 1 -- 172.21.15.69:0/3856568300 wait complete. 2024-05-09T09:19:22.528 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.523+0000 7faa19222740 20 client.15708 trim_cache size 0 max 16384 2024-05-09T09:19:22.528 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 msgr2=0x7fa9f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 0x7fa9f80a5050 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7faa000ed5c0 tx=0x7faa00009260 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 msgr2=0x7faa1010b0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7faa00031440 tx=0x7faa00032e20 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 shutdown_connections 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9f80a2b90 0x7fa9f80a5050 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faa10102320 0x7faa10109c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faa10101960 0x7faa10109730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 --2- 172.21.15.69:0/2298663327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faa10100770 0x7faa1010b0f0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 >> 172.21.15.69:0/2298663327 conn(0x7faa100fbce0 msgr2=0x7faa10104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 shutdown_connections 2024-05-09T09:19:22.529 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.524+0000 7faa1651d640 1 -- 172.21.15.69:0/2298663327 wait complete. 2024-05-09T09:19:22.534 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:19:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:22 smithi163 ceph-mon[34354]: pgmap v379: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:22 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1547125473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:22 smithi103 ceph-mon[34421]: pgmap v379: 129 pgs: 129 active+clean; 655 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:22 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1547125473' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1272873777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d4104c80 msgr2=0x7fc9d4107070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1272873777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d4104c80 0x7fc9d4107070 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc009900 tx=0x7fc9bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1272873777 shutdown_connections 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1272873777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d41075b0 0x7fc9d41099a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1272873777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d4104c80 0x7fc9d4107070 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.645 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1272873777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4102350 0x7fc9d4104740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1272873777 >> 172.21.15.69:0/1272873777 conn(0x7fc9d40fbce0 msgr2=0x7fc9d40fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1272873777 shutdown_connections 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.642+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1272873777 wait complete. 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 Processor -- start 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 -- start start 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d4102350 0x7fc9d4197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4104c80 0x7fc9d41981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410d540 con 0x7fc9d41075b0 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410d3c0 con 0x7fc9d4104c80 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d9b63640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc9d410d6c0 con 0x7fc9d4102350 2024-05-09T09:19:22.646 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56032/0 (socket says 172.21.15.69:56032) 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 -- 172.21.15.69:0/1777344778 learned_addr learned my addr 172.21.15.69:0/1777344778 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4104c80 0x7fc9d41981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d4102350 0x7fc9d4197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 -- 172.21.15.69:0/1777344778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d4102350 msgr2=0x7fc9d4197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d4102350 0x7fc9d4197c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 -- 172.21.15.69:0/1777344778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4104c80 msgr2=0x7fc9d41981a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4104c80 0x7fc9d41981a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 -- 172.21.15.69:0/1777344778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc9d419f870 con 0x7fc9d41075b0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.643+0000 7fc9d3fff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00b970 tx=0x7fc9cc00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.644+0000 7fc9d17fa640 1 -- 172.21.15.69:0/1777344778 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc019070 con 0x7fc9d41075b0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.644+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc9d419fb60 con 0x7fc9d41075b0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.644+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc9d41a00b0 con 0x7fc9d41075b0 2024-05-09T09:19:22.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.644+0000 7fc9d17fa640 1 -- 172.21.15.69:0/1777344778 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9cc00cb40 con 0x7fc9d41075b0 2024-05-09T09:19:22.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.644+0000 7fc9d17fa640 1 -- 172.21.15.69:0/1777344778 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9cc01d5d0 con 0x7fc9d41075b0 2024-05-09T09:19:22.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.646+0000 7fc9d17fa640 1 -- 172.21.15.69:0/1777344778 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc9cc01c070 con 0x7fc9d41075b0 2024-05-09T09:19:22.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.646+0000 7fc9dc875740 1 Processor -- start 2024-05-09T09:19:22.648 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.646+0000 7fc9dc875740 1 -- start start 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55606519f930 0x5560651a4430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5560651a4aa0 0x5560651a6f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556065103c40 con 0x55606519ef40 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5560650ab580 con 0x55606519f930 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9dc875740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5560650ffc00 con 0x5560651a4aa0 2024-05-09T09:19:22.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9d17fa640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 0x7fc9c00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56042/0 (socket says 172.21.15.69:56042) 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 -- 172.21.15.69:0/4237749044 learned_addr learned my addr 172.21.15.69:0/4237749044 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55606519f930 0x5560651a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9d3fff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 0x7fc9c00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 -- 172.21.15.69:0/4237749044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5560651a4aa0 msgr2=0x5560651a6f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5560651a4aa0 0x5560651a6f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 -- 172.21.15.69:0/4237749044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55606519f930 msgr2=0x5560651a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55606519f930 0x5560651a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 -- 172.21.15.69:0/4237749044 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556064e173f0 con 0x55606519ef40 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9d17fa640 1 -- 172.21.15.69:0/1777344778 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc9cc0ce920 con 0x7fc9d41075b0 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0314e0 tx=0x7fc9bc009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ec4d1c0bc3bdd02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9d3fff640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 0x7fc9c00a5050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0977f0 tx=0x7fc9cc018f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.647+0000 7fc9c9ffb640 1 -- 172.21.15.69:0/4237749044 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc035040 con 0x55606519ef40 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9c9ffb640 1 -- 172.21.15.69:0/4237749044 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9bc005500 con 0x55606519ef40 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9c9ffb640 1 -- 172.21.15.69:0/4237749044 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc004db0 con 0x55606519ef40 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 -- 172.21.15.69:0/4237749044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 msgr2=0x55606519f360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0314e0 tx=0x7fc9bc009680 comp rx=0 tx=0).stop 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 -- 172.21.15.69:0/4237749044 shutdown_connections 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5560651a4aa0 0x5560651a6f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55606519f930 0x5560651a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 --2- 172.21.15.69:0/4237749044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x55606519f360 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 -- 172.21.15.69:0/4237749044 >> 172.21.15.69:0/4237749044 conn(0x55606519a5b0 msgr2=0x55606519a9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 -- 172.21.15.69:0/4237749044 shutdown_connections 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.648+0000 7fc9dc875740 1 -- 172.21.15.69:0/4237749044 wait complete. 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 -- start start 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 Processor -- start 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651a9880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651b08c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5560651a79f0 con 0x55606519ef40 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5560650bec40 con 0x5560651a4aa0 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5560651a7b70 con 0x55606519f930 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9dc875740 10 client.0 _ll_register_callbacks cb 0x556064e16be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56046/0 (socket says 172.21.15.69:56046) 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3207621917 learned_addr learned my addr 172.21.15.69:0/3207621917 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651b08c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651a9880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3207621917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 msgr2=0x5560651a9880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651a9880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 msgr2=0x5560651b08c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651b08c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.649+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3207621917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556064ee17d0 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc004890 tx=0x7fc9bc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43496c2c05951456 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc004450 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc9bc004de0 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9bc005500 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc9bc05b050 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 10 client.15720 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5560651b3c30 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9bc005500 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc9bc045440 con 0x55606519ef40 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 10 client.15720 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 10 client.15720 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 20 client.15720 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5560651b4f80 0x5560651b7370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 4 client.15720 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.650+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556065111370 con 0x5560651b4f80 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.651+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5560651b4f80 0x5560651b7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.651+0000 7fc9b2ffd640 10 client.15720 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.651+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5560651b4f80 0x5560651b7370 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d2bbb8def9f9946 server_cookie=a13b5e5f7c76d8ae in_seq=0 out_seq=0 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3207621917 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556065111370 con 0x5560651b4f80 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9b2ffd640 10 client.15720 handle_command_reply: tid=0 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 client.15720 shutdown 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 msgr2=0x5560651a9340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc004890 tx=0x7fc9bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:22.660 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 shutdown_connections 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651b08c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5560651b4f80 0x5560651b7370 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651a9880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9b2ffd640 0 client.15720 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3207621917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651a9340 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 >> 172.21.15.69:0/3207621917 conn(0x55606519a5b0 msgr2=0x5560651997b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 shutdown_connections 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.652+0000 7fc9dc875740 1 -- 172.21.15.69:0/3207621917 wait complete. 2024-05-09T09:19:22.661 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.653+0000 7fc9dc875740 20 client.15720 trim_cache size 0 max 16384 2024-05-09T09:19:22.673 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 Processor -- start 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 -- start start 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519ef40 0x5560651a6e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651a8f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b4012400 con 0x55606519f930 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b400fcd0 con 0x5560651a4aa0 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9dc875740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b400db20 con 0x55606519ef40 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56052/0 (socket says 172.21.15.69:56052) 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 -- 172.21.15.69:0/540623438 learned_addr learned my addr 172.21.15.69:0/540623438 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.674 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519ef40 0x5560651a6e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651a8f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 -- 172.21.15.69:0/540623438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519ef40 msgr2=0x5560651a6e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519ef40 0x5560651a6e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 -- 172.21.15.69:0/540623438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 msgr2=0x5560651a8f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651a8f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9d37fe640 1 -- 172.21.15.69:0/540623438 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5560651b79d0 con 0x55606519f930 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.666+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651a8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.667+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400ab00 tx=0x7fc9c400a560 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f85b68a090fa30c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.667+0000 7fc9c97fa640 1 -- 172.21.15.69:0/540623438 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c401f030 con 0x55606519f930 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.667+0000 7fc9c97fa640 1 -- 172.21.15.69:0/540623438 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c401b880 con 0x55606519f930 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.667+0000 7fc9c97fa640 1 -- 172.21.15.69:0/540623438 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9c4004a00 con 0x55606519f930 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- 172.21.15.69:0/540623438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 msgr2=0x5560651b0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.675 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7fc9c400ab00 tx=0x7fc9c400a560 comp rx=0 tx=0).stop 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- 172.21.15.69:0/540623438 shutdown_connections 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651a8f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519f930 0x5560651b0850 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- 172.21.15.69:0/540623438 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519ef40 0x5560651a6e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- 172.21.15.69:0/540623438 >> 172.21.15.69:0/540623438 conn(0x55606519a5b0 msgr2=0x55606519a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- 172.21.15.69:0/540623438 shutdown_connections 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- 172.21.15.69:0/540623438 wait complete. 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- start start 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 Processor -- start 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651ae110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651af0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.676 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b400e190 con 0x55606519ef40 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b400f600 con 0x5560651a4aa0 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc9b400e860 con 0x55606519f930 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9dc875740 10 client.0 _ll_register_callbacks cb 0x7fc9bc0318d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.668+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56062/0 (socket says 172.21.15.69:56062) 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3842622646 learned_addr learned my addr 172.21.15.69:0/3842622646 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651ae110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3842622646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 msgr2=0x5560651ae110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651ae110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 msgr2=0x5560651af0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651af0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 -- 172.21.15.69:0/3842622646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5560651af6c0 con 0x55606519ef40 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:22.677 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9cbfff640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0458e0 tx=0x7fc9bc0315d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6621fd08ea83fc0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9b2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc9bc035040 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc9c401b880 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc9bc004250 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc9bc06f050 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9dc875740 10 client.15726 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5560651b09c0 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.669+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc9c401b880 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc9bc004d40 con 0x55606519ef40 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 10 client.15726 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 10 client.15726 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 20 client.15726 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55606521ba10 0x55606521de30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:22.678 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 4 client.15726 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55606510b6c0 con 0x55606521ba10 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55606521ba10 0x55606521de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9b2ffd640 10 client.15726 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.670+0000 7fc9d37fe640 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55606521ba10 0x55606521de30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a50d4ef78557ef8 server_cookie=d1b23fb6542435d7 in_seq=0 out_seq=0 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9b2ffd640 1 -- 172.21.15.69:0/3842622646 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55606510b6c0 con 0x55606521ba10 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9b2ffd640 10 client.15726 handle_command_reply: tid=0 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 client.15726 shutdown 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 msgr2=0x5560651adbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7fc9bc0458e0 tx=0x7fc9bc0315d0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 shutdown_connections 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5560651a4aa0 0x5560651af0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55606521ba10 0x55606521de30 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55606519f930 0x5560651ae110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.679 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 --2- 172.21.15.69:0/3842622646 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55606519ef40 0x5560651adbd0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9b2ffd640 0 client.15726 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 >> 172.21.15.69:0/3842622646 conn(0x55606519a5b0 msgr2=0x5560651a8f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 shutdown_connections 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 1 -- 172.21.15.69:0/3842622646 wait complete. 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9dc875740 20 client.15726 trim_cache size 0 max 16384 2024-05-09T09:19:22.680 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 msgr2=0x7fc9c00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 0x7fc9c00a5050 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc0977f0 tx=0x7fc9cc018f70 comp rx=0 tx=0).stop 2024-05-09T09:19:22.681 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 msgr2=0x7fc9d419f1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7fc9cc00b970 tx=0x7fc9cc00be30 comp rx=0 tx=0).stop 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 shutdown_connections 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9c00a2b90 0x7fc9c00a5050 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc9d41075b0 0x7fc9d419f1d0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc9d4104c80 0x7fc9d41981a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.671+0000 7fc9d9b63640 1 --2- 172.21.15.69:0/1777344778 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc9d4102350 0x7fc9d4197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.672+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 >> 172.21.15.69:0/1777344778 conn(0x7fc9d40fbce0 msgr2=0x7fc9d4102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.672+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 shutdown_connections 2024-05-09T09:19:22.682 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:22.672+0000 7fc9d9b63640 1 -- 172.21.15.69:0/1777344778 wait complete. 2024-05-09T09:19:23.683 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:19:23.684 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:19:23.684 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:19:23.795 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- 172.21.15.69:0/4258682458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024102370 msgr2=0x7fe024104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 --2- 172.21.15.69:0/4258682458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024102370 0x7fe024104760 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7fe014009900 tx=0x7fe01402a410 comp rx=0 tx=0).stop 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- 172.21.15.69:0/4258682458 shutdown_connections 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 --2- 172.21.15.69:0/4258682458 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0241075d0 0x7fe0241099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 --2- 172.21.15.69:0/4258682458 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024104ca0 0x7fe024107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 --2- 172.21.15.69:0/4258682458 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024102370 0x7fe024104760 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- 172.21.15.69:0/4258682458 >> 172.21.15.69:0/4258682458 conn(0x7fe0240fbce0 msgr2=0x7fe0240fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- 172.21.15.69:0/4258682458 shutdown_connections 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- 172.21.15.69:0/4258682458 wait complete. 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 Processor -- start 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.796+0000 7fe02a212640 1 -- start start 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024102370 0x7fe024101ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.796 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0241075d0 0x7fe024100860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe02410d510 con 0x7fe024104ca0 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe02410d390 con 0x7fe0241075d0 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe02410d690 con 0x7fe024102370 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56078/0 (socket says 172.21.15.69:56078) 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 -- 172.21.15.69:0/3203546355 learned_addr learned my addr 172.21.15.69:0/3203546355 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.798 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024102370 0x7fe024101ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 -- 172.21.15.69:0/3203546355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024102370 msgr2=0x7fe024101ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024102370 0x7fe024101ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 -- 172.21.15.69:0/3203546355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0241075d0 msgr2=0x7fe024100860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0241075d0 0x7fe024100860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 -- 172.21.15.69:0/3203546355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0241010f0 con 0x7fe024104ca0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe01800d740 tx=0x7fe01800dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe020ff9640 1 -- 172.21.15.69:0/3203546355 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018017070 con 0x7fe024104ca0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0241a1d60 con 0x7fe024104ca0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe020ff9640 1 -- 172.21.15.69:0/3203546355 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01800de50 con 0x7fe024104ca0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe020ff9640 1 -- 172.21.15.69:0/3203546355 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018004e00 con 0x7fe024104ca0 2024-05-09T09:19:23.799 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.797+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0241a21f0 con 0x7fe024104ca0 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.800+0000 7fe020ff9640 1 -- 172.21.15.69:0/3203546355 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe018004fa0 con 0x7fe024104ca0 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.800+0000 7fe020ff9640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 0x7fe00c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.800+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 0x7fe00c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe020ff9640 1 -- 172.21.15.69:0/3203546355 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fe0180cee70 con 0x7fe024104ca0 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 Processor -- start 2024-05-09T09:19:23.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 -- start start 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 0x7fe00c0a5050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe014000c00 tx=0x7fe0140315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 0x55dd460858e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd4608d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dd45fe9990 con 0x55dd46085eb0 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dd45f915b0 con 0x55dd460854c0 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe02cffb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dd45fe5950 con 0x55dd4608b020 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56086/0 (socket says 172.21.15.69:56086) 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.801+0000 7fe022ffd640 1 -- 172.21.15.69:0/1091748029 learned_addr learned my addr 172.21.15.69:0/1091748029 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe023fff640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd4608d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 -- 172.21.15.69:0/1091748029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 msgr2=0x55dd4608d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd4608d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe0237fe640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 0x55dd460858e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 -- 172.21.15.69:0/1091748029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 msgr2=0x55dd460858e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 0x55dd460858e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 -- 172.21.15.69:0/1091748029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd45dc57e0 con 0x55dd46085eb0 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe023fff640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd4608d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe0237fe640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 0x55dd460858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe022ffd640 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe0180977b0 tx=0x7fe0180b9f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d14d0bf46d66f95b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe009ffb640 1 -- 172.21.15.69:0/1091748029 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018020070 con 0x55dd46085eb0 2024-05-09T09:19:23.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe009ffb640 1 -- 172.21.15.69:0/1091748029 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe018099650 con 0x55dd46085eb0 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.802+0000 7fe009ffb640 1 -- 172.21.15.69:0/1091748029 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01809aab0 con 0x55dd46085eb0 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- 172.21.15.69:0/1091748029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 msgr2=0x55dd4608a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7fe0180977b0 tx=0x7fe0180b9f90 comp rx=0 tx=0).stop 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- 172.21.15.69:0/1091748029 shutdown_connections 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd4608d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608a9b0 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 --2- 172.21.15.69:0/1091748029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd460854c0 0x55dd460858e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- 172.21.15.69:0/1091748029 >> 172.21.15.69:0/1091748029 conn(0x55dd4604d0a0 msgr2=0x55dd46080f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- 172.21.15.69:0/1091748029 shutdown_connections 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- 172.21.15.69:0/1091748029 wait complete. 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.803+0000 7fe02cffb740 1 -- start start 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 Processor -- start 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd4608f860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd4608b020 0x55dd46096de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dd4608df70 con 0x55dd46085eb0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dd45fa4b40 con 0x55dd4608b020 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dd4608e0f0 con 0x55dd460854c0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe02cffb740 10 client.0 _ll_register_callbacks cb 0x55dd45f990f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56098/0 (socket says 172.21.15.69:56098) 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 -- 172.21.15.69:0/924085558 learned_addr learned my addr 172.21.15.69:0/924085558 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe0237fe640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd4608f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe023fff640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd4608b020 0x55dd46096de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 -- 172.21.15.69:0/924085558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 msgr2=0x55dd4608f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd4608f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 -- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd4608b020 msgr2=0x55dd46096de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd4608b020 0x55dd46096de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.804+0000 7fe022ffd640 1 -- 172.21.15.69:0/924085558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd45e25f70 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe022ffd640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe0180cf190 tx=0x7fe0180048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d25452e7f2d0174c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe018020070 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fe01809a810 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe018099650 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe0180979c0 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 10 client.15738 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd45cbd170 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe018099650 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe0180ccbd0 con 0x55dd46085eb0 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 10 client.15738 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:23.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 10 client.15738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 20 client.15738 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd4609aeb0 0x55dd4609d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 4 client.15738 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.805+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dd45ff70c0 con 0x55dd4609aeb0 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.806+0000 7fe0237fe640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd4609aeb0 0x55dd4609d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.806+0000 7fe002ffd640 10 client.15738 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.806+0000 7fe0237fe640 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd4609aeb0 0x55dd4609d2a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e27686691d0e4fc8 server_cookie=806517bb585fdc4c in_seq=0 out_seq=0 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe002ffd640 1 -- 172.21.15.69:0/924085558 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dd45ff70c0 con 0x55dd4609aeb0 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe002ffd640 10 client.15738 handle_command_reply: tid=0 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 client.15738 shutdown 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 msgr2=0x55dd4608fda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7fe0180cf190 tx=0x7fe0180048a0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 shutdown_connections 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd4608b020 0x55dd46096de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd46085eb0 0x55dd4608fda0 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd4609aeb0 0x55dd4609d2a0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 --2- 172.21.15.69:0/924085558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd4608f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe002ffd640 0 client.15738 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 >> 172.21.15.69:0/924085558 conn(0x55dd4604d0a0 msgr2=0x55dd46080080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 shutdown_connections 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 1 -- 172.21.15.69:0/924085558 wait complete. 2024-05-09T09:19:23.813 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.807+0000 7fe02cffb740 20 client.15738 trim_cache size 0 max 16384 2024-05-09T09:19:23.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 Processor -- start 2024-05-09T09:19:23.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 -- start start 2024-05-09T09:19:23.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd46087310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.847 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46087850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241aefe0 con 0x55dd4608b020 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241ac8b0 con 0x55dd46085eb0 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe02cffb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241aa700 con 0x55dd460854c0 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe023fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe023fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56110/0 (socket says 172.21.15.69:56110) 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe023fff640 1 -- 172.21.15.69:0/22205960 learned_addr learned my addr 172.21.15.69:0/22205960 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.843+0000 7fe022ffd640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46087850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe0237fe640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd46087310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 -- 172.21.15.69:0/22205960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 msgr2=0x55dd46087310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd46087310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 -- 172.21.15.69:0/22205960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 msgr2=0x55dd46087850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.848 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46087850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 -- 172.21.15.69:0/22205960 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dd46086290 con 0x55dd4608b020 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe0237fe640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd46087310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe023fff640 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009eb0 tx=0x7fe01c010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d9eb2920543a74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe0097fa640 1 -- 172.21.15.69:0/22205960 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c016070 con 0x55dd4608b020 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe0097fa640 1 -- 172.21.15.69:0/22205960 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01c010e90 con 0x55dd4608b020 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.844+0000 7fe0097fa640 1 -- 172.21.15.69:0/22205960 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe01c012c00 con 0x55dd4608b020 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- 172.21.15.69:0/22205960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 msgr2=0x55dd46096da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7fe01c009eb0 tx=0x7fe01c010740 comp rx=0 tx=0).stop 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- 172.21.15.69:0/22205960 shutdown_connections 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd4608b020 0x55dd46096da0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46087850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 --2- 172.21.15.69:0/22205960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd460854c0 0x55dd46087310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- 172.21.15.69:0/22205960 >> 172.21.15.69:0/22205960 conn(0x55dd4604d0a0 msgr2=0x55dd4608f860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.849 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- 172.21.15.69:0/22205960 shutdown_connections 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- 172.21.15.69:0/22205960 wait complete. 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 -- start start 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.845+0000 7fe02cffb740 1 Processor -- start 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46094f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd46095f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241aad70 con 0x55dd460854c0 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241ac1e0 con 0x55dd46085eb0 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0241ab440 con 0x55dd4608b020 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe02cffb740 10 client.0 _ll_register_callbacks cb 0x7fe01809f070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:23.850 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56120/0 (socket says 172.21.15.69:56120) 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 -- 172.21.15.69:0/3827725126 learned_addr learned my addr 172.21.15.69:0/3827725126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46094f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe023fff640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd46095f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 -- 172.21.15.69:0/3827725126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 msgr2=0x55dd46095f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd46095f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 -- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 msgr2=0x55dd46094f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46094f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 -- 172.21.15.69:0/3827725126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dd46096510 con 0x55dd460854c0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46094f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe023fff640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd46095f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe0237fe640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe014032fd0 tx=0x7fe01402a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9402d5e03c48250 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe002ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:23.851 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe014035050 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.846+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fe014050c10 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe01c010e90 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fe014050e10 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 10 client.15744 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55dd461016b0 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe01c010e90 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fe0140502f0 con 0x55dd460854c0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 10 client.15744 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 10 client.15744 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 20 client.15744 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd461020b0 0x55dd461044d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 4 client.15744 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55dd4608f650 con 0x55dd461020b0 2024-05-09T09:19:23.852 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.847+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd461020b0 0x55dd461044d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.848+0000 7fe002ffd640 10 client.15744 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.853 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.848+0000 7fe022ffd640 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd461020b0 0x55dd461044d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c1a50bd36edc4c3 server_cookie=4bd6ae314732c18d in_seq=0 out_seq=0 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe002ffd640 1 -- 172.21.15.69:0/3827725126 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55dd4608f650 con 0x55dd461020b0 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe002ffd640 10 client.15744 handle_command_reply: tid=0 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 client.15744 shutdown 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 msgr2=0x55dd46094a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7fe014032fd0 tx=0x7fe01402a410 comp rx=0 tx=0).stop 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 shutdown_connections 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dd461020b0 0x55dd461044d0 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dd4608b020 0x55dd46095f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dd46085eb0 0x55dd46094f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 --2- 172.21.15.69:0/3827725126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dd460854c0 0x55dd46094a20 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.854 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 >> 172.21.15.69:0/3827725126 conn(0x55dd4604d0a0 msgr2=0x55dd4608f800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe002ffd640 0 client.15744 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 shutdown_connections 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 1 -- 172.21.15.69:0/3827725126 wait complete. 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.849+0000 7fe02cffb740 20 client.15744 trim_cache size 0 max 16384 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "b545ac4c-8f31-4335-bc30-fee8ee66e7e3", 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-05-09T09:19:23.855 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:23.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 msgr2=0x7fe00c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 0x7fe00c0a5050 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe014000c00 tx=0x7fe0140315d0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.856 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 msgr2=0x7fe024100320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7fe01800d740 tx=0x7fe01800dc00 comp rx=0 tx=0).stop 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 shutdown_connections 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe00c0a2b90 0x7fe00c0a5050 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0241075d0 0x7fe024100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe024104ca0 0x7fe024100320 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 --2- 172.21.15.69:0/3203546355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe024102370 0x7fe024101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 >> 172.21.15.69:0/3203546355 conn(0x7fe0240fbce0 msgr2=0x7fe0241041a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 shutdown_connections 2024-05-09T09:19:23.857 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.850+0000 7fe02a212640 1 -- 172.21.15.69:0/3203546355 wait complete. 2024-05-09T09:19:23.860 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:19:23.860 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:19:23.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- 172.21.15.69:0/846757993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 msgr2=0x7f72c0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/846757993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0100b90 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f72bc009900 tx=0x7f72bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:23.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- 172.21.15.69:0/846757993 shutdown_connections 2024-05-09T09:19:23.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/846757993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0102320 0x7f72c0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.970 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/846757993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0101960 0x7f72c0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/846757993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0100b90 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- 172.21.15.69:0/846757993 >> 172.21.15.69:0/846757993 conn(0x7f72c00fbce0 msgr2=0x7f72c00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- 172.21.15.69:0/846757993 shutdown_connections 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- 172.21.15.69:0/846757993 wait complete. 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 Processor -- start 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- start start 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0101960 0x7f72c0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0102320 0x7f72c019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c3c0 con 0x7f72c0100770 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c240 con 0x7f72c0101960 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.969+0000 7f72c7ea5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72c010c540 con 0x7f72c0102320 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.971 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56132/0 (socket says 172.21.15.69:56132) 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 -- 172.21.15.69:0/1788311503 learned_addr learned my addr 172.21.15.69:0/1788311503 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5440640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0101960 0x7f72c0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c6442640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0102320 0x7f72c019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 -- 172.21.15.69:0/1788311503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0102320 msgr2=0x7f72c019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0102320 0x7f72c019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 -- 172.21.15.69:0/1788311503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0101960 msgr2=0x7f72c0198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0101960 0x7f72c0198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 -- 172.21.15.69:0/1788311503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72c019f810 con 0x7f72c0100770 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f72bc031610 tx=0x7f72bc0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72b6ffd640 1 -- 172.21.15.69:0/1788311503 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc03f030 con 0x7f72c0100770 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72b6ffd640 1 -- 172.21.15.69:0/1788311503 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72bc003ce0 con 0x7f72c0100770 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72c019faa0 con 0x7f72c0100770 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72b6ffd640 1 -- 172.21.15.69:0/1788311503 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc035070 con 0x7f72c0100770 2024-05-09T09:19:23.972 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.970+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72c019ff90 con 0x7f72c0100770 2024-05-09T09:19:23.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72b6ffd640 1 -- 172.21.15.69:0/1788311503 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f72bc042030 con 0x7f72c0100770 2024-05-09T09:19:23.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 Processor -- start 2024-05-09T09:19:23.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 -- start start 2024-05-09T09:19:23.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b0985b350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b09862f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556b097bfd00 con 0x556b0985b920 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556b097674c0 con 0x556b0985af30 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72cabb9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556b097bbcc0 con 0x556b09860a90 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56142/0 (socket says 172.21.15.69:56142) 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.973+0000 7f72c5c41640 1 -- 172.21.15.69:0/68935373 learned_addr learned my addr 172.21.15.69:0/68935373 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72b6ffd640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 0x7f72ac0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5440640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b09862f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c6442640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b0985b350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c6442640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 0x7f72ac0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72b6ffd640 1 -- 172.21.15.69:0/1788311503 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f72bc0efa10 con 0x7f72c0100770 2024-05-09T09:19:23.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 -- 172.21.15.69:0/68935373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 msgr2=0x556b09862f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b09862f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 -- 172.21.15.69:0/68935373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 msgr2=0x556b0985b350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b0985b350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 -- 172.21.15.69:0/68935373 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b0959d7d0 con 0x556b0985b920 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5440640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b09862f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c6442640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b0985b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c5c41640 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0b88c0 tx=0x7f72bc02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3df621e8dcb6222 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72c6442640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 0x7f72ac0a5050 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x556b0985c180 tx=0x7f72b8005c20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72abfff640 1 -- 172.21.15.69:0/68935373 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc049070 con 0x556b0985b920 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72abfff640 1 -- 172.21.15.69:0/68935373 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72bc0bbef0 con 0x556b0985b920 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.974+0000 7f72abfff640 1 -- 172.21.15.69:0/68935373 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc003800 con 0x556b0985b920 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- 172.21.15.69:0/68935373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 msgr2=0x556b09860420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0b88c0 tx=0x7f72bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:23.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- 172.21.15.69:0/68935373 shutdown_connections 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b09862f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b09860420 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 --2- 172.21.15.69:0/68935373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b0985b350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- 172.21.15.69:0/68935373 >> 172.21.15.69:0/68935373 conn(0x556b098565a0 msgr2=0x556b098569a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- 172.21.15.69:0/68935373 shutdown_connections 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- 172.21.15.69:0/68935373 wait complete. 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.975+0000 7f72cabb9740 1 -- start start 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 Processor -- start 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b09865010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 0x556b09865550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556b097bfd00 con 0x556b09860a90 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556b0977ab90 con 0x556b0985af30 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556b09863800 con 0x556b0985b920 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72cabb9740 10 client.0 _ll_register_callbacks cb 0x556b097b6da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56144/0 (socket says 172.21.15.69:56144) 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 -- 172.21.15.69:0/1988251809 learned_addr learned my addr 172.21.15.69:0/1988251809 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b09865010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c5440640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 0x556b09865550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 -- 172.21.15.69:0/1988251809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 msgr2=0x556b09865550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 0x556b09865550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 -- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 msgr2=0x556b09865010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b09865010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c6442640 1 -- 172.21.15.69:0/1988251809 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b094d33f0 con 0x556b09860a90 2024-05-09T09:19:23.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.976+0000 7f72c5440640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 0x556b09865550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72c6442640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f72b800ef50 tx=0x7f72b8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eea35ce293c487f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b8022070 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f72b8010d80 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72bc0bbef0 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f72b802e050 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 10 client.15756 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556b094930f0 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72bc0bbef0 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f72b801d040 con 0x556b09860a90 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 10 client.15756 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 10 client.15756 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 20 client.15756 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:23.984 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b09870ef0 0x556b09873390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 4 client.15756 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.977+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556b097cd430 con 0x556b09870ef0 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.978+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b09870ef0 0x556b09873390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.978+0000 7f72a97fa640 10 client.15756 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.978+0000 7f72c5c41640 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b09870ef0 0x556b09873390 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd501cc977e6e23f server_cookie=a5c6c332a159fb55 in_seq=0 out_seq=0 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72a97fa640 1 -- 172.21.15.69:0/1988251809 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556b097cd430 con 0x556b09870ef0 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72a97fa640 10 client.15756 handle_command_reply: tid=0 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 client.15756 shutdown 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 msgr2=0x556b0986c590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f72b800ef50 tx=0x7f72b8004a40 comp rx=0 tx=0).stop 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 shutdown_connections 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b09860a90 0x556b0986c590 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.985 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b09870ef0 0x556b09873390 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985b920 0x556b09865550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 --2- 172.21.15.69:0/1988251809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985af30 0x556b09865010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72a97fa640 0 client.15756 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 >> 172.21.15.69:0/1988251809 conn(0x556b098565a0 msgr2=0x556b098557e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 shutdown_connections 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 1 -- 172.21.15.69:0/1988251809 wait complete. 2024-05-09T09:19:23.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.979+0000 7f72cabb9740 20 client.15756 trim_cache size 0 max 16384 2024-05-09T09:19:23.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 Processor -- start 2024-05-09T09:19:23.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 -- start start 2024-05-09T09:19:23.997 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985af30 0x556b09855660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 0x556b09877c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01acdf0 con 0x556b0985b920 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01aa6c0 con 0x556b09860a90 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72cabb9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01a8510 con 0x556b0985af30 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72c5440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72c5440640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56148/0 (socket says 172.21.15.69:56148) 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.997+0000 7f72c5440640 1 -- 172.21.15.69:0/781844060 learned_addr learned my addr 172.21.15.69:0/781844060 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c6442640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 0x556b09877c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5c41640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985af30 0x556b09855660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 -- 172.21.15.69:0/781844060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985af30 msgr2=0x556b09855660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985af30 0x556b09855660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.998 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 -- 172.21.15.69:0/781844060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 msgr2=0x556b09877c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 0x556b09877c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 -- 172.21.15.69:0/781844060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556b098563d0 con 0x556b0985b920 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c6442640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 0x556b09877c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72c5440640 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f72b000dc60 tx=0x7f72b00127c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efdb90ff30ae2ee1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72ab7fe640 1 -- 172.21.15.69:0/781844060 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b000de10 con 0x556b0985b920 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72ab7fe640 1 -- 172.21.15.69:0/781844060 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b00035f0 con 0x556b0985b920 2024-05-09T09:19:23.999 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.998+0000 7f72ab7fe640 1 -- 172.21.15.69:0/781844060 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72b0010810 con 0x556b0985b920 2024-05-09T09:19:24.003 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- 172.21.15.69:0/781844060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 msgr2=0x556b0986c550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f72b000dc60 tx=0x7f72b00127c0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- 172.21.15.69:0/781844060 shutdown_connections 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b09860a90 0x556b09877c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985b920 0x556b0986c550 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 --2- 172.21.15.69:0/781844060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b0985af30 0x556b09855660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- 172.21.15.69:0/781844060 >> 172.21.15.69:0/781844060 conn(0x556b098565a0 msgr2=0x556b09854f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- 172.21.15.69:0/781844060 shutdown_connections 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- 172.21.15.69:0/781844060 wait complete. 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:23.999+0000 7f72cabb9740 1 -- start start 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 Processor -- start 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 0x556b0986ac90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:24.004 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b0986bb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01a8b80 con 0x556b0985af30 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01a9ff0 con 0x556b0985b920 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72c01a9250 con 0x556b09860a90 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72cabb9740 10 client.0 _ll_register_callbacks cb 0x7f72b800bb70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56162/0 (socket says 172.21.15.69:56162) 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 -- 172.21.15.69:0/2805729636 learned_addr learned my addr 172.21.15.69:0/2805729636 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:24.005 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5440640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 0x556b0986ac90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c6442640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b0986bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 -- 172.21.15.69:0/2805729636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 msgr2=0x556b0986bb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b0986bb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 -- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 msgr2=0x556b0986ac90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 0x556b0986ac90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5c41640 1 -- 172.21.15.69:0/2805729636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556b0986c200 con 0x556b0985af30 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.000+0000 7f72c5440640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 0x556b0986ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72c5c41640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0314e0 tx=0x7f72bc004520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0762d545b5dc810 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72bc049040 con 0x556b0985af30 2024-05-09T09:19:24.006 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f72bc003800 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72b00035f0 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f72bc0c02a0 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 10 client.15762 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x556b098d7ed0 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72b00035f0 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f72bc03b460 con 0x556b0985af30 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 10 client.15762 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 10 client.15762 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 20 client.15762 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b098d8740 0x556b098dab60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 4 client.15762 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.001+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556b097c7780 con 0x556b098d8740 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72c5440640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b098d8740 0x556b098dab60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:24.007 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72a97fa640 10 client.15762 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72c5440640 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b098d8740 0x556b098dab60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e6014f5d280145f2 server_cookie=9dc8ba3353511e2 in_seq=0 out_seq=0 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72a97fa640 1 -- 172.21.15.69:0/2805729636 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556b097c7780 con 0x556b098d8740 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72a97fa640 10 client.15762 handle_command_reply: tid=0 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 client.15762 shutdown 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 msgr2=0x556b0986a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f72bc0314e0 tx=0x7f72bc004520 comp rx=0 tx=0).stop 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 shutdown_connections 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556b098d8740 0x556b098dab60 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556b09860a90 0x556b0986bb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556b0985b920 0x556b0986ac90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72a97fa640 0 client.15762 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 --2- 172.21.15.69:0/2805729636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556b0985af30 0x556b0986a750 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.002+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 >> 172.21.15.69:0/2805729636 conn(0x556b098565a0 msgr2=0x556b098557e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:24.008 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 shutdown_connections 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72cabb9740 1 -- 172.21.15.69:0/2805729636 wait complete. 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72cabb9740 20 client.15762 trim_cache size 0 max 16384 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "b545ac4c-8f31-4335-bc30-fee8ee66e7e3": { 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "b545ac4c-8f31-4335-bc30-fee8ee66e7e3", 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,force" 2024-05-09T09:19:24.009 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:19:24.010 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:19:24.010 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 msgr2=0x7f72ac0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 0x7f72ac0a5050 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x556b0985c180 tx=0x7f72b8005c20 comp rx=0 tx=0).stop 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 msgr2=0x7f72c0197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f72bc031610 tx=0x7f72bc0043b0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 shutdown_connections 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72ac0a2b90 0x7f72ac0a5050 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72c0102320 0x7f72c019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72c0101960 0x7f72c0198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.011 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 --2- 172.21.15.69:0/1788311503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72c0100770 0x7f72c0197bf0 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:24.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 >> 172.21.15.69:0/1788311503 conn(0x7f72c00fbce0 msgr2=0x7f72c00689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:24.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 shutdown_connections 2024-05-09T09:19:24.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:24.003+0000 7f72c7ea5640 1 -- 172.21.15.69:0/1788311503 wait complete. 2024-05-09T09:19:24.013 INFO:tasks.cephfs.filesystem:scrub status for tag:b545ac4c-8f31-4335-bc30-fee8ee66e7e3 - {'path': '/', 'tag': 'b545ac4c-8f31-4335-bc30-fee8ee66e7e3', 'options': 'recursive,force'} 2024-05-09T09:19:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:24 smithi069 ceph-mon[30123]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:24 smithi163 ceph-mon[34354]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:24 smithi103 ceph-mon[34421]: pgmap v380: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:25.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:25 smithi069 ceph-mon[30123]: scrub queued for path: / 2024-05-09T09:19:25.512 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:25 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:25.512 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:25 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:19:25.512 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:25 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:25.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:25 smithi163 ceph-mon[34354]: scrub queued for path: / 2024-05-09T09:19:25.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:25 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:25.560 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:25 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:19:25.561 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:25 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:25 smithi103 ceph-mon[34421]: scrub queued for path: / 2024-05-09T09:19:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:25 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:25 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:19:25.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:25 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:26.532 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:26 smithi069 ceph-mon[30123]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:26 smithi163 ceph-mon[34354]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:26.629 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:26 smithi103 ceph-mon[34421]: pgmap v381: 129 pgs: 129 active+clean; 656 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:27.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:27 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:19:27.554 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:27 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:19:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:27 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:19:28.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:28 smithi069 ceph-mon[30123]: pgmap v382: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-05-09T09:19:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:28 smithi163 ceph-mon[34354]: pgmap v382: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-05-09T09:19:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:28 smithi103 ceph-mon[34421]: pgmap v382: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.2 KiB/s wr, 8 op/s 2024-05-09T09:19:30.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:30 smithi069 ceph-mon[30123]: pgmap v383: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-09T09:19:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:30 smithi163 ceph-mon[34354]: pgmap v383: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-09T09:19:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:30 smithi103 ceph-mon[34421]: pgmap v383: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-05-09T09:19:30.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 -- 172.21.15.69:0/1315566243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4101980 msgr2=0x7f18a4101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 --2- 172.21.15.69:0/1315566243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4101980 0x7f18a4101e00 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7f188c009900 tx=0x7f188c02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 -- 172.21.15.69:0/1315566243 shutdown_connections 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 --2- 172.21.15.69:0/1315566243 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4102340 0x7f18a4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 --2- 172.21.15.69:0/1315566243 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4101980 0x7f18a4101e00 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 --2- 172.21.15.69:0/1315566243 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 0x7f18a4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 -- 172.21.15.69:0/1315566243 >> 172.21.15.69:0/1315566243 conn(0x7f18a40fbd00 msgr2=0x7f18a40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 -- 172.21.15.69:0/1315566243 shutdown_connections 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.791+0000 7f18a9ada640 1 -- 172.21.15.69:0/1315566243 wait complete. 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 Processor -- start 2024-05-09T09:19:30.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 -- start start 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 0x7f18a4069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4101980 0x7f18a419c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18a410c470 con 0x7f18a4102340 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18a410c2f0 con 0x7f18a4100790 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a9ada640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18a410c5f0 con 0x7f18a4101980 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34790/0 (socket says 172.21.15.69:34790) 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 -- 172.21.15.69:0/2861414307 learned_addr learned my addr 172.21.15.69:0/2861414307 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 0x7f18a4069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f189bfff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4101980 0x7f18a419c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 -- 172.21.15.69:0/2861414307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4101980 msgr2=0x7f18a419c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4101980 0x7f18a419c8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 -- 172.21.15.69:0/2861414307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 msgr2=0x7f18a4069a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 0x7f18a4069a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.792+0000 7f18a3fff640 1 -- 172.21.15.69:0/2861414307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f18a419f800 con 0x7f18a4102340 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a3fff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f189c00b970 tx=0x7f189c00be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a17fa640 1 -- 172.21.15.69:0/2861414307 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c019070 con 0x7f18a4102340 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a17fa640 1 -- 172.21.15.69:0/2861414307 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f189c00cb40 con 0x7f18a4102340 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f18a419fa90 con 0x7f18a4102340 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a17fa640 1 -- 172.21.15.69:0/2861414307 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189c007650 con 0x7f18a4102340 2024-05-09T09:19:30.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.793+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18a41a0030 con 0x7f18a4102340 2024-05-09T09:19:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.795+0000 7f18a17fa640 1 -- 172.21.15.69:0/2861414307 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f189c01c070 con 0x7f18a4102340 2024-05-09T09:19:30.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 Processor -- start 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 -- start start 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfae670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfb6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f79cf12340 con 0x55f79cfaec40 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f79ceba290 con 0x55f79cfb3db0 2024-05-09T09:19:30.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f79cf0e300 con 0x55f79cfae250 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a17fa640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 0x7f18900a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34792/0 (socket says 172.21.15.69:34792) 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a37fe640 1 -- 172.21.15.69:0/4243656442 learned_addr learned my addr 172.21.15.69:0/4243656442 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f189bfff640 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfb6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a3fff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 0x7f18900a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a17fa640 1 -- 172.21.15.69:0/2861414307 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f189c0ce950 con 0x7f18a4102340 2024-05-09T09:19:30.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f189bfff640 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a37fe640 1 -- 172.21.15.69:0/4243656442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 msgr2=0x55f79cfae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.796+0000 7f18a37fe640 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfae670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f18a37fe640 1 -- 172.21.15.69:0/4243656442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 msgr2=0x55f79cfb6240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f18a37fe640 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfb6240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f18a37fe640 1 -- 172.21.15.69:0/4243656442 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f79cede8e0 con 0x55f79cfaec40 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f18a37fe640 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f189400bcb0 tx=0x7f18940107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e86e2da7113967d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f18a3fff640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 0x7f18900a50f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f189c09abe0 tx=0x7f189c00e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f1899ffb640 1 -- 172.21.15.69:0/4243656442 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1894004020 con 0x55f79cfaec40 2024-05-09T09:19:30.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f1899ffb640 1 -- 172.21.15.69:0/4243656442 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1894005500 con 0x55f79cfaec40 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.797+0000 7f1899ffb640 1 -- 172.21.15.69:0/4243656442 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f18940035f0 con 0x55f79cfaec40 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- 172.21.15.69:0/4243656442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 msgr2=0x55f79cfb3740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7f189400bcb0 tx=0x7f18940107a0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- 172.21.15.69:0/4243656442 shutdown_connections 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfb6240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb3740 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/4243656442 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfae670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- 172.21.15.69:0/4243656442 >> 172.21.15.69:0/4243656442 conn(0x55f79cfa98c0 msgr2=0x55f79cfa9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- 172.21.15.69:0/4243656442 shutdown_connections 2024-05-09T09:19:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- 172.21.15.69:0/4243656442 wait complete. 2024-05-09T09:19:30.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 -- start start 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.798+0000 7f18ac7ed740 1 Processor -- start 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfbf990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f79cfb6d60 con 0x55f79cfaec40 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f79cecd790 con 0x55f79cfb3db0 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f79cfb6ee0 con 0x55f79cfae250 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18ac7ed740 10 client.0 _ll_register_callbacks cb 0x55f79cf08a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34800/0 (socket says 172.21.15.69:34800) 2024-05-09T09:19:30.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 -- 172.21.15.69:0/2357068950 learned_addr learned my addr 172.21.15.69:0/2357068950 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f189bfff640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f189bfff640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfbf990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 -- 172.21.15.69:0/2357068950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 msgr2=0x55f79cfb8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 -- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 msgr2=0x55f79cfbf990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfbf990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 -- 172.21.15.69:0/2357068950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f79cc26210 con 0x55f79cfaec40 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f189bfff640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfbf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f189401a000 tx=0x7f189400b570 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4be82030b99373fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.799+0000 7f1882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189402b060 con 0x55f79cfaec40 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f1894004d90 con 0x55f79cfaec40 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1894005500 con 0x55f79cfaec40 2024-05-09T09:19:30.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1894032050 con 0x55f79cfaec40 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 10 client.15774 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f79ccf2620 con 0x55f79cfaec40 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1894005500 con 0x55f79cfaec40 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f1894011bd0 con 0x55f79cfaec40 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 10 client.15774 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 10 client.15774 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 20 client.15774 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfc3c00 0x55f79cfc5ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 4 client.15774 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.800+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f79cf1fa70 con 0x55f79cfc3c00 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.801+0000 7f189bfff640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfc3c00 0x55f79cfc5ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.801+0000 7f1882ffd640 10 client.15774 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.801+0000 7f189bfff640 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfc3c00 0x55f79cfc5ff0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bae1b68285a6561b server_cookie=f732c716f0922159 in_seq=0 out_seq=0 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f1882ffd640 1 -- 172.21.15.69:0/2357068950 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f79cf1fa70 con 0x55f79cfc3c00 2024-05-09T09:19:30.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f1882ffd640 10 client.15774 handle_command_reply: tid=0 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 client.15774 shutdown 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 msgr2=0x55f79cfb8950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7f189401a000 tx=0x7f189400b570 comp rx=0 tx=0).stop 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 shutdown_connections 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfb3db0 0x55f79cfbf990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfb8950 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfc3c00 0x55f79cfc5ff0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2357068950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f1882ffd640 0 client.15774 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 >> 172.21.15.69:0/2357068950 conn(0x55f79cfa98c0 msgr2=0x55f79cfa8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 shutdown_connections 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2357068950 wait complete. 2024-05-09T09:19:30.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.802+0000 7f18ac7ed740 20 client.15774 trim_cache size 0 max 16384 2024-05-09T09:19:30.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 Processor -- start 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 -- start start 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfb8410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41acd60 con 0x55f79cfaec40 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41aa630 con 0x55f79cfa98c0 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.820+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41a8480 con 0x55f79cfae250 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34804/0 (socket says 172.21.15.69:34804) 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 -- 172.21.15.69:0/1813776389 learned_addr learned my addr 172.21.15.69:0/1813776389 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfb8410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f18a37fe640 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 -- 172.21.15.69:0/1813776389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 msgr2=0x55f79cfb8950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 -- 172.21.15.69:0/1813776389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 msgr2=0x55f79cfb8410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfb8410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 -- 172.21.15.69:0/1813776389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f79cfaf020 con 0x55f79cfaec40 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f189bfff640 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x55f79cfb7500 tx=0x7f188c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4933aad5995bf3e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f18997fa640 1 -- 172.21.15.69:0/1813776389 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c035040 con 0x55f79cfaec40 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f18997fa640 1 -- 172.21.15.69:0/1813776389 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c04c3c0 con 0x55f79cfaec40 2024-05-09T09:19:30.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.821+0000 7f18997fa640 1 -- 172.21.15.69:0/1813776389 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f188c004ce0 con 0x55f79cfaec40 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 -- 172.21.15.69:0/1813776389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 msgr2=0x55f79cfbf950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x55f79cfb7500 tx=0x7f188c000bc0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 -- 172.21.15.69:0/1813776389 shutdown_connections 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfaec40 0x55f79cfbf950 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfae250 0x55f79cfb8950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/1813776389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfb8410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 -- 172.21.15.69:0/1813776389 >> 172.21.15.69:0/1813776389 conn(0x55f79cfb7620 msgr2=0x55f79cfb3420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 -- 172.21.15.69:0/1813776389 shutdown_connections 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.822+0000 7f18ac7ed740 1 -- 172.21.15.69:0/1813776389 wait complete. 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 -- start start 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 Processor -- start 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfbca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 0x55f79cfbdf70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41a8af0 con 0x55f79cfae250 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41a9f60 con 0x55f79cfa98c0 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18a41a91c0 con 0x55f79cfaec40 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18ac7ed740 10 client.0 _ll_register_callbacks cb 0x7f189400bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18a37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34818/0 (socket says 172.21.15.69:34818) 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f18a37fe640 1 -- 172.21.15.69:0/2623396971 learned_addr learned my addr 172.21.15.69:0/2623396971 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:30.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.823+0000 7f189bfff640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 0x55f79cfbdf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f189bfff640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfbca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 -- 172.21.15.69:0/2623396971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 msgr2=0x55f79cfbdf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 0x55f79cfbdf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 -- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 msgr2=0x55f79cfbca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfbca20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 -- 172.21.15.69:0/2623396971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f79cfbe560 con 0x55f79cfae250 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f189bfff640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 0x55f79cfbdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18a37fe640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f18940107e0 tx=0x7f1894004b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b126016ea7bab857 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f189402b070 con 0x55f79cfae250 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f189401bac0 con 0x55f79cfae250 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f188c04c3c0 con 0x55f79cfae250 2024-05-09T09:19:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f1894005b90 con 0x55f79cfae250 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18ac7ed740 10 client.15780 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55f79cfbf870 con 0x55f79cfae250 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f188c04c3c0 con 0x55f79cfae250 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.824+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f18940039c0 con 0x55f79cfae250 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 10 client.15780 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 10 client.15780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 20 client.15780 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfb3db0 0x55f79d02c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 4 client.15780 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f79cf19dc0 con 0x55f79cfb3db0 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f189bfff640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfb3db0 0x55f79d02c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f189bfff640 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfb3db0 0x55f79d02c480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eaf0e57e2f15938c server_cookie=67fa3ae41a7964ce in_seq=0 out_seq=0 2024-05-09T09:19:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.825+0000 7f1882ffd640 10 client.15780 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f1882ffd640 1 -- 172.21.15.69:0/2623396971 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f79cf19dc0 con 0x55f79cfb3db0 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f1882ffd640 10 client.15780 handle_command_reply: tid=0 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 client.15780 shutdown 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 msgr2=0x55f79cfbcf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7f18940107e0 tx=0x7f1894004b00 comp rx=0 tx=0).stop 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 shutdown_connections 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f79cfb3db0 0x55f79d02c480 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f79cfaec40 0x55f79cfbdf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f79cfae250 0x55f79cfbcf60 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f1882ffd640 0 client.15780 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 --2- 172.21.15.69:0/2623396971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f79cfa98c0 0x55f79cfbca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 >> 172.21.15.69:0/2623396971 conn(0x55f79cfb7620 msgr2=0x55f79cfb2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 shutdown_connections 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 1 -- 172.21.15.69:0/2623396971 wait complete. 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.833+0000 7f18ac7ed740 20 client.15780 trim_cache size 0 max 16384 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:19:30.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:19:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 msgr2=0x7f18900a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 0x7f18900a50f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f189c09abe0 tx=0x7f189c00e040 comp rx=0 tx=0).stop 2024-05-09T09:19:30.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 msgr2=0x7f18a419eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7f189c00b970 tx=0x7f189c00be30 comp rx=0 tx=0).stop 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 shutdown_connections 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f18900a2c30 0x7f18900a50f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f18a4102340 0x7f18a419eef0 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f18a4101980 0x7f18a419c8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 --2- 172.21.15.69:0/2861414307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f18a4100790 0x7f18a4069a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 >> 172.21.15.69:0/2861414307 conn(0x7f18a40fbd00 msgr2=0x7f18a4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 shutdown_connections 2024-05-09T09:19:30.835 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:30.834+0000 7f18a9ada640 1 -- 172.21.15.69:0/2861414307 wait complete. 2024-05-09T09:19:32.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:32 smithi069 ceph-mon[30123]: pgmap v384: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T09:19:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:32 smithi163 ceph-mon[34354]: pgmap v384: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T09:19:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:32 smithi103 ceph-mon[34421]: pgmap v384: 129 pgs: 129 active+clean; 666 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 1.2 KiB/s wr, 6 op/s 2024-05-09T09:19:34.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:34 smithi069 ceph-mon[30123]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:34 smithi163 ceph-mon[34354]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:34 smithi103 ceph-mon[34421]: pgmap v385: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:35 smithi163 ceph-mon[34354]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-09T09:19:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:35 smithi103 ceph-mon[34421]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-09T09:19:35.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:35 smithi069 ceph-mon[30123]: pgmap v386: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 1.5 KiB/s wr, 7 op/s 2024-05-09T09:19:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:38 smithi069 ceph-mon[30123]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:38 smithi163 ceph-mon[34354]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:38 smithi103 ceph-mon[34421]: pgmap v387: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1.7 KiB/s wr, 9 op/s 2024-05-09T09:19:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:40 smithi069 ceph-mon[30123]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-09T09:19:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:40 smithi163 ceph-mon[34354]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-09T09:19:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:40 smithi103 ceph-mon[34421]: pgmap v388: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 767 B/s wr, 7 op/s 2024-05-09T09:19:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:42 smithi069 ceph-mon[30123]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:19:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:42 smithi163 ceph-mon[34354]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:19:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:42 smithi103 ceph-mon[34421]: pgmap v389: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 -- 172.21.15.69:0/3531926604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4101980 msgr2=0x7f53b4101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 --2- 172.21.15.69:0/3531926604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4101980 0x7f53b4101e00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009900 tx=0x7f53a002a410 comp rx=0 tx=0).stop 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 -- 172.21.15.69:0/3531926604 shutdown_connections 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 --2- 172.21.15.69:0/3531926604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4102340 0x7f53b4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 --2- 172.21.15.69:0/3531926604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4101980 0x7f53b4101e00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 --2- 172.21.15.69:0/3531926604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 0x7f53b4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 -- 172.21.15.69:0/3531926604 >> 172.21.15.69:0/3531926604 conn(0x7f53b40fbd00 msgr2=0x7f53b40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 -- 172.21.15.69:0/3531926604 shutdown_connections 2024-05-09T09:19:43.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.967+0000 7f53baefc640 1 -- 172.21.15.69:0/3531926604 wait complete. 2024-05-09T09:19:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 Processor -- start 2024-05-09T09:19:43.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 -- start start 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 0x7f53b4197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4101980 0x7f53b4197ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c440 con 0x7f53b4102340 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c2c0 con 0x7f53b4100790 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53baefc640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53b410c5c0 con 0x7f53b4101980 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53b9499640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53b9499640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47842/0 (socket says 172.21.15.69:47842) 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53b9499640 1 -- 172.21.15.69:0/3306667319 learned_addr learned my addr 172.21.15.69:0/3306667319 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53b8c98640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 0x7f53b4197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.968+0000 7f53b3fff640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4101980 0x7f53b4197ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 -- 172.21.15.69:0/3306667319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4101980 msgr2=0x7f53b4197ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:43.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4101980 0x7f53b4197ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 -- 172.21.15.69:0/3306667319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 msgr2=0x7f53b4197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 0x7f53b4197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 -- 172.21.15.69:0/3306667319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f53b4102c20 con 0x7f53b4102340 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b9499640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009fd0 tx=0x7f53ac00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b1ffb640 1 -- 172.21.15.69:0/3306667319 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac017070 con 0x7f53b4102340 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b1ffb640 1 -- 172.21.15.69:0/3306667319 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53ac003e50 con 0x7f53b4102340 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f53b4102f10 con 0x7f53b4102340 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53b1ffb640 1 -- 172.21.15.69:0/3306667319 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53ac0106c0 con 0x7f53b4102340 2024-05-09T09:19:43.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.969+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f53b4103430 con 0x7f53b4102340 2024-05-09T09:19:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 Processor -- start 2024-05-09T09:19:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53b1ffb640 1 -- 172.21.15.69:0/3306667319 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f53ac010860 con 0x7f53b4102340 2024-05-09T09:19:43.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 -- start start 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e77670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 0x560136e7c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e7cdb0 0x560136e7f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560136ddb340 con 0x560136e7cdb0 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560136d83290 con 0x560136e77250 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.973+0000 7f53bdc09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560136dd7300 con 0x560136e77c40 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e77670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e77670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36458/0 (socket says 172.21.15.69:36458) 2024-05-09T09:19:43.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b3fff640 1 -- 172.21.15.69:0/1012033027 learned_addr learned my addr 172.21.15.69:0/1012033027 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 0x560136e7c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e7cdb0 0x560136e7f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 -- 172.21.15.69:0/1012033027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 msgr2=0x560136e77670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e77670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 -- 172.21.15.69:0/1012033027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e7cdb0 msgr2=0x560136e7f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e7cdb0 0x560136e7f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 -- 172.21.15.69:0/1012033027 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560136da78e0 con 0x560136e77c40 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 0x560136e7c740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f53a800bbc0 tx=0x7f53a80107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a27e9e5bc3a54cd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f53b1ffb640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 0x7f53a40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.974+0000 7f539effd640 1 -- 172.21.15.69:0/1012033027 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8004030 con 0x560136e77c40 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.975+0000 7f539effd640 1 -- 172.21.15.69:0/1012033027 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8005500 con 0x560136e77c40 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.975+0000 7f53b3fff640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 0x7f53a40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.975+0000 7f539effd640 1 -- 172.21.15.69:0/1012033027 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a8004ce0 con 0x560136e77c40 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.975+0000 7f53b1ffb640 1 -- 172.21.15.69:0/3306667319 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f53ac0cf0a0 con 0x7f53b4102340 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53b3fff640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 0x7f53a40a5170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a00315b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:43.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 -- 172.21.15.69:0/1012033027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 msgr2=0x560136e7c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 0x560136e7c740 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f53a800bbc0 tx=0x7f53a80107a0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 -- 172.21.15.69:0/1012033027 shutdown_connections 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e7cdb0 0x560136e7f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77c40 0x560136e7c740 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1012033027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e77670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 -- 172.21.15.69:0/1012033027 >> 172.21.15.69:0/1012033027 conn(0x560136e728c0 msgr2=0x560136e72cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 -- 172.21.15.69:0/1012033027 shutdown_connections 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.976+0000 7f53bdc09740 1 -- 172.21.15.69:0/1012033027 wait complete. 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.977+0000 7f53bdc09740 1 -- start start 2024-05-09T09:19:43.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.977+0000 7f53bdc09740 1 Processor -- start 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e814c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e88a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560136e7fd60 con 0x560136e77c40 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560136d96790 con 0x560136e77250 2024-05-09T09:19:43.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47850/0 (socket says 172.21.15.69:47850) 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 -- 172.21.15.69:0/1942866743 learned_addr learned my addr 172.21.15.69:0/1942866743 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560136e7fee0 con 0x560136e7cdb0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e814c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53bdc09740 10 client.0 _ll_register_callbacks cb 0x560136dd1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b9499640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e88a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 -- 172.21.15.69:0/1942866743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 msgr2=0x560136e88a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e88a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 -- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 msgr2=0x560136e814c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e814c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b3fff640 1 -- 172.21.15.69:0/1942866743 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560136aef210 con 0x560136e77c40 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.978+0000 7f53b9499640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e88a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009230 tx=0x7f53a002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74c3b261e4db7658 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:43.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f5393fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00037f0 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a8005500 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f53a0003990 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f53a0003b90 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f53bdc09740 10 client.15792 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.979+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560136bbb620 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a8005500 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f53a003c450 con 0x560136e77c40 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 10 client.15792 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 10 client.15792 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 20 client.15792 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136e8d440 0x560136e8f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 4 client.15792 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:43.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.980+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560136de8a70 con 0x560136e8d440 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.981+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136e8d440 0x560136e8f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.981+0000 7f5393fff640 10 client.15792 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.981+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136e8d440 0x560136e8f8e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90c90c2bbb5a2721 server_cookie=1c052bb7bede2995 in_seq=0 out_seq=0 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f5393fff640 1 -- 172.21.15.69:0/1942866743 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560136de8a70 con 0x560136e8d440 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f5393fff640 10 client.15792 handle_command_reply: tid=0 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 client.15792 shutdown 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 msgr2=0x560136e81a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7f53a0009230 tx=0x7f53a002a410 comp rx=0 tx=0).stop 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 shutdown_connections 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136e8d440 0x560136e8f8e0 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e88a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e81a00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.986 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1942866743 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e814c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 >> 172.21.15.69:0/1942866743 conn(0x560136e728c0 msgr2=0x560136e71a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.982+0000 7f5393fff640 0 client.15792 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.983+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 shutdown_connections 2024-05-09T09:19:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.983+0000 7f53bdc09740 1 -- 172.21.15.69:0/1942866743 wait complete. 2024-05-09T09:19:43.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:43.983+0000 7f53bdc09740 20 client.15792 trim_cache size 0 max 16384 2024-05-09T09:19:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 Processor -- start 2024-05-09T09:19:44.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 -- start start 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e7c480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e951b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41aee90 con 0x560136e77c40 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ac760 con 0x560136e77250 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53bdc09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41aa5b0 con 0x560136e7cdb0 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47858/0 (socket says 172.21.15.69:47858) 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.011+0000 7f53b3fff640 1 -- 172.21.15.69:0/1750884591 learned_addr learned my addr 172.21.15.69:0/1750884591 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e7c480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b9499640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e951b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 -- 172.21.15.69:0/1750884591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 msgr2=0x560136e951b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:44.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e951b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 -- 172.21.15.69:0/1750884591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 msgr2=0x560136e7c480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e7c480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 -- 172.21.15.69:0/1750884591 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560136e70f00 con 0x560136e77c40 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b9499640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b8c98640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f53b3fff640 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53a00096c0 tx=0x7f53a00319a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e19b7346c2f01f6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f539e7fc640 1 -- 172.21.15.69:0/1750884591 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a0003d30 con 0x560136e77c40 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f539e7fc640 1 -- 172.21.15.69:0/1750884591 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0066070 con 0x560136e77c40 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.012+0000 7f539e7fc640 1 -- 172.21.15.69:0/1750884591 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a00616b0 con 0x560136e77c40 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 -- 172.21.15.69:0/1750884591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 msgr2=0x560136e88a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7f53a00096c0 tx=0x7f53a00319a0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 -- 172.21.15.69:0/1750884591 shutdown_connections 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e7cdb0 0x560136e951b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e88a00 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 --2- 172.21.15.69:0/1750884591 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e77250 0x560136e7c480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 -- 172.21.15.69:0/1750884591 >> 172.21.15.69:0/1750884591 conn(0x560136e728c0 msgr2=0x560136e72ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 -- 172.21.15.69:0/1750884591 shutdown_connections 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.013+0000 7f53bdc09740 1 -- 172.21.15.69:0/1750884591 wait complete. 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 -- start start 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 Processor -- start 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77250 0x560136e88910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e7cdb0 0x560136e85a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41aac20 con 0x560136e77c40 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ac090 con 0x560136e7cdb0 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f53b41ab2f0 con 0x560136e77250 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53bdc09740 10 client.0 _ll_register_callbacks cb 0x7f53a0009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53b3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47864/0 (socket says 172.21.15.69:47864) 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53b3fff640 1 -- 172.21.15.69:0/2237331287 learned_addr learned my addr 172.21.15.69:0/2237331287 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53b9499640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e7cdb0 0x560136e85a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.014+0000 7f53b8c98640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77250 0x560136e88910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 -- 172.21.15.69:0/2237331287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77250 msgr2=0x560136e88910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77250 0x560136e88910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 -- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e7cdb0 msgr2=0x560136e85a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e7cdb0 0x560136e85a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 -- 172.21.15.69:0/2237331287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560136ef1a10 con 0x560136e77c40 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53b3fff640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f53a002a930 tx=0x7f53a0031730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6cee178f53184ce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f5393fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f53a006f060 con 0x560136e77c40 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f53a0066070 con 0x560136e77c40 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f53a003cc80 con 0x560136e77c40 2024-05-09T09:19:44.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f53a00035f0 con 0x560136e77c40 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53bdc09740 10 client.15798 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x560136ef2bb0 con 0x560136e77c40 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.015+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f53a0066070 con 0x560136e77c40 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f53a0004a40 con 0x560136e77c40 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 10 client.15798 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 10 client.15798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 20 client.15798 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136ef3500 0x560136ef5920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 4 client.15798 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560136de2dc0 con 0x560136ef3500 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.016+0000 7f53b8c98640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136ef3500 0x560136ef5920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.017+0000 7f53b8c98640 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136ef3500 0x560136ef5920 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e3f46429f41063c server_cookie=bfd19b64351e2a80 in_seq=0 out_seq=0 2024-05-09T09:19:44.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.017+0000 7f5393fff640 10 client.15798 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.020+0000 7f5393fff640 1 -- 172.21.15.69:0/2237331287 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560136de2dc0 con 0x560136ef3500 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.020+0000 7f5393fff640 10 client.15798 handle_command_reply: tid=0 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.020+0000 7f53bdc09740 1 client.15798 shutdown 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 msgr2=0x560136e854c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f53a002a930 tx=0x7f53a0031730 comp rx=0 tx=0).stop 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 shutdown_connections 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560136e7cdb0 0x560136e85a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560136e77c40 0x560136e854c0 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560136ef3500 0x560136ef5920 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 --2- 172.21.15.69:0/2237331287 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560136e77250 0x560136e88910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f5393fff640 0 client.15798 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 >> 172.21.15.69:0/2237331287 conn(0x560136e728c0 msgr2=0x560136e8f450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 shutdown_connections 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 1 -- 172.21.15.69:0/2237331287 wait complete. 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.021+0000 7f53bdc09740 20 client.15798 trim_cache size 0 max 16384 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:19:44.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:19:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:19:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 msgr2=0x7f53a40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:44.025 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 0x7f53a40a5170 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f53a0000c00 tx=0x7f53a00315b0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 msgr2=0x7f53b419f060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7f53ac009fd0 tx=0x7f53ac00b260 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 shutdown_connections 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f53a40a2cb0 0x7f53a40a5170 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f53b4102340 0x7f53b419f060 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f53b4101980 0x7f53b4197ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 --2- 172.21.15.69:0/3306667319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f53b4100790 0x7f53b4197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 >> 172.21.15.69:0/3306667319 conn(0x7f53b40fbd00 msgr2=0x7f53b40fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 shutdown_connections 2024-05-09T09:19:44.026 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:44.022+0000 7f53baefc640 1 -- 172.21.15.69:0/3306667319 wait complete. 2024-05-09T09:19:44.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:44 smithi069 ceph-mon[30123]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-05-09T09:19:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:44 smithi163 ceph-mon[34354]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-05-09T09:19:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:44 smithi103 ceph-mon[34421]: pgmap v390: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 767 B/s wr, 8 op/s 2024-05-09T09:19:46.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:46 smithi069 ceph-mon[30123]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:46 smithi163 ceph-mon[34354]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:46 smithi103 ceph-mon[34421]: pgmap v391: 129 pgs: 129 active+clean; 667 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:48.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:48 smithi069 ceph-mon[30123]: pgmap v392: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:48 smithi163 ceph-mon[34354]: pgmap v392: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:48 smithi103 ceph-mon[34421]: pgmap v392: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 426 B/s wr, 7 op/s 2024-05-09T09:19:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:50 smithi069 ceph-mon[30123]: pgmap v393: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:50 smithi163 ceph-mon[34354]: pgmap v393: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:50 smithi103 ceph-mon[34421]: pgmap v393: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:19:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:52 smithi069 ceph-mon[30123]: pgmap v394: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:52 smithi163 ceph-mon[34354]: pgmap v394: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:52 smithi103 ceph-mon[34421]: pgmap v394: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:54.013 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:19:54.014 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 -- 172.21.15.69:0/1731320780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c100770 msgr2=0x7f419c100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/1731320780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c100770 0x7f419c100b90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f4198009900 tx=0x7f419802a410 comp rx=0 tx=0).stop 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 -- 172.21.15.69:0/1731320780 shutdown_connections 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/1731320780 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c102320 0x7f419c108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/1731320780 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c101960 0x7f419c101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/1731320780 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c100770 0x7f419c100b90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 -- 172.21.15.69:0/1731320780 >> 172.21.15.69:0/1731320780 conn(0x7f419c0fbce0 msgr2=0x7f419c0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.175 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 -- 172.21.15.69:0/1731320780 shutdown_connections 2024-05-09T09:19:54.176 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.175+0000 7f41a3ffb640 1 -- 172.21.15.69:0/1731320780 wait complete. 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 Processor -- start 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 -- start start 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 0x7f419c197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 0x7f419c198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c102320 0x7f419c19f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10d510 con 0x7f419c101960 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10d390 con 0x7f419c100770 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a3ffb640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f419c10d690 con 0x7f419c102320 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a1596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 0x7f419c198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.177 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a1596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 0x7f419c198170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51244/0 (socket says 172.21.15.69:51244) 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a1d97640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 0x7f419c197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a1596640 1 -- 172.21.15.69:0/2612159872 learned_addr learned my addr 172.21.15.69:0/2612159872 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.176+0000 7f41a2598640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c102320 0x7f419c19f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 -- 172.21.15.69:0/2612159872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c102320 msgr2=0x7f419c19f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c102320 0x7f419c19f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 -- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 msgr2=0x7f419c198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 0x7f419c198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 -- 172.21.15.69:0/2612159872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f419c109470 con 0x7f419c100770 2024-05-09T09:19:54.178 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a1d97640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 0x7f419c197c30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4198009230 tx=0x7f4198004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f4192ffd640 1 -- 172.21.15.69:0/2612159872 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f419803f070 con 0x7f419c100770 2024-05-09T09:19:54.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f419c109700 con 0x7f419c100770 2024-05-09T09:19:54.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f4192ffd640 1 -- 172.21.15.69:0/2612159872 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4198003cd0 con 0x7f419c100770 2024-05-09T09:19:54.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f4192ffd640 1 -- 172.21.15.69:0/2612159872 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4198035070 con 0x7f419c100770 2024-05-09T09:19:54.179 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.177+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f419c109bc0 con 0x7f419c100770 2024-05-09T09:19:54.180 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 Processor -- start 2024-05-09T09:19:54.181 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 -- start start 2024-05-09T09:19:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7e5420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7e7f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.182 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55efdb744d00 con 0x55efdb7e0920 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efdb6ec4c0 con 0x55efdb7dff30 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.181+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55efdb740cc0 con 0x55efdb7e5a90 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:50218/0 (socket says 172.21.15.69:50218) 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 -- 172.21.15.69:0/3357736974 learned_addr learned my addr 172.21.15.69:0/3357736974 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a1596640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7e7f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f4192ffd640 1 -- 172.21.15.69:0/2612159872 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f419803b460 con 0x7f419c100770 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 -- 172.21.15.69:0/3357736974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 msgr2=0x55efdb7e7f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7e7f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 -- 172.21.15.69:0/3357736974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 msgr2=0x55efdb7e5420 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a1d97640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7e5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7e5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a2598640 1 -- 172.21.15.69:0/3357736974 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efdb5227d0 con 0x55efdb7dff30 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.182+0000 7f41a1d97640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:54.183 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f41a2598640 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4194009fd0 tx=0x7f41940118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4bca6a627d2dabf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f4187fff640 1 -- 172.21.15.69:0/3357736974 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194004d20 con 0x55efdb7dff30 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f4187fff640 1 -- 172.21.15.69:0/3357736974 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4194004050 con 0x55efdb7dff30 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f4187fff640 1 -- 172.21.15.69:0/3357736974 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194005240 con 0x55efdb7dff30 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f4192ffd640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 0x7f41880a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.183+0000 7f41a1d97640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 0x7f41880a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f4192ffd640 1 -- 172.21.15.69:0/2612159872 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f41980ef840 con 0x7f419c100770 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a1d97640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 0x7f41880a5050 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41980ed330 tx=0x7f41980f4040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3357736974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 msgr2=0x55efdb7e0350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f4194009fd0 tx=0x7f41940118d0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3357736974 shutdown_connections 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7e7f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7e5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3357736974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7e0350 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3357736974 >> 172.21.15.69:0/3357736974 conn(0x55efdb7db5a0 msgr2=0x55efdb7db9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3357736974 shutdown_connections 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.184+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3357736974 wait complete. 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 -- start start 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 Processor -- start 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 0x55efdb7ea8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7f1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55efdb7e8a40 con 0x55efdb7dff30 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a2598640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.193 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a2598640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51252/0 (socket says 172.21.15.69:51252) 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.185+0000 7f41a2598640 1 -- 172.21.15.69:0/1661141317 learned_addr learned my addr 172.21.15.69:0/1661141317 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55efdb6ffb90 con 0x55efdb7e0920 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55efdb7e8bc0 con 0x55efdb7e5a90 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a1596640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 0x55efdb7ea8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a6ddf740 10 client.0 _ll_register_callbacks cb 0x55efdb73bda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 -- 172.21.15.69:0/1661141317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 msgr2=0x55efdb7f1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7f1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 -- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 msgr2=0x55efdb7ea8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 0x55efdb7ea8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efdb4583f0 con 0x55efdb7dff30 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a1596640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 0x55efdb7ea8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a2598640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f419401b040 tx=0x7f41940044a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5294d17206333c42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41857fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:54.194 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4194004050 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.186+0000 7f41857fa640 1 -- 172.21.15.69:0/1661141317 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4194028020 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41857fa640 1 -- 172.21.15.69:0/1661141317 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f4194004c50 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41857fa640 1 -- 172.21.15.69:0/1661141317 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f4194033050 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 10 client.15807 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55efdb4180f0 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41857fa640 1 -- 172.21.15.69:0/1661141317 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f4194005040 con 0x55efdb7dff30 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 10 client.15807 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 10 client.15807 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 20 client.15807 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb7f8140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 4 client.15807 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.187+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f4194004050 con 0x55efdb7f5cd0 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.188+0000 7f41a1596640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb7f8140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.188+0000 7f41857fa640 10 client.15807 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.188+0000 7f41a1596640 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb7f8140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=49528233ea7b90d7 server_cookie=85bae6d2cc26c510 in_seq=0 out_seq=0 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41857fa640 1 -- 172.21.15.69:0/1661141317 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f4194004050 con 0x55efdb7f5cd0 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41857fa640 10 client.15807 handle_command_reply: tid=0 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 client.15807 shutdown 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 msgr2=0x55efdb7ea390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7f419401b040 tx=0x7f41940044a0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 shutdown_connections 2024-05-09T09:19:54.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb7f8140 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41857fa640 0 client.15807 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e5a90 0x55efdb7f1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e0920 0x55efdb7ea8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/1661141317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7dff30 0x55efdb7ea390 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 >> 172.21.15.69:0/1661141317 conn(0x55efdb7db5a0 msgr2=0x55efdb7da7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 shutdown_connections 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 1 -- 172.21.15.69:0/1661141317 wait complete. 2024-05-09T09:19:54.197 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.189+0000 7f41a6ddf740 20 client.15807 trim_cache size 0 max 16384 2024-05-09T09:19:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 Processor -- start 2024-05-09T09:19:54.224 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 -- start start 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7ea830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 0x55efdb7f18a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1aeeb0 con 0x55efdb7e5a90 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1ac780 con 0x55efdb7dff30 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1aa5d0 con 0x55efdb7e0920 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a2598640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a1596640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 0x55efdb7f18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a2598640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51268/0 (socket says 172.21.15.69:51268) 2024-05-09T09:19:54.225 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.224+0000 7f41a2598640 1 -- 172.21.15.69:0/3766749835 learned_addr learned my addr 172.21.15.69:0/3766749835 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 -- 172.21.15.69:0/3766749835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 msgr2=0x55efdb7ea830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7ea830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 -- 172.21.15.69:0/3766749835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 msgr2=0x55efdb7e96a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 -- 172.21.15.69:0/3766749835 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55efdb7e8920 con 0x55efdb7e0920 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a2598640 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:54.226 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.225+0000 7f41a1596640 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 0x55efdb7f18a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55efdb7f8920 tx=0x7f418c005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20faab794e35211a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.230+0000 7f41877fe640 1 -- 172.21.15.69:0/3766749835 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c00f8b0 con 0x55efdb7e0920 2024-05-09T09:19:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.230+0000 7f41877fe640 1 -- 172.21.15.69:0/3766749835 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f418c005020 con 0x55efdb7e0920 2024-05-09T09:19:54.230 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.230+0000 7f41877fe640 1 -- 172.21.15.69:0/3766749835 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c004a00 con 0x55efdb7e0920 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3766749835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 msgr2=0x55efdb7f18a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 0x55efdb7f18a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x55efdb7f8920 tx=0x7f418c005f90 comp rx=0 tx=0).stop 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3766749835 shutdown_connections 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e5a90 0x55efdb7e96a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7e0920 0x55efdb7f18a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3766749835 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7dff30 0x55efdb7ea830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3766749835 >> 172.21.15.69:0/3766749835 conn(0x55efdb7db5a0 msgr2=0x55efdb7e8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.231 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3766749835 shutdown_connections 2024-05-09T09:19:54.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3766749835 wait complete. 2024-05-09T09:19:54.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 -- start start 2024-05-09T09:19:54.232 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.231+0000 7f41a6ddf740 1 Processor -- start 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7dff30 0x55efdb7ee840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e5a90 0x55efdb85a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1aac40 con 0x55efdb7e0920 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1ac0b0 con 0x55efdb7e5a90 2024-05-09T09:19:54.235 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f419c1ab310 con 0x55efdb7dff30 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a1596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a1596640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51276/0 (socket says 172.21.15.69:51276) 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a1596640 1 -- 172.21.15.69:0/3465132575 learned_addr learned my addr 172.21.15.69:0/3465132575 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a6ddf740 10 client.0 _ll_register_callbacks cb 0x7f418c009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a2598640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7dff30 0x55efdb7ee840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.232+0000 7f41a2598640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e5a90 0x55efdb85a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 -- 172.21.15.69:0/3465132575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7dff30 msgr2=0x55efdb7ee840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7dff30 0x55efdb7ee840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 -- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e5a90 msgr2=0x55efdb85a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e5a90 0x55efdb85a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 -- 172.21.15.69:0/3465132575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55efdb85a780 con 0x55efdb7e0920 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a1596640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f418c000c00 tx=0x7f418c012160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f42db487f8b473d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41857fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:54.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f418c005450 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f418c005020 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f418c0055f0 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f418c03c050 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a6ddf740 10 client.15813 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.233+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55efdb85ba40 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f418c005020 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f418c031020 con 0x55efdb7e0920 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 10 client.15813 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 20 client.15813 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb85e390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 4 client.15813 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55efdb752430 con 0x55efdb7f5cd0 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41a2598640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb85e390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.234+0000 7f41857fa640 10 client.15813 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a2598640 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb85e390 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a96d648c7415b118 server_cookie=968b864e6a116428 in_seq=0 out_seq=0 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41857fa640 1 -- 172.21.15.69:0/3465132575 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55efdb752430 con 0x55efdb7f5cd0 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41857fa640 10 client.15813 handle_command_reply: tid=0 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 client.15813 shutdown 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 msgr2=0x55efdb7eed80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7f418c000c00 tx=0x7f418c012160 comp rx=0 tx=0).stop 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 shutdown_connections 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55efdb7e5a90 0x55efdb85a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55efdb7e0920 0x55efdb7eed80 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55efdb7f5cd0 0x55efdb85e390 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41857fa640 0 client.15813 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 --2- 172.21.15.69:0/3465132575 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55efdb7dff30 0x55efdb7ee840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 >> 172.21.15.69:0/3465132575 conn(0x55efdb7db5a0 msgr2=0x55efdb7de4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 shutdown_connections 2024-05-09T09:19:54.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 1 -- 172.21.15.69:0/3465132575 wait complete. 2024-05-09T09:19:54.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.235+0000 7f41a6ddf740 20 client.15813 trim_cache size 0 max 16384 2024-05-09T09:19:54.239 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:54.239 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-05-09T09:19:54.239 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-05-09T09:19:54.239 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 msgr2=0x7f41880a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 0x7f41880a5050 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f41980ed330 tx=0x7f41980f4040 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 msgr2=0x7f419c197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 0x7f419c197c30 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f4198009230 tx=0x7f4198004270 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 shutdown_connections 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f41880a2b90 0x7f41880a5050 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f419c102320 0x7f419c19f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f419c101960 0x7f419c198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 --2- 172.21.15.69:0/2612159872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f419c100770 0x7f419c197c30 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 >> 172.21.15.69:0/2612159872 conn(0x7f419c0fbce0 msgr2=0x7f419c104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 shutdown_connections 2024-05-09T09:19:54.240 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.236+0000 7f41a3ffb640 1 -- 172.21.15.69:0/2612159872 wait complete. 2024-05-09T09:19:54.247 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:19:54.248 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 -- 172.21.15.69:0/3788778827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c102a70 msgr2=0x7f620c102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 --2- 172.21.15.69:0/3788778827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c102a70 0x7f620c102ef0 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x7f61fc009900 tx=0x7f61fc02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 -- 172.21.15.69:0/3788778827 shutdown_connections 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 --2- 172.21.15.69:0/3788778827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c103430 0x7f620c109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 --2- 172.21.15.69:0/3788778827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c102a70 0x7f620c102ef0 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 --2- 172.21.15.69:0/3788778827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.387+0000 7f6213396640 1 -- 172.21.15.69:0/3788778827 >> 172.21.15.69:0/3788778827 conn(0x7f620c0fce30 msgr2=0x7f620c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 -- 172.21.15.69:0/3788778827 shutdown_connections 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 -- 172.21.15.69:0/3788778827 wait complete. 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 Processor -- start 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 -- start start 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c199dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c102a70 0x7f620c19a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.388+0000 7f6213396640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6213396640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10d4e0 con 0x7f620c103430 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6213396640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10d360 con 0x7f620c101880 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51302/0 (socket says 172.21.15.69:51302) 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 -- 172.21.15.69:0/2697695765 learned_addr learned my addr 172.21.15.69:0/2697695765 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.391 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211132640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c199dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f620c10d660 con 0x7f620c102a70 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6210931640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c102a70 0x7f620c19a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 -- 172.21.15.69:0/2697695765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c102a70 msgr2=0x7f620c19a300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c102a70 0x7f620c19a300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 -- 172.21.15.69:0/2697695765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 msgr2=0x7f620c199dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c199dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f620c1a19e0 con 0x7f620c103430 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211132640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6211933640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f620400eb10 tx=0x7f620400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204004b20 con 0x7f620c103430 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6204004cc0 con 0x7f620c103430 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.389+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f620c1a1cd0 con 0x7f620c103430 2024-05-09T09:19:54.392 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.390+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6204010760 con 0x7f620c103430 2024-05-09T09:19:54.394 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.390+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f620c1a21a0 con 0x7f620c103430 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.391+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f620c068aa0 con 0x7f620c103430 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.392+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6204005000 con 0x7f620c103430 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.393+0000 7f62027fc640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 0x7f61f80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.393+0000 7f6211132640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 0x7f61f80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.393+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f62040d0070 con 0x7f620c103430 2024-05-09T09:19:54.395 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.394+0000 7f6211132640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 0x7f61f80a5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f620c1028d0 tx=0x7f6208005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.398 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.398+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f620409cc50 con 0x7f620c103430 2024-05-09T09:19:54.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:54 smithi069 ceph-mon[30123]: pgmap v395: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:54 smithi163 ceph-mon[34354]: pgmap v395: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:54 smithi103 ceph-mon[34421]: pgmap v395: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:54.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.810+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f620c1a2490 con 0x7f620c103430 2024-05-09T09:19:54.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.811+0000 7f62027fc640 1 -- 172.21.15.69:0/2697695765 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 25 v25) ==== 94+0+11879 (secure 0 0 0) 0x7f620409c550 con 0x7f620c103430 2024-05-09T09:19:54.811 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:19:54.811 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":25,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":25,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:18:44.567983+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24490":{"gid":24490,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3224689113","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3224689113},{"type":"v1","addr":"172.21.15.163:6837","nonce":3224689113}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24491,24511]},"id":1}]} 2024-05-09T09:19:54.811 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 25 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 msgr2=0x7f61f80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 0x7f61f80a5050 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f620c1028d0 tx=0x7f6208005f90 comp rx=0 tx=0).stop 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 msgr2=0x7f620c1a1370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f620400eb10 tx=0x7f620400c2d0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 shutdown_connections 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f61f80a2b90 0x7f61f80a5050 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f620c103430 0x7f620c1a1370 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.814 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f620c102a70 0x7f620c19a300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 --2- 172.21.15.69:0/2697695765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f620c101880 0x7f620c199dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 >> 172.21.15.69:0/2697695765 conn(0x7f620c0fce30 msgr2=0x7f620c073a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 shutdown_connections 2024-05-09T09:19:54.815 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.814+0000 7f6213396640 1 -- 172.21.15.69:0/2697695765 wait complete. 2024-05-09T09:19:54.829 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:19:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 -- 172.21.15.69:0/3720717715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140101960 msgr2=0x7f5140101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 --2- 172.21.15.69:0/3720717715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140101960 0x7f5140101de0 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f5130009900 tx=0x7f513002a410 comp rx=0 tx=0).stop 2024-05-09T09:19:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 -- 172.21.15.69:0/3720717715 shutdown_connections 2024-05-09T09:19:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 --2- 172.21.15.69:0/3720717715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140102320 0x7f5140108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.936 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 --2- 172.21.15.69:0/3720717715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140101960 0x7f5140101de0 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 --2- 172.21.15.69:0/3720717715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140100770 0x7f5140100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 -- 172.21.15.69:0/3720717715 >> 172.21.15.69:0/3720717715 conn(0x7f51400fbce0 msgr2=0x7f51400fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 -- 172.21.15.69:0/3720717715 shutdown_connections 2024-05-09T09:19:54.937 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.936+0000 7f5147ed4640 1 -- 172.21.15.69:0/3720717715 wait complete. 2024-05-09T09:19:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 Processor -- start 2024-05-09T09:19:54.938 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 -- start start 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140101960 0x7f5140198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140102320 0x7f514019f1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f514010d510 con 0x7f5140100770 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f514010d390 con 0x7f5140101960 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5147ed4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f514010d690 con 0x7f5140102320 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5145c70640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5145c70640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51328/0 (socket says 172.21.15.69:51328) 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.937+0000 7f5145c70640 1 -- 172.21.15.69:0/471501976 learned_addr learned my addr 172.21.15.69:0/471501976 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f514546f640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140101960 0x7f5140198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 -- 172.21.15.69:0/471501976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140102320 msgr2=0x7f514019f1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140102320 0x7f514019f1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 -- 172.21.15.69:0/471501976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140101960 msgr2=0x7f5140198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.939 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140101960 0x7f5140198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 -- 172.21.15.69:0/471501976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f514019f850 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5145c70640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f513c00d950 tx=0x7f513c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5136ffd640 1 -- 172.21.15.69:0/471501976 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c017020 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5136ffd640 1 -- 172.21.15.69:0/471501976 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513c009c30 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f514019fb40 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.938+0000 7f5136ffd640 1 -- 172.21.15.69:0/471501976 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c00f720 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.939+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f51401a00e0 con 0x7f5140100770 2024-05-09T09:19:54.940 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.941+0000 7f5136ffd640 1 -- 172.21.15.69:0/471501976 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f513c01a020 con 0x7f5140100770 2024-05-09T09:19:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.941+0000 7f5136ffd640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 0x7f512c0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.942+0000 7f5136ffd640 1 -- 172.21.15.69:0/471501976 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f513c0ceaf0 con 0x7f5140100770 2024-05-09T09:19:54.941 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.942+0000 7f514546f640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 0x7f512c0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.942+0000 7f514546f640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 0x7f512c0a50d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f51300315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.942+0000 7f514abe5740 1 Processor -- start 2024-05-09T09:19:54.942 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 -- start start 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x558435683360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x558435688430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584355e7c40 con 0x558435688aa0 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55843558f580 con 0x558435683930 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514abe5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5584355e3c00 con 0x558435682f40 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51332/0 (socket says 172.21.15.69:51332) 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f5146471640 1 -- 172.21.15.69:0/3363757388 learned_addr learned my addr 172.21.15.69:0/3363757388 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f5145c70640 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x558435683360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f514546f640 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x558435688430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.943+0000 7f5146471640 1 -- 172.21.15.69:0/3363757388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 msgr2=0x558435683360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f5146471640 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x558435683360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f5146471640 1 -- 172.21.15.69:0/3363757388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 msgr2=0x558435688430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f5146471640 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x558435688430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f5146471640 1 -- 172.21.15.69:0/3363757388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584352fb3f0 con 0x558435688aa0 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f5146471640 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5138007bb0 tx=0x7f513800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=63d8205275053e02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f512bfff640 1 -- 172.21.15.69:0/3363757388 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138014070 con 0x558435688aa0 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f512bfff640 1 -- 172.21.15.69:0/3363757388 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f513800fb30 con 0x558435688aa0 2024-05-09T09:19:54.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.944+0000 7f512bfff640 1 -- 172.21.15.69:0/3363757388 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513800ead0 con 0x558435688aa0 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 -- 172.21.15.69:0/3363757388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 msgr2=0x55843568af30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f5138007bb0 tx=0x7f513800c6a0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 -- 172.21.15.69:0/3363757388 shutdown_connections 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x55843568af30 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x558435688430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 --2- 172.21.15.69:0/3363757388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x558435683360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 -- 172.21.15.69:0/3363757388 >> 172.21.15.69:0/3363757388 conn(0x55843567e5b0 msgr2=0x55843567e9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 -- 172.21.15.69:0/3363757388 shutdown_connections 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.945+0000 7f514abe5740 1 -- 172.21.15.69:0/3363757388 wait complete. 2024-05-09T09:19:54.949 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.946+0000 7f514abe5740 1 -- start start 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.946+0000 7f514abe5740 1 Processor -- start 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.946+0000 7f514abe5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.946+0000 7f514abe5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435683930 0x55843568d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.946+0000 7f514abe5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:50310/0 (socket says 172.21.15.69:50310) 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51334/0 (socket says 172.21.15.69:51334) 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 -- 172.21.15.69:0/884830832 learned_addr learned my addr 172.21.15.69:0/884830832 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55843568ba50 con 0x558435688aa0 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584355a2c40 con 0x558435682f40 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55843568bbd0 con 0x558435683930 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514546f640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435683930 0x55843568d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.950 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 -- 172.21.15.69:0/884830832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435683930 msgr2=0x55843568d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435683930 0x55843568d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 -- 172.21.15.69:0/884830832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 msgr2=0x558435694650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5145c70640 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584353c57d0 con 0x558435682f40 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f5146471640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514abe5740 10 client.0 _ll_register_callbacks cb 0x5584352fabe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.947+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f513800fb30 con 0x558435682f40 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f5145c70640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f513c0973a0 tx=0x7f513c09aa60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4b55d0123048af13 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f51297fa640 10 client.25313 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f51297fa640 1 -- 172.21.15.69:0/884830832 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f513c09b6d0 con 0x558435682f40 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f51297fa640 1 -- 172.21.15.69:0/884830832 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f513c0975b0 con 0x558435682f40 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f51297fa640 1 -- 172.21.15.69:0/884830832 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f513c09afd0 con 0x558435682f40 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 10 client.25313 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:54.951 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({fsmap=25}) -- 0x558435697810 con 0x558435682f40 2024-05-09T09:19:54.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f51297fa640 1 -- 172.21.15.69:0/884830832 <== mon.2 v2:172.21.15.103:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f513c0a37b0 con 0x558435682f40 2024-05-09T09:19:54.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 10 client.25313 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:54.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 10 client.25313 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:19:54.952 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 20 client.25313 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x55843569b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 4 client.25313 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.948+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5584355f5370 con 0x558435698f70 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.949+0000 7f514546f640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x55843569b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.949+0000 7f51297fa640 10 client.25313 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:54.953 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.949+0000 7f514546f640 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x55843569b410 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=956696b92ecae2ea server_cookie=f857443071225864 in_seq=0 out_seq=0 2024-05-09T09:19:54.954 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f51297fa640 1 -- 172.21.15.69:0/884830832 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5584355f5370 con 0x558435698f70 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f51297fa640 10 client.25313 handle_command_reply: tid=0 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 client.25313 shutdown 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 msgr2=0x55843568d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f513c0973a0 tx=0x7f513c09aa60 comp rx=0 tx=0).stop 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 shutdown_connections 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x55843569b410 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435694650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f51297fa640 0 client.25313 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435683930 0x55843568d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 --2- 172.21.15.69:0/884830832 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x55843568d0d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 >> 172.21.15.69:0/884830832 conn(0x55843567e5b0 msgr2=0x55843567d7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 shutdown_connections 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 1 -- 172.21.15.69:0/884830832 wait complete. 2024-05-09T09:19:54.955 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.950+0000 7f514abe5740 20 client.25313 trim_cache size 0 max 16384 2024-05-09T09:19:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 Processor -- start 2024-05-09T09:19:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 -- start start 2024-05-09T09:19:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x558435681c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.979 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435688aa0 0x55843568cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51401acd60 con 0x558435683930 2024-05-09T09:19:54.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51401aa5c0 con 0x558435682f40 2024-05-09T09:19:54.980 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514abe5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51401a8410 con 0x558435688aa0 2024-05-09T09:19:54.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51350/0 (socket says 172.21.15.69:51350) 2024-05-09T09:19:54.981 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 -- 172.21.15.69:0/789903301 learned_addr learned my addr 172.21.15.69:0/789903301 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f5145c70640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x558435681c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f5146471640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435688aa0 0x55843568cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 -- 172.21.15.69:0/789903301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435688aa0 msgr2=0x55843568cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435688aa0 0x55843568cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 -- 172.21.15.69:0/789903301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 msgr2=0x558435681c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x558435681c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f514546f640 1 -- 172.21.15.69:0/789903301 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55843566fd20 con 0x558435683930 2024-05-09T09:19:54.982 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.977+0000 7f5145c70640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x558435681c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:54.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.978+0000 7f514546f640 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f513003e040 tx=0x7f513003f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69a726c9932c075d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.978+0000 7f512b7fe640 1 -- 172.21.15.69:0/789903301 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130035050 con 0x558435683930 2024-05-09T09:19:54.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.978+0000 7f512b7fe640 1 -- 172.21.15.69:0/789903301 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5130056070 con 0x558435683930 2024-05-09T09:19:54.983 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.978+0000 7f512b7fe640 1 -- 172.21.15.69:0/789903301 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5130051520 con 0x558435683930 2024-05-09T09:19:54.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- 172.21.15.69:0/789903301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 msgr2=0x55843568c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f513003e040 tx=0x7f513003f000 comp rx=0 tx=0).stop 2024-05-09T09:19:54.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- 172.21.15.69:0/789903301 shutdown_connections 2024-05-09T09:19:54.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435688aa0 0x55843568cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.988 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435683930 0x55843568c6f0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 --2- 172.21.15.69:0/789903301 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435682f40 0x558435681c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- 172.21.15.69:0/789903301 >> 172.21.15.69:0/789903301 conn(0x55843567e5b0 msgr2=0x558435681560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- 172.21.15.69:0/789903301 shutdown_connections 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- 172.21.15.69:0/789903301 wait complete. 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 -- start start 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.979+0000 7f514abe5740 1 Processor -- start 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x5584356faff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x5584356fb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f51401a8a80 con 0x558435688aa0 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f51401a9ef0 con 0x558435683930 2024-05-09T09:19:54.989 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f51401a9150 con 0x558435682f40 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514abe5740 10 client.0 _ll_register_callbacks cb 0x7f513c09a930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51356/0 (socket says 172.21.15.69:51356) 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 -- 172.21.15.69:0/362604314 learned_addr learned my addr 172.21.15.69:0/362604314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f514546f640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x5584356fb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5145c70640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x5584356faff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 -- 172.21.15.69:0/362604314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 msgr2=0x5584356faff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x5584356faff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.990 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 -- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 msgr2=0x5584356fb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.980+0000 7f5146471640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x5584356fb530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f5146471640 1 -- 172.21.15.69:0/362604314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558435702b60 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f5146471640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f513800e5a0 tx=0x7f5138004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d201f81d368bf749 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f51297fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5138014040 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5130056070 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f513800f650 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f5138032050 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f514abe5740 10 client.15837 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x558435703ed0 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.981+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5130056070 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f51380046b0 con 0x558435688aa0 2024-05-09T09:19:54.991 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 10 client.15837 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 10 client.15837 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 20 client.15837 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x558435706550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 4 client.15837 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.982+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5584355ef6c0 con 0x558435698f70 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f5145c70640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x558435706550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f51297fa640 10 client.15837 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f5145c70640 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x558435706550 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=7ac3476a69ffea74 server_cookie=620ce90af2fd695a in_seq=0 out_seq=0 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f51297fa640 1 -- 172.21.15.69:0/362604314 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5584355ef6c0 con 0x558435698f70 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f51297fa640 10 client.15837 handle_command_reply: tid=0 2024-05-09T09:19:54.992 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f514abe5740 1 client.15837 shutdown 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.983+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 msgr2=0x558435702570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f513800e5a0 tx=0x7f5138004a40 comp rx=0 tx=0).stop 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 shutdown_connections 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x558435698f70 0x558435706550 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558435688aa0 0x558435702570 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f51297fa640 0 client.15837 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558435683930 0x5584356fb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 --2- 172.21.15.69:0/362604314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558435682f40 0x5584356faff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 >> 172.21.15.69:0/362604314 conn(0x55843567e5b0 msgr2=0x55843567cf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 shutdown_connections 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 1 -- 172.21.15.69:0/362604314 wait complete. 2024-05-09T09:19:54.993 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.984+0000 7f514abe5740 20 client.15837 trim_cache size 0 max 16384 2024-05-09T09:19:54.994 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 msgr2=0x7f512c0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 0x7f512c0a50d0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f5130000c00 tx=0x7f51300315d0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 msgr2=0x7f5140197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f513c00d950 tx=0x7f513c00de10 comp rx=0 tx=0).stop 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 shutdown_connections 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f512c0a2c10 0x7f512c0a50d0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5140102320 0x7f514019f1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5140101960 0x7f5140198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.995 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 --2- 172.21.15.69:0/471501976 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5140100770 0x7f5140197c40 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:54.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 >> 172.21.15.69:0/471501976 conn(0x7f51400fbce0 msgr2=0x7f5140104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:54.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 shutdown_connections 2024-05-09T09:19:54.996 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:54.985+0000 7f5147ed4640 1 -- 172.21.15.69:0/471501976 wait complete. 2024-05-09T09:19:54.999 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:19:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 -- 172.21.15.69:0/3832413951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0102a70 msgr2=0x7f17c0102ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/3832413951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0102a70 0x7f17c0102ef0 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7f17ac009900 tx=0x7f17ac02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 -- 172.21.15.69:0/3832413951 shutdown_connections 2024-05-09T09:19:55.110 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/3832413951 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c0109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/3832413951 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0102a70 0x7f17c0102ef0 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/3832413951 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0101880 0x7f17c0101ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 -- 172.21.15.69:0/3832413951 >> 172.21.15.69:0/3832413951 conn(0x7f17c00fce10 msgr2=0x7f17c00ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 -- 172.21.15.69:0/3832413951 shutdown_connections 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 -- 172.21.15.69:0/3832413951 wait complete. 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.106+0000 7f17c6e5d640 1 Processor -- start 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 -- start start 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0102a70 0x7f17c0198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17c010d500 con 0x7f17c0101880 2024-05-09T09:19:55.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17c010d380 con 0x7f17c0103430 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c6e5d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17c010d680 con 0x7f17c0102a70 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51374/0 (socket says 172.21.15.69:51374) 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 -- 172.21.15.69:0/1494272274 learned_addr learned my addr 172.21.15.69:0/1494272274 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17bffff640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0102a70 0x7f17c0198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c53fa640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 -- 172.21.15.69:0/1494272274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0102a70 msgr2=0x7f17c0198130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0102a70 0x7f17c0198130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 -- 172.21.15.69:0/1494272274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 msgr2=0x7f17c019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c4bf9640 1 -- 172.21.15.69:0/1494272274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f17c019f810 con 0x7f17c0101880 2024-05-09T09:19:55.112 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.107+0000 7f17c53fa640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f17b400d9c0 tx=0x7f17b400de80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17bdffb640 1 -- 172.21.15.69:0/1494272274 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4017070 con 0x7f17c0101880 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17bdffb640 1 -- 172.21.15.69:0/1494272274 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b4009b40 con 0x7f17c0101880 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f17c019fb00 con 0x7f17c0101880 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17bdffb640 1 -- 172.21.15.69:0/1494272274 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b400f620 con 0x7f17c0101880 2024-05-09T09:19:55.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.108+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f17c019ffd0 con 0x7f17c0101880 2024-05-09T09:19:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.112+0000 7f17bdffb640 1 -- 172.21.15.69:0/1494272274 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f17b401a070 con 0x7f17c0101880 2024-05-09T09:19:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.112+0000 7f17bdffb640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 0x7f17b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.117 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.112+0000 7f17bdffb640 1 -- 172.21.15.69:0/1494272274 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f17b40cf3a0 con 0x7f17c0101880 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.112+0000 7f17c9b6b740 1 Processor -- start 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 -- start start 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17bffff640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 0x7f17b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc63c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc66790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651adbbac40 con 0x5651adc5ee60 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5651adb62580 con 0x5651adc64300 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651adbb6c00 con 0x5651adc61820 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.118 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51386/0 (socket says 172.21.15.69:51386) 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 -- 172.21.15.69:0/2583511881 learned_addr learned my addr 172.21.15.69:0/2583511881 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c53fa640 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc66790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17bffff640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 0x7f17b00a5050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f17ac000c00 tx=0x7f17ac0315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 -- 172.21.15.69:0/2583511881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 msgr2=0x5651adc63c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc63c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 -- 172.21.15.69:0/2583511881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 msgr2=0x5651adc66790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc66790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 -- 172.21.15.69:0/2583511881 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651ad8ce3f0 con 0x5651adc5ee60 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.113+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17b4098250 tx=0x7f17b409c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=322ab1d6d95923e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.114+0000 7f17aaffd640 1 -- 172.21.15.69:0/2583511881 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b40a44f0 con 0x5651adc5ee60 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.114+0000 7f17aaffd640 1 -- 172.21.15.69:0/2583511881 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b4099920 con 0x5651adc5ee60 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17aaffd640 1 -- 172.21.15.69:0/2583511881 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b409cbb0 con 0x5651adc5ee60 2024-05-09T09:19:55.119 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- 172.21.15.69:0/2583511881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 msgr2=0x5651adc61250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f17b4098250 tx=0x7f17b409c040 comp rx=0 tx=0).stop 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- 172.21.15.69:0/2583511881 shutdown_connections 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc66790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc63c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/2583511881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc61250 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- 172.21.15.69:0/2583511881 >> 172.21.15.69:0/2583511881 conn(0x5651adc51530 msgr2=0x5651adc51930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- 172.21.15.69:0/2583511881 shutdown_connections 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- 172.21.15.69:0/2583511881 wait complete. 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 -- start start 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.115+0000 7f17c9b6b740 1 Processor -- start 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 0x5651adc68c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc69150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c53fa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.120 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c53fa640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51400/0 (socket says 172.21.15.69:51400) 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c53fa640 1 -- 172.21.15.69:0/4136145414 learned_addr learned my addr 172.21.15.69:0/4136145414 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 0x5651adc68c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651adc67350 con 0x5651adc64300 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5651adb75c40 con 0x5651adc5ee60 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.116+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651adc674d0 con 0x5651adc61820 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 -- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 msgr2=0x5651adc69150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc69150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 -- 172.21.15.69:0/4136145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 msgr2=0x5651adc68c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 0x5651adc68c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651ad9987d0 con 0x5651adc64300 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 0x5651adc68c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c53fa640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f17b80129a0 tx=0x7f17b8012e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=330853ec035abddd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f179ffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:55.121 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c9b6b740 10 client.0 _ll_register_callbacks cb 0x5651ad8cdbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b4099920 con 0x5651adc64300 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f179ffff640 1 -- 172.21.15.69:0/4136145414 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b8010e20 con 0x5651adc64300 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f179ffff640 1 -- 172.21.15.69:0/4136145414 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f17b8004b20 con 0x5651adc64300 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f179ffff640 1 -- 172.21.15.69:0/4136145414 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17b8004da0 con 0x5651adc64300 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.117+0000 7f17c9b6b740 10 client.15849 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:55.122 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5651adc73440 con 0x5651adc64300 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f179ffff640 1 -- 172.21.15.69:0/4136145414 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f17b8025410 con 0x5651adc64300 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 10 client.15849 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 10 client.15849 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 20 client.15849 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc74ac0 0x5651adc76f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 4 client.15849 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651adbc8370 con 0x5651adc74ac0 2024-05-09T09:19:55.124 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.118+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc74ac0 0x5651adc76f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.119+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc74ac0 0x5651adc76f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=22d8e765911c7ae2 server_cookie=6eef7e1a9ec3bd32 in_seq=0 out_seq=0 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.119+0000 7f179ffff640 10 client.15849 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.119+0000 7f179ffff640 1 -- 172.21.15.69:0/4136145414 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651adbc8370 con 0x5651adc74ac0 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.119+0000 7f179ffff640 10 client.15849 handle_command_reply: tid=0 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.119+0000 7f17c9b6b740 1 client.15849 shutdown 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 msgr2=0x5651adc70190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f17b80129a0 tx=0x7f17b8012e60 comp rx=0 tx=0).stop 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 shutdown_connections 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc74ac0 0x5651adc76f80 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc64300 0x5651adc70190 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f179ffff640 0 client.15849 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc69150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4136145414 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc5ee60 0x5651adc68c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 >> 172.21.15.69:0/4136145414 conn(0x5651adc51530 msgr2=0x5651adc5e4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.125 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 shutdown_connections 2024-05-09T09:19:55.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4136145414 wait complete. 2024-05-09T09:19:55.126 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.120+0000 7f17c9b6b740 20 client.15849 trim_cache size 0 max 16384 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 Processor -- start 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 -- start start 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc700b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc50630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01aaac0 con 0x5651adc5ee60 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.146+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01a8390 con 0x5651adc64300 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c9b6b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01a61e0 con 0x5651adc61820 2024-05-09T09:19:55.149 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51406/0 (socket says 172.21.15.69:51406) 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 -- 172.21.15.69:0/4132108867 learned_addr learned my addr 172.21.15.69:0/4132108867 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c53fa640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc50630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17bffff640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 -- 172.21.15.69:0/4132108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 msgr2=0x5651adc700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 -- 172.21.15.69:0/4132108867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 msgr2=0x5651adc50630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc50630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 -- 172.21.15.69:0/4132108867 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651adc42d20 con 0x5651adc5ee60 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17bffff640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f17b409b810 tx=0x7f17b4086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=588a616a67db09b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.147+0000 7f17aa7fc640 1 -- 172.21.15.69:0/4132108867 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4099b80 con 0x5651adc5ee60 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.148+0000 7f17aa7fc640 1 -- 172.21.15.69:0/4132108867 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f17b4009cf0 con 0x5651adc5ee60 2024-05-09T09:19:55.150 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.148+0000 7f17aa7fc640 1 -- 172.21.15.69:0/4132108867 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b409cbb0 con 0x5651adc5ee60 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.148+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4132108867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 msgr2=0x5651adc5f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.148+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f17b409b810 tx=0x7f17b4086040 comp rx=0 tx=0).stop 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4132108867 shutdown_connections 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc64300 0x5651adc50630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc61820 0x5651adc700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/4132108867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc5f580 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4132108867 >> 172.21.15.69:0/4132108867 conn(0x5651adc51530 msgr2=0x5651adc4ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4132108867 shutdown_connections 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 -- 172.21.15.69:0/4132108867 wait complete. 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 -- start start 2024-05-09T09:19:55.151 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.149+0000 7f17c9b6b740 1 Processor -- start 2024-05-09T09:19:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.154 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51414/0 (socket says 172.21.15.69:51414) 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 -- 172.21.15.69:0/3709187099 learned_addr learned my addr 172.21.15.69:0/3709187099 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc61820 0x5651adc6d900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc64300 0x5651adc6e7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01a6850 con 0x5651adc5ee60 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01a7cc0 con 0x5651adc61820 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f17c01a6f20 con 0x5651adc64300 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17bffff640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc61820 0x5651adc6d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 -- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc64300 msgr2=0x5651adc6e7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc64300 0x5651adc6e7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 -- 172.21.15.69:0/3709187099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc61820 msgr2=0x5651adc6d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.155 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc61820 0x5651adc6d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.150+0000 7f17c4bf9640 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651adc6edc0 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c4bf9640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f17b40cce90 tx=0x7f17b4097c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbe36211caecaf2f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 10 client.0 _ll_register_callbacks cb 0x7f17b8009a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f179ffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f17b4009cf0 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f179ffff640 1 -- 172.21.15.69:0/3709187099 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f17b4065060 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f179ffff640 1 -- 172.21.15.69:0/3709187099 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f17b40cf170 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f179ffff640 1 -- 172.21.15.69:0/3709187099 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f17b4099b80 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 10 client.15855 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5651adc6fd60 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f179ffff640 1 -- 172.21.15.69:0/3709187099 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f17b4086120 con 0x5651adc5ee60 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 10 client.15855 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 10 client.15855 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 20 client.15855 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:55.156 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc701d0 0x5651adcdd110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 4 client.15855 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.151+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "damage ls"}) -- 0x5651adbc26c0 con 0x5651adc701d0 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17bffff640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc701d0 0x5651adcdd110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17bffff640 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc701d0 0x5651adcdd110 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=19c573cefd56c2ec server_cookie=9b6c176e3d3ebc69 in_seq=0 out_seq=0 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f179ffff640 10 client.15855 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f179ffff640 1 -- 172.21.15.69:0/3709187099 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x5651adbc26c0 con 0x5651adc701d0 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f179ffff640 10 client.15855 handle_command_reply: tid=0 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17c9b6b740 1 client.15855 shutdown 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 msgr2=0x5651adc6d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f17b40cce90 tx=0x7f17b4097c30 comp rx=0 tx=0).stop 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.152+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 shutdown_connections 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x5651adc701d0 0x5651adcdd110 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5651adc64300 0x5651adc6e7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.157 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5651adc61820 0x5651adc6d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 --2- 172.21.15.69:0/3709187099 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5651adc5ee60 0x5651adc6d3c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 >> 172.21.15.69:0/3709187099 conn(0x5651adc51530 msgr2=0x5651adc507b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f179ffff640 0 client.15855 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 shutdown_connections 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 1 -- 172.21.15.69:0/3709187099 wait complete. 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c9b6b740 20 client.15855 trim_cache size 0 max 16384 2024-05-09T09:19:55.158 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 msgr2=0x7f17b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 0x7f17b00a5050 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f17ac000c00 tx=0x7f17ac0315d0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 msgr2=0x7f17c0197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.153+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7f17b400d9c0 tx=0x7f17b400de80 comp rx=0 tx=0).stop 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 shutdown_connections 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f17b00a2b90 0x7f17b00a5050 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f17c0103430 0x7f17c019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f17c0102a70 0x7f17c0198130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.159 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 --2- 172.21.15.69:0/1494272274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f17c0101880 0x7f17c0197bf0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 >> 172.21.15.69:0/1494272274 conn(0x7f17c00fce10 msgr2=0x7f17c0105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 shutdown_connections 2024-05-09T09:19:55.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.154+0000 7f17c6e5d640 1 -- 172.21.15.69:0/1494272274 wait complete. 2024-05-09T09:19:55.165 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:19:55.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 -- 172.21.15.69:0/438580045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 msgr2=0x7f3928109d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.274 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 --2- 172.21.15.69:0/438580045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f3928109d50 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f3920009cd0 tx=0x7f392002a8e0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 -- 172.21.15.69:0/438580045 shutdown_connections 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 --2- 172.21.15.69:0/438580045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f3928109d50 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 --2- 172.21.15.69:0/438580045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f3928102f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 --2- 172.21.15.69:0/438580045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f39280fff50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 -- 172.21.15.69:0/438580045 >> 172.21.15.69:0/438580045 conn(0x7f39280fb460 msgr2=0x7f39280fd8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 -- 172.21.15.69:0/438580045 shutdown_connections 2024-05-09T09:19:55.275 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.274+0000 7f392d3d5640 1 -- 172.21.15.69:0/438580045 wait complete. 2024-05-09T09:19:55.277 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 Processor -- start 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 -- start start 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f3928197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f39281980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f392810d4e0 con 0x7f39281034c0 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f392810d360 con 0x7f3928102b00 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f392d3d5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f392810d660 con 0x7f39280ffb30 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51436/0 (socket says 172.21.15.69:51436) 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f39277fe640 1 -- 172.21.15.69:0/928809358 learned_addr learned my addr 172.21.15.69:0/928809358 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f3926ffd640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f3928197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.278 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.275+0000 7f39267fc640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f39281980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 -- 172.21.15.69:0/928809358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 msgr2=0x7f3928197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f3928197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 -- 172.21.15.69:0/928809358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 msgr2=0x7f39281980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f39281980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 -- 172.21.15.69:0/928809358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f392819f7a0 con 0x7f39281034c0 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f3926ffd640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f3928197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39267fc640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f39281980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f39277fe640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3920004af0 tx=0x7f3920003c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.280 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f390ffff640 1 -- 172.21.15.69:0/928809358 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3920004450 con 0x7f39281034c0 2024-05-09T09:19:55.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f392819fa30 con 0x7f39281034c0 2024-05-09T09:19:55.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f390ffff640 1 -- 172.21.15.69:0/928809358 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f39200045f0 con 0x7f39281034c0 2024-05-09T09:19:55.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f390ffff640 1 -- 172.21.15.69:0/928809358 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39200466c0 con 0x7f39281034c0 2024-05-09T09:19:55.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.276+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f392819ffd0 con 0x7f39281034c0 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.279+0000 7f39301c1740 1 Processor -- start 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.279+0000 7f390ffff640 1 -- 172.21.15.69:0/928809358 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3920046860 con 0x7f39281034c0 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 -- start start 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1aa360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a59e10ec40 con 0x55a59e1afaa0 2024-05-09T09:19:55.286 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a59e0b6580 con 0x55a59e1a9f40 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39301c1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a59e10ac00 con 0x55a59e1aa930 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f3926ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51442/0 (socket says 172.21.15.69:51442) 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f3926ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51832/0 (socket says 172.21.15.69:51832) 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 -- 172.21.15.69:0/3985370848 learned_addr learned my addr 172.21.15.69:0/3985370848 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1aa360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 -- 172.21.15.69:0/3985370848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 msgr2=0x55a59e1af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 -- 172.21.15.69:0/3985370848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 msgr2=0x55a59e1aa360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1aa360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f39267fc640 1 -- 172.21.15.69:0/3985370848 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59de223f0 con 0x55a59e1afaa0 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f3926ffd640 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:55.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.280+0000 7f390ffff640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 0x7f39100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f39267fc640 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f391c014e60 tx=0x7f391c00bad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326aa0d6a320f884 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f3926ffd640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 0x7f39100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f390ffff640 1 -- 172.21.15.69:0/928809358 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f392002e030 con 0x7f39281034c0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f390d7fa640 1 -- 172.21.15.69:0/3985370848 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c00db70 con 0x55a59e1afaa0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f390d7fa640 1 -- 172.21.15.69:0/3985370848 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391c00dd10 con 0x55a59e1afaa0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.281+0000 7f390d7fa640 1 -- 172.21.15.69:0/3985370848 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c005280 con 0x55a59e1afaa0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- 172.21.15.69:0/3985370848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 msgr2=0x55a59e1b1f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7f391c014e60 tx=0x7f391c00bad0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f3926ffd640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 0x7f39100a5050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a59e1aad30 tx=0x7f391800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- 172.21.15.69:0/3985370848 shutdown_connections 2024-05-09T09:19:55.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1b1f30 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 --2- 172.21.15.69:0/3985370848 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1aa360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- 172.21.15.69:0/3985370848 >> 172.21.15.69:0/3985370848 conn(0x55a59e1a55b0 msgr2=0x55a59e1a59b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- 172.21.15.69:0/3985370848 shutdown_connections 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- 172.21.15.69:0/3985370848 wait complete. 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 -- start start 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.282+0000 7f39301c1740 1 Processor -- start 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1b4360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 0x55a59e1b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a59e1b29f0 con 0x55a59e1afaa0 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a59e0c9c40 con 0x55a59e1aa930 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a59e1b2b70 con 0x55a59e1a9f40 2024-05-09T09:19:55.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39301c1740 10 client.0 _ll_register_callbacks cb 0x55a59de21be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f3926ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51454/0 (socket says 172.21.15.69:51454) 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 -- 172.21.15.69:0/3393126474 learned_addr learned my addr 172.21.15.69:0/3393126474 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f3926ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1b4360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51848/0 (socket says 172.21.15.69:51848) 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 -- 172.21.15.69:0/3393126474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 msgr2=0x55a59e1b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39267fc640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 0x55a59e1b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 -- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 msgr2=0x55a59e1b48a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 0x55a59e1b48a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 -- 172.21.15.69:0/3393126474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59deec7d0 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39267fc640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 0x55a59e1b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f39277fe640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f39200c55f0 tx=0x7f39200bcd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3236f81c24dac2e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f390aff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.283+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39200bceb0 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f39200bb1b0 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f391c00dd10 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f39200c1260 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 10 client.15867 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a59e1bec50 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391c00dd10 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f3920030070 con 0x55a59e1afaa0 2024-05-09T09:19:55.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 10 client.15867 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 20 client.15867 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e1c2760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 4 client.15867 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a59e11c370 con 0x55a59e1c02e0 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.284+0000 7f3926ffd640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e1c2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.285+0000 7f3926ffd640 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e1c2760 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=170ded9f4a9fa7e8 server_cookie=50e29f0258e197ca in_seq=0 out_seq=0 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.285+0000 7f390aff5640 10 client.15867 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f390aff5640 1 -- 172.21.15.69:0/3393126474 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a59e11c370 con 0x55a59e1c02e0 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f390aff5640 10 client.15867 handle_command_reply: tid=0 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 client.15867 shutdown 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 msgr2=0x55a59e1bb8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7f39200c55f0 tx=0x7f39200bcd80 comp rx=0 tx=0).stop 2024-05-09T09:19:55.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 shutdown_connections 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1bb8e0 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1aa930 0x55a59e1b48a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e1c2760 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f390aff5640 0 client.15867 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 --2- 172.21.15.69:0/3393126474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 >> 172.21.15.69:0/3393126474 conn(0x55a59e1a55b0 msgr2=0x55a59e1a47b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 shutdown_connections 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 1 -- 172.21.15.69:0/3393126474 wait complete. 2024-05-09T09:19:55.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.286+0000 7f39301c1740 20 client.15867 trim_cache size 0 max 16384 2024-05-09T09:19:55.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.299+0000 7f39301c1740 1 Processor -- start 2024-05-09T09:19:55.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.299+0000 7f39301c1740 1 -- start start 2024-05-09T09:19:55.300 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.299+0000 7f39301c1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1a3f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.299+0000 7f39301c1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39301c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 0x55a59e1a4460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39301c1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39281aaa00 con 0x55a59e1aa930 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39301c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39281a82d0 con 0x55a59e1afaa0 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39301c1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39281a6120 con 0x55a59e1a9f40 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51466/0 (socket says 172.21.15.69:51466) 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 -- 172.21.15.69:0/3415410521 learned_addr learned my addr 172.21.15.69:0/3415410521 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f3926ffd640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1a3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39277fe640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 0x55a59e1a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 -- 172.21.15.69:0/3415410521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 msgr2=0x55a59e1a3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1a3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.301 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 -- 172.21.15.69:0/3415410521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 msgr2=0x55a59e1a4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 0x55a59e1a4460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 -- 172.21.15.69:0/3415410521 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59e196d20 con 0x55a59e1aa930 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39277fe640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 0x55a59e1a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f39267fc640 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f391c0058c0 tx=0x7f391c00b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad7284103cb11bba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.300+0000 7f390cff9640 1 -- 172.21.15.69:0/3415410521 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c00dd40 con 0x55a59e1aa930 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.301+0000 7f390cff9640 1 -- 172.21.15.69:0/3415410521 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f391c0140d0 con 0x55a59e1aa930 2024-05-09T09:19:55.302 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.301+0000 7f390cff9640 1 -- 172.21.15.69:0/3415410521 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f391c005290 con 0x55a59e1aa930 2024-05-09T09:19:55.306 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.301+0000 7f39301c1740 1 -- 172.21.15.69:0/3415410521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 msgr2=0x55a59e1bb8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.301+0000 7f39301c1740 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7f391c0058c0 tx=0x7f391c00b5b0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- 172.21.15.69:0/3415410521 shutdown_connections 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1afaa0 0x55a59e1a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1aa930 0x55a59e1bb8a0 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- 172.21.15.69:0/3415410521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1a9f40 0x55a59e1a3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- 172.21.15.69:0/3415410521 >> 172.21.15.69:0/3415410521 conn(0x55a59e1a55b0 msgr2=0x55a59e1abd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- 172.21.15.69:0/3415410521 shutdown_connections 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- 172.21.15.69:0/3415410521 wait complete. 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- start start 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 Processor -- start 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1b91a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1b96e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f39281a6790 con 0x55a59e1afaa0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f39281a7c00 con 0x55a59e1a9f40 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.302+0000 7f39301c1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f39281a6e60 con 0x55a59e1aa930 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51478/0 (socket says 172.21.15.69:51478) 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 -- 172.21.15.69:0/222626889 learned_addr learned my addr 172.21.15.69:0/222626889 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f3926ffd640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1b91a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39267fc640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1b96e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 -- 172.21.15.69:0/222626889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 msgr2=0x55a59e1b96e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1b96e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39301c1740 10 client.0 _ll_register_callbacks cb 0x7f3920009c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 -- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 msgr2=0x55a59e1b91a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1b91a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 -- 172.21.15.69:0/222626889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59e1bace0 con 0x55a59e1afaa0 2024-05-09T09:19:55.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f391c0140d0 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39277fe640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f39200ee1a0 tx=0x7f39200bc750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=934afab871fb4576 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f390aff5640 10 client.15873 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f390aff5640 1 -- 172.21.15.69:0/222626889 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f39200bc810 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f390aff5640 1 -- 172.21.15.69:0/222626889 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f39200dbaf0 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f390aff5640 1 -- 172.21.15.69:0/222626889 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f39200b9150 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39301c1740 10 client.15873 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.303+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x55a59e1bace0 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f390aff5640 1 -- 172.21.15.69:0/222626889 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f39200a2020 con 0x55a59e1afaa0 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 10 client.15873 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 20 client.15873 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e228e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 4 client.15873 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:19:55.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a59e1b4150 con 0x55a59e1c02e0 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.304+0000 7f3926ffd640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e228e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f390aff5640 10 client.15873 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f3926ffd640 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e228e50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=154173e323390012 server_cookie=f1b8e094fa60f427 in_seq=0 out_seq=0 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f390aff5640 1 -- 172.21.15.69:0/222626889 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a59e1b4150 con 0x55a59e1c02e0 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f390aff5640 10 client.15873 handle_command_reply: tid=0 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 client.15873 shutdown 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 msgr2=0x55a59e1ba6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7f39200ee1a0 tx=0x7f39200bc750 comp rx=0 tx=0).stop 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 shutdown_connections 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59e1afaa0 0x55a59e1ba6f0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59e1c02e0 0x55a59e228e50 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59e1aa930 0x55a59e1b96e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 --2- 172.21.15.69:0/222626889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59e1a9f40 0x55a59e1b91a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 >> 172.21.15.69:0/222626889 conn(0x55a59e1a55b0 msgr2=0x55a59e1abd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f390aff5640 0 client.15873 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:55.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 shutdown_connections 2024-05-09T09:19:55.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 1 -- 172.21.15.69:0/222626889 wait complete. 2024-05-09T09:19:55.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.305+0000 7f39301c1740 20 client.15873 trim_cache size 0 max 16384 2024-05-09T09:19:55.311 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 msgr2=0x7f39100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 0x7f39100a5050 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x55a59e1aad30 tx=0x7f391800b410 comp rx=0 tx=0).stop 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 msgr2=0x7f392819f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7f3920004af0 tx=0x7f3920003c10 comp rx=0 tx=0).stop 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 shutdown_connections 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f39100a2b90 0x7f39100a5050 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f39281034c0 0x7f392819f130 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3928102b00 0x7f39281980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 --2- 172.21.15.69:0/928809358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f39280ffb30 0x7f3928197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:55.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 >> 172.21.15.69:0/928809358 conn(0x7f39280fb460 msgr2=0x7f3928105e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:55.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 shutdown_connections 2024-05-09T09:19:55.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:55.306+0000 7f392d3d5640 1 -- 172.21.15.69:0/928809358 wait complete. 2024-05-09T09:19:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2697695765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:55 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2697695765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:55 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2697695765' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:19:56.318 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:19:56.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:19:56.318 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:19:56.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 -- 172.21.15.69:0/4045700039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a0104cc0 msgr2=0x7fa7a01070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 --2- 172.21.15.69:0/4045700039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a0104cc0 0x7fa7a01070b0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7fa790009900 tx=0x7fa79002a410 comp rx=0 tx=0).stop 2024-05-09T09:19:56.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 -- 172.21.15.69:0/4045700039 shutdown_connections 2024-05-09T09:19:56.428 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 --2- 172.21.15.69:0/4045700039 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a01075f0 0x7fa7a01099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 --2- 172.21.15.69:0/4045700039 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a0104cc0 0x7fa7a01070b0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 --2- 172.21.15.69:0/4045700039 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 0x7fa7a0104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 -- 172.21.15.69:0/4045700039 >> 172.21.15.69:0/4045700039 conn(0x7fa7a00fbce0 msgr2=0x7fa7a00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 -- 172.21.15.69:0/4045700039 shutdown_connections 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.427+0000 7fa7a7529640 1 -- 172.21.15.69:0/4045700039 wait complete. 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 Processor -- start 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 -- start start 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 0x7fa7a0197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a0104cc0 0x7fa7a0198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010d4f0 con 0x7fa7a01075f0 2024-05-09T09:19:56.429 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010d370 con 0x7fa7a0102390 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a7529640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a010d670 con 0x7fa7a0104cc0 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a5ac6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a5ac6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51494/0 (socket says 172.21.15.69:51494) 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/823277246 learned_addr learned my addr 172.21.15.69:0/823277246 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.428+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 0x7fa7a0197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a0104cc0 0x7fa7a0198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/823277246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a0104cc0 msgr2=0x7fa7a0198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a0104cc0 0x7fa7a0198100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/823277246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 msgr2=0x7fa7a0197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 0x7fa7a0197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.430 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/823277246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa7a019f7e0 con 0x7fa7a01075f0 2024-05-09T09:19:56.431 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa79800eaa0 tx=0x7fa79800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7967fc640 1 -- 172.21.15.69:0/823277246 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79800cdb0 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa7a019fad0 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7967fc640 1 -- 172.21.15.69:0/823277246 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa798005020 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa7a0100410 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.429+0000 7fa7967fc640 1 -- 172.21.15.69:0/823277246 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79801f060 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.432+0000 7fa7967fc640 1 -- 172.21.15.69:0/823277246 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa7980054d0 con 0x7fa7a01075f0 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.432+0000 7fa7967fc640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 0x7fa78c0ad860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 Processor -- start 2024-05-09T09:19:56.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 0x7fa78c0ad860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 -- start start 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7967fc640 1 -- 172.21.15.69:0/823277246 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa7980cf8a0 con 0x7fa7a01075f0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 0x56385766e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x5638576739b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5638575d2990 con 0x56385766eeb0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56385757a5b0 con 0x563857674020 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5638575ce950 con 0x56385766e4c0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 0x7fa78c0ad860 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00bc30 tx=0x7fa79c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a5ac6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a52c5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 0x56385766e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a5ac6640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:50470/0 (socket says 172.21.15.69:50470) 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/935007247 learned_addr learned my addr 172.21.15.69:0/935007247 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.433+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x5638576739b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/935007247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 msgr2=0x56385766e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 0x56385766e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/935007247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 msgr2=0x5638576739b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x5638576739b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 -- 172.21.15.69:0/935007247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638573ae7e0 con 0x563857674020 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 0x56385766e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x5638576739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7980a44c0 tx=0x7fa79809bf70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=720c84545f573b28 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa78b7fe640 1 -- 172.21.15.69:0/935007247 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79809a130 con 0x563857674020 2024-05-09T09:19:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa78b7fe640 1 -- 172.21.15.69:0/935007247 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa7980984a0 con 0x563857674020 2024-05-09T09:19:56.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.434+0000 7fa78b7fe640 1 -- 172.21.15.69:0/935007247 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79809b280 con 0x563857674020 2024-05-09T09:19:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 -- 172.21.15.69:0/935007247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 msgr2=0x5638576764b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fa7980a44c0 tx=0x7fa79809bf70 comp rx=0 tx=0).stop 2024-05-09T09:19:56.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 -- 172.21.15.69:0/935007247 shutdown_connections 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563857674020 0x5638576764b0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x5638576739b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/935007247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56385766e4c0 0x56385766e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 -- 172.21.15.69:0/935007247 >> 172.21.15.69:0/935007247 conn(0x5638576360a0 msgr2=0x563857669f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 -- 172.21.15.69:0/935007247 shutdown_connections 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.435+0000 7fa7aa23c740 1 -- 172.21.15.69:0/935007247 wait complete. 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 -- start start 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 Processor -- start 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x5638576788e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7a4ac4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7a4ac4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51514/0 (socket says 172.21.15.69:51514) 2024-05-09T09:19:56.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/694807775 learned_addr learned my addr 172.21.15.69:0/694807775 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x5638576788e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563857676fd0 con 0x56385766eeb0 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56385758db40 con 0x56385766e4c0 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.436+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563857677150 con 0x563857674020 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/694807775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 msgr2=0x56385767fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 msgr2=0x5638576788e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x5638576788e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385740ef70 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x5638576788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa7900314e0 tx=0x7fa790004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de0f265b71055d37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa788ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7aa23c740 10 client.0 _ll_register_callbacks cb 0x5638575820f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa7980984a0 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.437+0000 7fa788ff9640 1 -- 172.21.15.69:0/694807775 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790035040 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa788ff9640 1 -- 172.21.15.69:0/694807775 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa790005030 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa788ff9640 1 -- 172.21.15.69:0/694807775 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa790005230 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 10 client.15885 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5638572a6170 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa788ff9640 1 -- 172.21.15.69:0/694807775 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa7900053d0 con 0x56385766eeb0 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 10 client.15885 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:56.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 10 client.15885 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 20 client.15885 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576841e0 0x563857686670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 4 client.15885 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638575e00c0 con 0x5638576841e0 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.438+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576841e0 0x563857686670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.439+0000 7fa788ff9640 10 client.15885 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.439+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576841e0 0x563857686670 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=709e140c71f5ece0 server_cookie=70cb1a9689526ef2 in_seq=0 out_seq=0 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.439+0000 7fa788ff9640 1 -- 172.21.15.69:0/694807775 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638575e00c0 con 0x5638576841e0 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.439+0000 7fa788ff9640 10 client.15885 handle_command_reply: tid=0 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 client.15885 shutdown 2024-05-09T09:19:56.444 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 msgr2=0x563857678e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7fa7900314e0 tx=0x7fa790004270 comp rx=0 tx=0).stop 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 shutdown_connections 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576841e0 0x563857686670 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa788ff9640 0 client.15885 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766eeb0 0x563857678e20 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/694807775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x5638576788e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 >> 172.21.15.69:0/694807775 conn(0x5638576360a0 msgr2=0x563857669080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 shutdown_connections 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 1 -- 172.21.15.69:0/694807775 wait complete. 2024-05-09T09:19:56.445 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.440+0000 7fa7aa23c740 20 client.15885 trim_cache size 0 max 16384 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.457+0000 7fa7aa23c740 1 Processor -- start 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.457+0000 7fa7aa23c740 1 -- start start 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 0x56385766d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01aaa50 con 0x56385766e4c0 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a8320 con 0x5638576841e0 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7aa23c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a6170 con 0x563857674020 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51518/0 (socket says 172.21.15.69:51518) 2024-05-09T09:19:56.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 -- 172.21.15.69:0/3013460509 learned_addr learned my addr 172.21.15.69:0/3013460509 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 0x56385766d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 -- 172.21.15.69:0/3013460509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 msgr2=0x56385767fe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 -- 172.21.15.69:0/3013460509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 msgr2=0x56385766d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 0x56385766d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 -- 172.21.15.69:0/3013460509 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563857680c40 con 0x56385766e4c0 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 0x56385766d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00b9c0 tx=0x7fa79c013000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93bb8977ab90461f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.458+0000 7fa78affd640 1 -- 172.21.15.69:0/3013460509 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c02a430 con 0x56385766e4c0 2024-05-09T09:19:56.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa78affd640 1 -- 172.21.15.69:0/3013460509 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa79c020910 con 0x56385766e4c0 2024-05-09T09:19:56.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa78affd640 1 -- 172.21.15.69:0/3013460509 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa79c005460 con 0x56385766e4c0 2024-05-09T09:19:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3013460509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 msgr2=0x563857678f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00b9c0 tx=0x7fa79c013000 comp rx=0 tx=0).stop 2024-05-09T09:19:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3013460509 shutdown_connections 2024-05-09T09:19:56.464 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638576841e0 0x56385766d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563857674020 0x56385767fe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3013460509 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56385766e4c0 0x563857678f10 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.459+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3013460509 >> 172.21.15.69:0/3013460509 conn(0x5638576360a0 msgr2=0x56385766cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3013460509 shutdown_connections 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3013460509 wait complete. 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 -- start start 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 Processor -- start 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x56385767db70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.460+0000 7fa7aa23c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.465 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51530/0 (socket says 172.21.15.69:51530) 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/3297926225 learned_addr learned my addr 172.21.15.69:0/3297926225 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 0x56385767f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x56385767db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a67e0 con 0x563857674020 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a7c50 con 0x56385766e4c0 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa7a01a6eb0 con 0x5638576841e0 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 0x56385767f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 msgr2=0x56385767f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 0x56385767f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 msgr2=0x56385767db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x56385767db70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56385767f6b0 con 0x563857674020 2024-05-09T09:19:56.466 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a5ac6640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 0x56385767f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x56385767db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa7a4ac4640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a930 tx=0x7fa790009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71737e75e51aacba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.461+0000 7fa788ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 10 client.0 _ll_register_callbacks cb 0x7fa790009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa788ff9640 1 -- 172.21.15.69:0/3297926225 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa790000e60 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa79c020910 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa788ff9640 1 -- 172.21.15.69:0/3297926225 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fa790004c60 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa788ff9640 1 -- 172.21.15.69:0/3297926225 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fa790005850 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 10 client.15891 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5638576eac20 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa788ff9640 1 -- 172.21.15.69:0/3297926225 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fa790004e80 con 0x563857674020 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 10 client.15891 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 10 client.15891 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:56.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 20 client.15891 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576eb500 0x5638576ed920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 4 client.15891 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.462+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x7fa79c020910 con 0x5638576eb500 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.463+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576eb500 0x5638576ed920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.463+0000 7fa788ff9640 10 client.15891 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.463+0000 7fa7a52c5640 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576eb500 0x5638576ed920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aad115725e6c8701 server_cookie=4aa186523a6e1cda in_seq=0 out_seq=0 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa788ff9640 1 -- 172.21.15.69:0/3297926225 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x7fa79c020910 con 0x5638576eb500 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa788ff9640 10 client.15891 handle_command_reply: tid=0 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 client.15891 shutdown 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 msgr2=0x56385767e0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a930 tx=0x7fa790009260 comp rx=0 tx=0).stop 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 shutdown_connections 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638576eb500 0x5638576ed920 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638576841e0 0x56385767f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.468 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563857674020 0x56385767e0b0 secure :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7fa79002a930 tx=0x7fa790009260 comp rx=0 tx=0).stop 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa788ff9640 0 client.15891 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.464+0000 7fa7aa23c740 1 --2- 172.21.15.69:0/3297926225 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56385766e4c0 0x56385767db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.465+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 >> 172.21.15.69:0/3297926225 conn(0x5638576360a0 msgr2=0x56385766d490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.465+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 shutdown_connections 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.465+0000 7fa7aa23c740 1 -- 172.21.15.69:0/3297926225 wait complete. 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.465+0000 7fa7aa23c740 20 client.15891 trim_cache size 0 max 16384 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "1ec6291f-a5ba-40cc-9035-f80f668d8d52", 2024-05-09T09:19:56.469 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-05-09T09:19:56.470 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:56.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 msgr2=0x7fa78c0ad860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.470 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 0x7fa78c0ad860 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7fa79c00bc30 tx=0x7fa79c00b390 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 msgr2=0x7fa7a019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7fa79800eaa0 tx=0x7fa79800ef60 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 shutdown_connections 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa78c0ab3a0 0x7fa78c0ad860 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa7a01075f0 0x7fa7a019f170 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa7a0104cc0 0x7fa7a0198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 --2- 172.21.15.69:0/823277246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa7a0102390 0x7fa7a0197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 >> 172.21.15.69:0/823277246 conn(0x7fa7a00fbce0 msgr2=0x7fa7a0102d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 shutdown_connections 2024-05-09T09:19:56.471 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.466+0000 7fa7a7529640 1 -- 172.21.15.69:0/823277246 wait complete. 2024-05-09T09:19:56.477 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:19:56.477 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:19:56.500 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:56 smithi069 ceph-mon[30123]: pgmap v396: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:56.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:56 smithi163 ceph-mon[34354]: pgmap v396: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 -- 172.21.15.69:0/3997838029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8101090 msgr2=0x7fc0f8101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/3997838029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8101090 0x7fc0f8101510 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0009860 tx=0x7fc0e002a340 comp rx=0 tx=0).stop 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 -- 172.21.15.69:0/3997838029 shutdown_connections 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/3997838029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101a50 0x7fc0f8108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/3997838029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f8101090 0x7fc0f8101510 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/3997838029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f81002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 -- 172.21.15.69:0/3997838029 >> 172.21.15.69:0/3997838029 conn(0x7fc0f80fb470 msgr2=0x7fc0f80fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 -- 172.21.15.69:0/3997838029 shutdown_connections 2024-05-09T09:19:56.612 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.609+0000 7fc0fd81c640 1 -- 172.21.15.69:0/3997838029 wait complete. 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 Processor -- start 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 -- start start 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f8197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f8101a50 0x7fc0f819f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810d530 con 0x7fc0f8101090 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810d3b0 con 0x7fc0f8101a50 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0fd81c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f810d6b0 con 0x7fc0f80ffeb0 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51546/0 (socket says 172.21.15.69:51546) 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0effff640 1 -- 172.21.15.69:0/1011898690 learned_addr learned my addr 172.21.15.69:0/1011898690 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.610+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f8197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f8101a50 0x7fc0f819f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 -- 172.21.15.69:0/1011898690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 msgr2=0x7fc0f8197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f8197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 -- 172.21.15.69:0/1011898690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f8101a50 msgr2=0x7fc0f819f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f8101a50 0x7fc0f819f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 -- 172.21.15.69:0/1011898690 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0f819f800 con 0x7fc0f8101090 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f8197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0effff640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0035040 tx=0x7fc0e0033d90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0fc81a640 1 -- 172.21.15.69:0/1011898690 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0037040 con 0x7fc0f8101090 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0fc81a640 1 -- 172.21.15.69:0/1011898690 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e0030850 con 0x7fc0f8101090 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0f819fa90 con 0x7fc0f8101090 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0fc81a640 1 -- 172.21.15.69:0/1011898690 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e0004920 con 0x7fc0f8101090 2024-05-09T09:19:56.614 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.611+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0f819ff80 con 0x7fc0f8101090 2024-05-09T09:19:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 Processor -- start 2024-05-09T09:19:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 -- start start 2024-05-09T09:19:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e74830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e77330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5616b5dcfd00 con 0x5616b5e74ea0 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5616b5d774c0 con 0x5616b5e6fd30 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc100607740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5616b5dcbcc0 con 0x5616b5e6f340 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0fc81a640 1 -- 172.21.15.69:0/1011898690 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc0e0004ac0 con 0x7fc0f8101090 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0f77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0f77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51944/0 (socket says 172.21.15.69:51944) 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0f77fe640 1 -- 172.21.15.69:0/3251638756 learned_addr learned my addr 172.21.15.69:0/3251638756 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e74830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f77fe640 1 -- 172.21.15.69:0/3251638756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 msgr2=0x5616b5e74830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e74830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f77fe640 1 -- 172.21.15.69:0/3251638756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 msgr2=0x5616b5e77330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.615+0000 7fc0effff640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e77330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0fc81a640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 0x7fc0e40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e77330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f77fe640 1 -- 172.21.15.69:0/3251638756 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616b5bad7d0 con 0x5616b5e6f340 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e74830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.616+0000 7fc0effff640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e77330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0fc81a640 1 -- 172.21.15.69:0/1011898690 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc0e00ef170 con 0x7fc0f8101090 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009f60 tx=0x7fc0f0009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83e2f42cc28fad30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 0x7fc0e40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0edffb640 1 -- 172.21.15.69:0/3251638756 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f001d070 con 0x5616b5e6f340 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0edffb640 1 -- 172.21.15.69:0/3251638756 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0f000dad0 con 0x5616b5e6f340 2024-05-09T09:19:56.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.617+0000 7fc0edffb640 1 -- 172.21.15.69:0/3251638756 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0f0018c30 con 0x5616b5e6f340 2024-05-09T09:19:56.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:56 smithi103 ceph-mon[34421]: pgmap v396: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 255 B/s wr, 5 op/s 2024-05-09T09:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 -- 172.21.15.69:0/3251638756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 msgr2=0x5616b5e6f760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 0x7fc0e40a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fd0 tx=0x7fc0e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc0f0009f60 tx=0x7fc0f0009f90 comp rx=0 tx=0).stop 2024-05-09T09:19:56.624 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 -- 172.21.15.69:0/3251638756 shutdown_connections 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e77330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e74830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 --2- 172.21.15.69:0/3251638756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6f340 0x5616b5e6f760 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 -- 172.21.15.69:0/3251638756 >> 172.21.15.69:0/3251638756 conn(0x5616b5e665a0 msgr2=0x5616b5e669a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 -- 172.21.15.69:0/3251638756 shutdown_connections 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.618+0000 7fc100607740 1 -- 172.21.15.69:0/3251638756 wait complete. 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.619+0000 7fc100607740 1 -- start start 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.619+0000 7fc100607740 1 Processor -- start 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.619+0000 7fc100607740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6fd30 0x5616b5e79d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51562/0 (socket says 172.21.15.69:51562) 2024-05-09T09:19:56.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/1813323298 learned_addr learned my addr 172.21.15.69:0/1813323298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0effff640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6fd30 0x5616b5e79d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5616b5e77e50 con 0x5616b5e6f340 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5616b5d8ab90 con 0x5616b5e74ea0 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5616b5e77fd0 con 0x5616b5e6fd30 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc100607740 10 client.0 _ll_register_callbacks cb 0x5616b5dc6da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/1813323298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6fd30 msgr2=0x5616b5e79d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6fd30 0x5616b5e79d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 msgr2=0x5616b5e80d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.626 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616b5ae33f0 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012e10 tx=0x7fc0e8004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e04cb48c16938cae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.620+0000 7fc0d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0f000dad0 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/1813323298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e800fba0 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/1813323298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc0e800fd40 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/1813323298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0e80035f0 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 10 client.15900 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({fsmap=25}) -- 0x5616b5aa30f0 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/1813323298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0e8003790 con 0x5616b5e6f340 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 10 client.15900 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 10 client.15900 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 20 client.15900 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:56.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5e85450 0x5616b5e878c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 4 client.15900 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.621+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc0f000dad0 con 0x5616b5e85450 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.622+0000 7fc0effff640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5e85450 0x5616b5e878c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.622+0000 7fc0d6ffd640 10 client.15900 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.622+0000 7fc0effff640 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5e85450 0x5616b5e878c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf9f7b97805b120a server_cookie=c79b89a2ca674db5 in_seq=0 out_seq=0 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/1813323298 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc0f000dad0 con 0x5616b5e85450 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc0d6ffd640 10 client.15900 handle_command_reply: tid=0 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 client.15900 shutdown 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 msgr2=0x5616b5e797e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012e10 tx=0x7fc0e8004730 comp rx=0 tx=0).stop 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 shutdown_connections 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5e85450 0x5616b5e878c0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.628 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e6fd30 0x5616b5e79d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 --2- 172.21.15.69:0/1813323298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6f340 0x5616b5e797e0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 >> 172.21.15.69:0/1813323298 conn(0x5616b5e665a0 msgr2=0x5616b5e657e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc0d6ffd640 0 client.15900 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 shutdown_connections 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 1 -- 172.21.15.69:0/1813323298 wait complete. 2024-05-09T09:19:56.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.623+0000 7fc100607740 20 client.15900 trim_cache size 0 max 16384 2024-05-09T09:19:56.654 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 Processor -- start 2024-05-09T09:19:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 -- start start 2024-05-09T09:19:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.655 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e79930 0x5616b5e69280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81aabf0 con 0x5616b5e6fd30 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a84c0 con 0x5616b5e79930 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.654+0000 7fc100607740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a6310 con 0x5616b5e74ea0 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51578/0 (socket says 172.21.15.69:51578) 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/3210424532 learned_addr learned my addr 172.21.15.69:0/3210424532 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0effff640 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e79930 0x5616b5e69280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/3210424532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e74ea0 msgr2=0x5616b5e80d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/3210424532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e79930 msgr2=0x5616b5e69280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e79930 0x5616b5e69280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.656 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 -- 172.21.15.69:0/3210424532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5616b5e663d0 con 0x5616b5e6fd30 2024-05-09T09:19:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012580 tx=0x7fc0e8023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e436c31e7db0be8b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0ed7fa640 1 -- 172.21.15.69:0/3210424532 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e803b070 con 0x5616b5e6fd30 2024-05-09T09:19:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0ed7fa640 1 -- 172.21.15.69:0/3210424532 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e8025030 con 0x5616b5e6fd30 2024-05-09T09:19:56.657 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.655+0000 7fc0ed7fa640 1 -- 172.21.15.69:0/3210424532 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e8004730 con 0x5616b5e6fd30 2024-05-09T09:19:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 -- 172.21.15.69:0/3210424532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 msgr2=0x5616b5e793f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8012580 tx=0x7fc0e8023000 comp rx=0 tx=0).stop 2024-05-09T09:19:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 -- 172.21.15.69:0/3210424532 shutdown_connections 2024-05-09T09:19:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e79930 0x5616b5e69280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.658 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e74ea0 0x5616b5e80d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 --2- 172.21.15.69:0/3210424532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e6fd30 0x5616b5e793f0 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 -- 172.21.15.69:0/3210424532 >> 172.21.15.69:0/3210424532 conn(0x5616b5e665a0 msgr2=0x5616b5e66980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.656+0000 7fc100607740 1 -- 172.21.15.69:0/3210424532 shutdown_connections 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 -- 172.21.15.69:0/3210424532 wait complete. 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 -- start start 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 Processor -- start 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e7e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.659 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 0x5616b5e7fc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.664 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a6980 con 0x5616b5e74ea0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a7df0 con 0x5616b5e6fd30 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.657+0000 7fc100607740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0f81a7050 con 0x5616b5e79930 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc100607740 10 client.0 _ll_register_callbacks cb 0x7fc0e8009ca0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51590/0 (socket says 172.21.15.69:51590) 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 -- 172.21.15.69:0/3426227508 learned_addr learned my addr 172.21.15.69:0/3426227508 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 0x5616b5e7fc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e7e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 -- 172.21.15.69:0/3426227508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 msgr2=0x5616b5e7fc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 0x5616b5e7fc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 -- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 msgr2=0x5616b5e7e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e7e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 -- 172.21.15.69:0/3426227508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5616b5e80240 con 0x5616b5e74ea0 2024-05-09T09:19:56.665 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0f77fe640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 0x5616b5e7fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0effff640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00ec940 tx=0x7fc0e0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2349cc50f6e63dc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0d6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0e00bbd50 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fc0e00b7cc0 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.658+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0e8025030 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7fc0e00bb650 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 10 client.15906 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5616b5eebd30 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0e00bfa00 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7fc0e003a070 con 0x5616b5e74ea0 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 10 client.15906 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 10 client.15906 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 20 client.15906 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:56.666 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5eec6e0 0x5616b5eeeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 4 client.15906 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x7fc0e8025030 con 0x5616b5eec6e0 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.659+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5eec6e0 0x5616b5eeeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc0f6ffd640 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5eec6e0 0x5616b5eeeb00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d53089a7769f914 server_cookie=e3ce24545644f8 in_seq=0 out_seq=0 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc0d6ffd640 10 client.15906 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc0d6ffd640 1 -- 172.21.15.69:0/3426227508 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x7fc0e8025030 con 0x5616b5eec6e0 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc0d6ffd640 10 client.15906 handle_command_reply: tid=0 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 client.15906 shutdown 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 msgr2=0x5616b5e7ec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7fc0e00ec940 tx=0x7fc0e0005f90 comp rx=0 tx=0).stop 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 shutdown_connections 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5616b5eec6e0 0x5616b5eeeb00 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5616b5e79930 0x5616b5e7fc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc0d6ffd640 0 client.15906 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:56.667 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5616b5e74ea0 0x5616b5e7ec90 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 --2- 172.21.15.69:0/3426227508 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5616b5e6fd30 0x5616b5e7e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 >> 172.21.15.69:0/3426227508 conn(0x5616b5e665a0 msgr2=0x5616b5e64f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 shutdown_connections 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.660+0000 7fc100607740 1 -- 172.21.15.69:0/3426227508 wait complete. 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc100607740 20 client.15906 trim_cache size 0 max 16384 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout: "1ec6291f-a5ba-40cc-9035-f80f668d8d52": { 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-05-09T09:19:56.668 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "1ec6291f-a5ba-40cc-9035-f80f668d8d52", 2024-05-09T09:19:56.669 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,force" 2024-05-09T09:19:56.669 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:19:56.669 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:19:56.669 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:19:56.669 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 msgr2=0x7fc0e40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 0x7fc0e40a5050 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fc0e8009fd0 tx=0x7fc0e8005f90 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 msgr2=0x7fc0f8198120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7fc0e0035040 tx=0x7fc0e0033d90 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 shutdown_connections 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0e40a2b90 0x7fc0e40a5050 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0f8101a50 0x7fc0f819f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0f8101090 0x7fc0f8198120 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 --2- 172.21.15.69:0/1011898690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0f80ffeb0 0x7fc0f8197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 >> 172.21.15.69:0/1011898690 conn(0x7fc0f80fb470 msgr2=0x7fc0f81044a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 shutdown_connections 2024-05-09T09:19:56.670 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:19:56.661+0000 7fc0fd81c640 1 -- 172.21.15.69:0/1011898690 wait complete. 2024-05-09T09:19:56.671 INFO:tasks.cephfs.filesystem:scrub status for tag:1ec6291f-a5ba-40cc-9035-f80f668d8d52 - {'path': '/', 'tag': '1ec6291f-a5ba-40cc-9035-f80f668d8d52', 'options': 'recursive,force'} 2024-05-09T09:19:57.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 -- 172.21.15.69:0/826657837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78100790 msgr2=0x7eff78100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/826657837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78100790 0x7eff78100bb0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7eff6c009900 tx=0x7eff6c02a410 comp rx=0 tx=0).stop 2024-05-09T09:19:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 -- 172.21.15.69:0/826657837 shutdown_connections 2024-05-09T09:19:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/826657837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78102340 0x7eff78108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/826657837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 0x7eff78101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/826657837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78100790 0x7eff78100bb0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 -- 172.21.15.69:0/826657837 >> 172.21.15.69:0/826657837 conn(0x7eff780fbd00 msgr2=0x7eff780fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 -- 172.21.15.69:0/826657837 shutdown_connections 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.116+0000 7eff7ef9e640 1 -- 172.21.15.69:0/826657837 wait complete. 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 Processor -- start 2024-05-09T09:19:57.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 -- start start 2024-05-09T09:19:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 0x7eff78196a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 0x7eff78196f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff7810c410 con 0x7eff78102340 2024-05-09T09:19:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff7810c290 con 0x7eff78101980 2024-05-09T09:19:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7ef9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff7810c590 con 0x7eff78100790 2024-05-09T09:19:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7d53b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7d53b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51608/0 (socket says 172.21.15.69:51608) 2024-05-09T09:19:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff7d53b640 1 -- 172.21.15.69:0/944405547 learned_addr learned my addr 172.21.15.69:0/944405547 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.117+0000 7eff77fff640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 0x7eff78196f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 0x7eff78196a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 -- 172.21.15.69:0/944405547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 msgr2=0x7eff78196a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 0x7eff78196a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 -- 172.21.15.69:0/944405547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 msgr2=0x7eff78196f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 0x7eff78196f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 -- 172.21.15.69:0/944405547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7eff7819e650 con 0x7eff78102340 2024-05-09T09:19:57.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 0x7eff78196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7d53b640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7eff7000eb10 tx=0x7eff7000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff75ffb640 1 -- 172.21.15.69:0/944405547 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70004b20 con 0x7eff78102340 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff75ffb640 1 -- 172.21.15.69:0/944405547 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff70004cc0 con 0x7eff78102340 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff75ffb640 1 -- 172.21.15.69:0/944405547 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff70010760 con 0x7eff78102340 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7eff78068f40 con 0x7eff78102340 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.118+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7eff78069490 con 0x7eff78102340 2024-05-09T09:19:57.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.121+0000 7eff81d89740 1 Processor -- start 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff75ffb640 1 -- 172.21.15.69:0/944405547 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7eff70005020 con 0x7eff78102340 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 -- start start 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a17670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17c40 0x5623e2a1c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5623e297b340 con 0x5623e2a1cdb0 2024-05-09T09:19:57.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5623e2923290 con 0x5623e2a17c40 2024-05-09T09:19:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff81d89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5623e2977300 con 0x5623e2a17250 2024-05-09T09:19:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff77fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17c40 0x5623e2a1c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff77fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51616/0 (socket says 172.21.15.69:51616) 2024-05-09T09:19:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff77fff640 1 -- 172.21.15.69:0/1044531584 learned_addr learned my addr 172.21.15.69:0/1044531584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.122+0000 7eff77fff640 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a17670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff75ffb640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 0x7eff680a7300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 -- 172.21.15.69:0/1044531584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 msgr2=0x5623e2a17670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a17670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 -- 172.21.15.69:0/1044531584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17c40 msgr2=0x5623e2a1c740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17c40 0x5623e2a1c740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 -- 172.21.15.69:0/1044531584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e29478e0 con 0x5623e2a1cdb0 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff7d53b640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 0x7eff680a7300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff77fff640 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff640058c0 tx=0x7eff64005850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=204666b95aeb00e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff75ffb640 1 -- 172.21.15.69:0/944405547 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7eff700cfb50 con 0x7eff78102340 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff62ffd640 1 -- 172.21.15.69:0/1044531584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64004030 con 0x5623e2a1cdb0 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff62ffd640 1 -- 172.21.15.69:0/1044531584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64005500 con 0x5623e2a1cdb0 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.123+0000 7eff62ffd640 1 -- 172.21.15.69:0/1044531584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff64004ce0 con 0x5623e2a1cdb0 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- 172.21.15.69:0/1044531584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 msgr2=0x5623e2a1f240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7eff640058c0 tx=0x7eff64005850 comp rx=0 tx=0).stop 2024-05-09T09:19:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff7d53b640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 0x7eff680a7300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7eff7009bfc0 tx=0x7eff70066c00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- 172.21.15.69:0/1044531584 shutdown_connections 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a1f240 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17c40 0x5623e2a1c740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 --2- 172.21.15.69:0/1044531584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a17670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- 172.21.15.69:0/1044531584 >> 172.21.15.69:0/1044531584 conn(0x5623e2a128c0 msgr2=0x5623e2a12cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- 172.21.15.69:0/1044531584 shutdown_connections 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- 172.21.15.69:0/1044531584 wait complete. 2024-05-09T09:19:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.124+0000 7eff81d89740 1 -- start start 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 Processor -- start 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17250 0x5623e2a214c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 0x5623e2a28a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5623e2a1fd60 con 0x5623e2a17c40 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5623e2936790 con 0x5623e2a17250 2024-05-09T09:19:57.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5623e2a1fee0 con 0x5623e2a1cdb0 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff81d89740 10 client.0 _ll_register_callbacks cb 0x5623e2971a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51626/0 (socket says 172.21.15.69:51626) 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 -- 172.21.15.69:0/1058138008 learned_addr learned my addr 172.21.15.69:0/1058138008 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff77fff640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17250 0x5623e2a214c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff77fff640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 0x5623e2a28a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 -- 172.21.15.69:0/1058138008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 msgr2=0x5623e2a28a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 0x5623e2a28a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 -- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17250 msgr2=0x5623e2a214c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17250 0x5623e2a214c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff7cd3a640 1 -- 172.21.15.69:0/1058138008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e268f210 con 0x5623e2a17c40 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.125+0000 7eff77fff640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 0x5623e2a28a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7eff6c005940 tx=0x7eff6c0318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86bbb2bafeb9b0cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff57fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:57.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c035040 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7eff6c004240 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff64005500 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7eff6c004440 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff81d89740 10 client.15918 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5623e275b620 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.126+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff64005500 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7eff6c004be0 con 0x5623e2a17c40 2024-05-09T09:19:57.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 10 client.15918 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 10 client.15918 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 20 client.15918 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a313a0 0x5623e2a337b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 4 client.15918 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5623e2988a70 con 0x5623e2a313a0 2024-05-09T09:19:57.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff77fff640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a313a0 0x5623e2a337b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff57fff640 10 client.15918 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.127+0000 7eff77fff640 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a313a0 0x5623e2a337b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=700b28ee992d9b9d server_cookie=5c5de46d3f09757f in_seq=0 out_seq=0 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff57fff640 1 -- 172.21.15.69:0/1058138008 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5623e2988a70 con 0x5623e2a313a0 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff57fff640 10 client.15918 handle_command_reply: tid=0 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 client.15918 shutdown 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 msgr2=0x5623e2a21a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7eff6c005940 tx=0x7eff6c0318e0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 shutdown_connections 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a313a0 0x5623e2a337b0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a1cdb0 0x5623e2a28a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff57fff640 0 client.15918 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17c40 0x5623e2a21a00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 --2- 172.21.15.69:0/1058138008 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a17250 0x5623e2a214c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 >> 172.21.15.69:0/1058138008 conn(0x5623e2a128c0 msgr2=0x5623e2a11a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 shutdown_connections 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 1 -- 172.21.15.69:0/1058138008 wait complete. 2024-05-09T09:19:57.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.128+0000 7eff81d89740 20 client.15918 trim_cache size 0 max 16384 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.143+0000 7eff81d89740 1 Processor -- start 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.143+0000 7eff81d89740 1 -- start start 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a0d620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a12ca0 0x5623e2a18e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff781aab70 con 0x5623e2a1cdb0 2024-05-09T09:19:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff781a8440 con 0x5623e2a12ca0 2024-05-09T09:19:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff81d89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff781a6290 con 0x5623e2a17250 2024-05-09T09:19:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51634/0 (socket says 172.21.15.69:51634) 2024-05-09T09:19:57.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 -- 172.21.15.69:0/3567405740 learned_addr learned my addr 172.21.15.69:0/3567405740 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff77fff640 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a0d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff77fff640 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a12ca0 0x5623e2a18e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 -- 172.21.15.69:0/3567405740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 msgr2=0x5623e2a0d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a0d620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 -- 172.21.15.69:0/3567405740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a12ca0 msgr2=0x5623e2a18e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a12ca0 0x5623e2a18e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 -- 172.21.15.69:0/3567405740 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5623e2a33e30 con 0x5623e2a1cdb0 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.144+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7eff6c006010 tx=0x7eff6c005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcd123fc99e2b399 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.145+0000 7eff627fc640 1 -- 172.21.15.69:0/3567405740 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c035070 con 0x5623e2a1cdb0 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.145+0000 7eff627fc640 1 -- 172.21.15.69:0/3567405740 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6c003a30 con 0x5623e2a1cdb0 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.145+0000 7eff627fc640 1 -- 172.21.15.69:0/3567405740 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c005260 con 0x5623e2a1cdb0 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.145+0000 7eff81d89740 1 -- 172.21.15.69:0/3567405740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 msgr2=0x5623e2a20280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.145+0000 7eff81d89740 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7eff6c006010 tx=0x7eff6c005970 comp rx=0 tx=0).stop 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 -- 172.21.15.69:0/3567405740 shutdown_connections 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a12ca0 0x5623e2a18e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a1cdb0 0x5623e2a20280 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 --2- 172.21.15.69:0/3567405740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a17250 0x5623e2a0d620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 -- 172.21.15.69:0/3567405740 >> 172.21.15.69:0/3567405740 conn(0x5623e2a128c0 msgr2=0x5623e2a1c4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 -- 172.21.15.69:0/3567405740 shutdown_connections 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 -- 172.21.15.69:0/3567405740 wait complete. 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 -- start start 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.146+0000 7eff81d89740 1 Processor -- start 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a12ca0 0x5623e2a251d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 0x5623e2a26720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7eff781a6900 con 0x5623e2a17250 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7eff781a7d70 con 0x5623e2a1cdb0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7eff781a6fd0 con 0x5623e2a12ca0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff7cd3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51650/0 (socket says 172.21.15.69:51650) 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff7cd3a640 1 -- 172.21.15.69:0/2757097745 learned_addr learned my addr 172.21.15.69:0/2757097745 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff77fff640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a12ca0 0x5623e2a251d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff81d89740 10 client.0 _ll_register_callbacks cb 0x7eff700093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.147+0000 7eff77fff640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 0x5623e2a26720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 -- 172.21.15.69:0/2757097745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a12ca0 msgr2=0x5623e2a251d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a12ca0 0x5623e2a251d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 -- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 msgr2=0x5623e2a26720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 0x5623e2a26720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 -- 172.21.15.69:0/2757097745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5623e2a26d10 con 0x5623e2a17250 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff77fff640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 0x5623e2a26720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff7cd3a640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7eff6c02a930 tx=0x7eff6c031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaa3370f585d9534 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff57fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7eff6c035030 con 0x5623e2a17250 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7eff6c003a30 con 0x5623e2a17250 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7eff6c005640 con 0x5623e2a17250 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7eff6c069050 con 0x5623e2a17250 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff81d89740 10 client.15924 fetch_fsmap learned FSMap version 25 2024-05-09T09:19:57.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x5623e2a28000 con 0x5623e2a17250 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.148+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7eff6c003a30 con 0x5623e2a17250 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7eff6c0046c0 con 0x5623e2a17250 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 10 client.15924 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 10 client.15924 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 20 client.15924 populate_metadata read hostname 'smithi069' 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a97d90 0x5623e2a9a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 4 client.15924 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5623e2982dc0 con 0x5623e2a97d90 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff77fff640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a97d90 0x5623e2a9a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.149+0000 7eff57fff640 10 client.15924 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.150+0000 7eff77fff640 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a97d90 0x5623e2a9a1b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=436bf2eaba83830a server_cookie=33e315e5b3d9bf02 in_seq=0 out_seq=0 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff57fff640 1 -- 172.21.15.69:0/2757097745 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5623e2982dc0 con 0x5623e2a97d90 2024-05-09T09:19:57.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff57fff640 10 client.15924 handle_command_reply: tid=0 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 client.15924 shutdown 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 msgr2=0x5623e2a25710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7eff6c02a930 tx=0x7eff6c031400 comp rx=0 tx=0).stop 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 shutdown_connections 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5623e2a1cdb0 0x5623e2a26720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5623e2a17250 0x5623e2a25710 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5623e2a97d90 0x5623e2a9a1b0 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 --2- 172.21.15.69:0/2757097745 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5623e2a12ca0 0x5623e2a251d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.153+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 >> 172.21.15.69:0/2757097745 conn(0x5623e2a128c0 msgr2=0x5623e2a20280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff57fff640 0 client.15924 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 shutdown_connections 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff81d89740 1 -- 172.21.15.69:0/2757097745 wait complete. 2024-05-09T09:19:57.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff81d89740 20 client.15924 trim_cache size 0 max 16384 2024-05-09T09:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:19:57.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 msgr2=0x7eff680a7300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 0x7eff680a7300 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7eff7009bfc0 tx=0x7eff70066c00 comp rx=0 tx=0).stop 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 msgr2=0x7eff7819dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.154+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7eff7000eb10 tx=0x7eff7000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 shutdown_connections 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7eff680a4e40 0x7eff680a7300 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7eff78102340 0x7eff7819dfe0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7eff78101980 0x7eff78196f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 --2- 172.21.15.69:0/944405547 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7eff78100790 0x7eff78196a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 >> 172.21.15.69:0/944405547 conn(0x7eff780fbd00 msgr2=0x7eff78104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 shutdown_connections 2024-05-09T09:19:57.175 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:19:57.155+0000 7eff7ef9e640 1 -- 172.21.15.69:0/944405547 wait complete. 2024-05-09T09:19:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:57 smithi069 ceph-mon[30123]: scrub queued for path: / 2024-05-09T09:19:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:57 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:57 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:19:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:57 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:57 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:19:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:57 smithi163 ceph-mon[34354]: scrub queued for path: / 2024-05-09T09:19:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:57 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:57 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:19:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:57 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:57 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:19:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:57 smithi103 ceph-mon[34421]: scrub queued for path: / 2024-05-09T09:19:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:57 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:57 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:19:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:57 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:19:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:57 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:19:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:58 smithi069 ceph-mon[30123]: pgmap v397: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:58 smithi163 ceph-mon[34354]: pgmap v397: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:58 smithi103 ceph-mon[34421]: pgmap v397: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 7 op/s 2024-05-09T09:19:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:19:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:19:59 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:19:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:19:59 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:19:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:19:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:19:59 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:20:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:00 smithi069 ceph-mon[30123]: pgmap v398: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-09T09:20:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:20:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:20:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:20:00.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:00 smithi069 ceph-mon[30123]: overall HEALTH_OK 2024-05-09T09:20:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:00 smithi163 ceph-mon[34354]: pgmap v398: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-09T09:20:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:20:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:20:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:20:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:00 smithi163 ceph-mon[34354]: overall HEALTH_OK 2024-05-09T09:20:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:00 smithi103 ceph-mon[34421]: pgmap v398: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 6 op/s 2024-05-09T09:20:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:20:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:20:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:20:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:00 smithi103 ceph-mon[34421]: overall HEALTH_OK 2024-05-09T09:20:01.349 INFO:tasks.workunit.client.0.smithi069.stderr:Note: switching to 'be85c202969d25af6e130678e7d73a20e9c4ca11'. 2024-05-09T09:20:01.349 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.349 INFO:tasks.workunit.client.0.smithi069.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-05-09T09:20:01.349 INFO:tasks.workunit.client.0.smithi069.stderr:changes and commit them, and you can discard any commits you make in this 2024-05-09T09:20:01.349 INFO:tasks.workunit.client.0.smithi069.stderr:state without impacting any branches by switching back to a branch. 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr:If you want to create a new branch to retain commits you create, you may 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr:do so (now or later) by using -c with the switch command. Example: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: git switch -c 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr:Or undo this operation with: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: git switch - 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr: 2024-05-09T09:20:01.350 INFO:tasks.workunit.client.0.smithi069.stderr:HEAD is now at be85c202969 Merge pull request #57242 from cbodley/wip-65767-squid 2024-05-09T09:20:01.354 DEBUG:teuthology.orchestra.run.smithi069:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-05-09T09:20:01.378 INFO:tasks.workunit.client.0.smithi069.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-05-09T09:20:01.380 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T09:20:01.380 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-05-09T09:20:02.103 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-05-09T09:20:02.157 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-05-09T09:20:02.198 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-05-09T09:20:02.198 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T09:20:02.198 INFO:tasks.workunit.client.0.smithi069.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-05-09T09:20:02.237 INFO:tasks.workunit.client.0.smithi069.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-05-09T09:20:02.240 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:20:02.240 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-05-09T09:20:02.297 INFO:tasks.workunit:Running workunits matching suites/dbench.sh on client.0... 2024-05-09T09:20:02.299 INFO:tasks.workunit:Running workunit suites/dbench.sh... 2024-05-09T09:20:02.299 DEBUG:teuthology.orchestra.run.smithi069:workunit test suites/dbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=be85c202969d25af6e130678e7d73a20e9c4ca11 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/dbench.sh 2024-05-09T09:20:02.360 INFO:tasks.workunit.client.0.smithi069.stderr:+ dbench 1 2024-05-09T09:20:02.366 INFO:tasks.workunit.client.0.smithi069.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-09T09:20:02.367 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:20:02.367 INFO:tasks.workunit.client.0.smithi069.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-09T09:20:02.369 INFO:tasks.workunit.client.0.smithi069.stdout:0 of 1 processes prepared for launch 0 sec 2024-05-09T09:20:02.469 INFO:tasks.workunit.client.0.smithi069.stdout:1 of 1 processes prepared for launch 0 sec 2024-05-09T09:20:02.469 INFO:tasks.workunit.client.0.smithi069.stdout:releasing clients 2024-05-09T09:20:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:02 smithi069 ceph-mon[30123]: pgmap v399: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-09T09:20:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:02 smithi163 ceph-mon[34354]: pgmap v399: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-09T09:20:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:02 smithi103 ceph-mon[34421]: pgmap v399: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 0 B/s wr, 6 op/s 2024-05-09T09:20:03.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 3175 34.12 MB/sec warmup 1 sec latency 9.996 ms 2024-05-09T09:20:04.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 5954 23.55 MB/sec warmup 2 sec latency 13.307 ms 2024-05-09T09:20:04.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:04 smithi069 ceph-mon[30123]: pgmap v400: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T09:20:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:04 smithi163 ceph-mon[34354]: pgmap v400: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T09:20:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:04 smithi103 ceph-mon[34421]: pgmap v400: 129 pgs: 129 active+clean; 668 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 255 B/s wr, 8 op/s 2024-05-09T09:20:05.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 8310 20.43 MB/sec warmup 3 sec latency 12.539 ms 2024-05-09T09:20:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:05 smithi103 ceph-mon[34421]: pgmap v401: 129 pgs: 129 active+clean; 929 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 KiB/s wr, 8 op/s 2024-05-09T09:20:05.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:05 smithi069 ceph-mon[30123]: pgmap v401: 129 pgs: 129 active+clean; 929 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 KiB/s wr, 8 op/s 2024-05-09T09:20:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:05 smithi163 ceph-mon[34354]: pgmap v401: 129 pgs: 129 active+clean; 929 KiB data, 345 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 26 KiB/s wr, 8 op/s 2024-05-09T09:20:06.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 11246 19.58 MB/sec warmup 4 sec latency 18.587 ms 2024-05-09T09:20:07.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 14431 19.42 MB/sec warmup 5 sec latency 16.751 ms 2024-05-09T09:20:08.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 17109 18.32 MB/sec warmup 6 sec latency 12.179 ms 2024-05-09T09:20:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:08 smithi069 ceph-mon[30123]: pgmap v402: 129 pgs: 129 active+clean; 5.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 424 KiB/s wr, 66 op/s 2024-05-09T09:20:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:08 smithi163 ceph-mon[34354]: pgmap v402: 129 pgs: 129 active+clean; 5.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 424 KiB/s wr, 66 op/s 2024-05-09T09:20:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:08 smithi103 ceph-mon[34421]: pgmap v402: 129 pgs: 129 active+clean; 5.0 MiB data, 362 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 424 KiB/s wr, 66 op/s 2024-05-09T09:20:09.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 19303 17.39 MB/sec warmup 7 sec latency 14.831 ms 2024-05-09T09:20:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 -- 172.21.15.69:0/3261859759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500100790 msgr2=0x7f9500100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 --2- 172.21.15.69:0/3261859759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500100790 0x7f9500100bb0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009900 tx=0x7f94f402a410 comp rx=0 tx=0).stop 2024-05-09T09:20:10.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 -- 172.21.15.69:0/3261859759 shutdown_connections 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 --2- 172.21.15.69:0/3261859759 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500102340 0x7f9500108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 --2- 172.21.15.69:0/3261859759 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 0x7f9500101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 --2- 172.21.15.69:0/3261859759 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500100790 0x7f9500100bb0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 -- 172.21.15.69:0/3261859759 >> 172.21.15.69:0/3261859759 conn(0x7f95000fbd00 msgr2=0x7f95000fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 -- 172.21.15.69:0/3261859759 shutdown_connections 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.248+0000 7f95070b3640 1 -- 172.21.15.69:0/3261859759 wait complete. 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 Processor -- start 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 -- start start 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500100790 0x7f9500104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 0x7f9500102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f950010c400 con 0x7f9500102340 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f950010c280 con 0x7f9500100790 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f95070b3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f950010c580 con 0x7f9500101980 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53312/0 (socket says 172.21.15.69:53312) 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 -- 172.21.15.69:0/3908696420 learned_addr learned my addr 172.21.15.69:0/3908696420 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9504e4f640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500100790 0x7f9500104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f94fffff640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 0x7f9500102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 -- 172.21.15.69:0/3908696420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 msgr2=0x7f9500102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 0x7f9500102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 -- 172.21.15.69:0/3908696420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500100790 msgr2=0x7f9500104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500100790 0x7f9500104720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 -- 172.21.15.69:0/3908696420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9500103b30 con 0x7f9500102340 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.249+0000 7f9505650640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f94f800eb10 tx=0x7f94f800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.250+0000 7f94fdffb640 1 -- 172.21.15.69:0/3908696420 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8004b20 con 0x7f9500102340 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.250+0000 7f94fdffb640 1 -- 172.21.15.69:0/3908696420 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f8004cc0 con 0x7f9500102340 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.250+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f95001a1cb0 con 0x7f9500102340 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.250+0000 7f94fdffb640 1 -- 172.21.15.69:0/3908696420 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f8010760 con 0x7f9500102340 2024-05-09T09:20:10.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.250+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f95001a21b0 con 0x7f9500102340 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.252+0000 7f94fdffb640 1 -- 172.21.15.69:0/3908696420 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f94f8005020 con 0x7f9500102340 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.252+0000 7f9509dc3740 1 Processor -- start 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.252+0000 7f9509dc3740 1 -- start start 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e83740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e83db0 0x55a223e86240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a223de2340 con 0x55a223e7ec40 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a223d8a290 con 0x55a223e7e250 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9509dc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a223dde300 con 0x55a223e83db0 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e83740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:38942/0 (socket says 172.21.15.69:38942) 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fffff640 1 -- 172.21.15.69:0/3953598567 learned_addr learned my addr 172.21.15.69:0/3953598567 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fffff640 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e83db0 0x55a223e86240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fdffb640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 0x7f94f00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 -- 172.21.15.69:0/3953598567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e83db0 msgr2=0x55a223e86240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e83db0 0x55a223e86240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9505650640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 0x7f94f00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 -- 172.21.15.69:0/3953598567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 msgr2=0x55a223e7e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 -- 172.21.15.69:0/3953598567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a223dae8e0 con 0x55a223e7ec40 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fffff640 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:10.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94fdffb640 1 -- 172.21.15.69:0/3908696420 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f94f80cf9b0 con 0x7f9500102340 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f9504e4f640 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e83740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009230 tx=0x7f94f4005e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8008533744ff81c5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.253+0000 7f94eaffd640 1 -- 172.21.15.69:0/3953598567 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4040070 con 0x55a223e7ec40 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f94eaffd640 1 -- 172.21.15.69:0/3953598567 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f4004030 con 0x55a223e7ec40 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f94eaffd640 1 -- 172.21.15.69:0/3953598567 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4035040 con 0x55a223e7ec40 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9505650640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 0x7f94f00a50f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94f806ef20 tx=0x7f94f806ef50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 -- 172.21.15.69:0/3953598567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 msgr2=0x55a223e83740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e83740 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7f94f4009230 tx=0x7f94f4005e20 comp rx=0 tx=0).stop 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 -- 172.21.15.69:0/3953598567 shutdown_connections 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e83db0 0x55a223e86240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e83740 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 --2- 172.21.15.69:0/3953598567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e7e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 -- 172.21.15.69:0/3953598567 >> 172.21.15.69:0/3953598567 conn(0x55a223e798c0 msgr2=0x55a223e79cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 -- 172.21.15.69:0/3953598567 shutdown_connections 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.254+0000 7f9509dc3740 1 -- 172.21.15.69:0/3953598567 wait complete. 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 -- start start 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 Processor -- start 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e88970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e83db0 0x55a223e8f9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a223e86d00 con 0x55a223e83db0 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a223d9d790 con 0x55a223e7e250 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a223e86e80 con 0x55a223e7ec40 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9509dc3740 10 client.0 _ll_register_callbacks cb 0x55a223dd8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:10.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f94fffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e83db0 0x55a223e8f9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f94fffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e88970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f94fffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:38944/0 (socket says 172.21.15.69:38944) 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f94fffff640 1 -- 172.21.15.69:0/2057657651 learned_addr learned my addr 172.21.15.69:0/2057657651 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 -- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 msgr2=0x55a223e88430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 -- 172.21.15.69:0/2057657651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e83db0 msgr2=0x55a223e8f9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e83db0 0x55a223e8f9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f9504e4f640 1 -- 172.21.15.69:0/2057657651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a223af6210 con 0x55a223e7ec40 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.255+0000 7f94fffff640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:10.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9504e4f640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e88970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55a223e7a830 tx=0x7f94f402a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1e73bef0882ae58e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4052060 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f4004030 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f94f40316b0 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f94f4059050 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 10 client.25351 fetch_fsmap learned FSMap version 25 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a223bc2620 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f4004030 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f94f40035f0 con 0x55a223e7ec40 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 10 client.25351 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 10 client.25351 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 20 client.25351 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223e93900 0x55a223e95cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 4 client.25351 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.256+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a223defa70 con 0x55a223e93900 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.257+0000 7f94fffff640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223e93900 0x55a223e95cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.257+0000 7f94dffff640 10 client.25351 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.257+0000 7f94fffff640 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223e93900 0x55a223e95cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0454e8d4b3e93f0 server_cookie=7d8a89f32ea543e8 in_seq=0 out_seq=0 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f94dffff640 1 -- 172.21.15.69:0/2057657651 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a223defa70 con 0x55a223e93900 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f94dffff640 10 client.25351 handle_command_reply: tid=0 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 client.25351 shutdown 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 msgr2=0x55a223e88970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e88970 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x55a223e7a830 tx=0x7f94f402a410 comp rx=0 tx=0).stop 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 shutdown_connections 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e83db0 0x55a223e8f9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223e93900 0x55a223e95cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e88970 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 --2- 172.21.15.69:0/2057657651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e7e250 0x55a223e88430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f94dffff640 0 client.25351 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 >> 172.21.15.69:0/2057657651 conn(0x55a223e798c0 msgr2=0x55a223e78a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 shutdown_connections 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 1 -- 172.21.15.69:0/2057657651 wait complete. 2024-05-09T09:20:10.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.258+0000 7f9509dc3740 20 client.25351 trim_cache size 0 max 16384 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.273+0000 7f9509dc3740 1 Processor -- start 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.273+0000 7f9509dc3740 1 -- start start 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7e250 0x55a223e7c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223e7cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95001aee90 con 0x55a223e7ec40 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95001ac760 con 0x55a223e83db0 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9509dc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95001aa5b0 con 0x55a223e7e250 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53336/0 (socket says 172.21.15.69:53336) 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 -- 172.21.15.69:0/4156693119 learned_addr learned my addr 172.21.15.69:0/4156693119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94fffff640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7e250 0x55a223e7c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94fffff640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223e7cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 -- 172.21.15.69:0/4156693119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7e250 msgr2=0x55a223e7c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7e250 0x55a223e7c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 -- 172.21.15.69:0/4156693119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 msgr2=0x55a223e7cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223e7cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 -- 172.21.15.69:0/4156693119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a223e96060 con 0x55a223e7ec40 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94fffff640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223e7cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94f4003b60 tx=0x7f94f4009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c41b2db29841287 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94ea7fc640 1 -- 172.21.15.69:0/4156693119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4052070 con 0x55a223e7ec40 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94ea7fc640 1 -- 172.21.15.69:0/4156693119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f4005520 con 0x55a223e7ec40 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.274+0000 7f94ea7fc640 1 -- 172.21.15.69:0/4156693119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94f4035070 con 0x55a223e7ec40 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- 172.21.15.69:0/4156693119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 msgr2=0x55a223e8f970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7f94f4003b60 tx=0x7f94f4009780 comp rx=0 tx=0).stop 2024-05-09T09:20:10.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- 172.21.15.69:0/4156693119 shutdown_connections 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223e7cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7ec40 0x55a223e8f970 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4156693119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7e250 0x55a223e7c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- 172.21.15.69:0/4156693119 >> 172.21.15.69:0/4156693119 conn(0x55a223e798c0 msgr2=0x55a223e7fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- 172.21.15.69:0/4156693119 shutdown_connections 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- 172.21.15.69:0/4156693119 wait complete. 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 -- start start 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.275+0000 7f9509dc3740 1 Processor -- start 2024-05-09T09:20:10.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e8cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223ef7d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f95001aac20 con 0x55a223e7e250 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f95001ac090 con 0x55a223e83db0 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f95001ab2f0 con 0x55a223e7ec40 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9509dc3740 10 client.0 _ll_register_callbacks cb 0x7f94ec009ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53344/0 (socket says 172.21.15.69:53344) 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 -- 172.21.15.69:0/4014501823 learned_addr learned my addr 172.21.15.69:0/4014501823 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223ef7d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e8cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 -- 172.21.15.69:0/4014501823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 msgr2=0x55a223e8cde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e8cde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 -- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 msgr2=0x55a223ef7d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223ef7d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 -- 172.21.15.69:0/4014501823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a223ef82f0 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94fffff640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x55a223e96460 tx=0x7f94f4000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fde0d5faf081221b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.276+0000 7f94dffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f94ec025e60 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7f94ec0118d0 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f94f4005520 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=25) ==== 24+0+0 (secure 0 0 0) 0x7f94ec035050 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 10 client.15942 fetch_fsmap learned FSMap version 25 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=25,monmap=4+}) -- 0x55a223ef9650 con 0x55a223e7e250 2024-05-09T09:20:10.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f94f4005520 con 0x55a223e7e250 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 25) ==== 8346+0+0 (secure 0 0 0) 0x7f94ec023450 con 0x55a223e7e250 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 10 client.15942 fetch_fsmap finished waiting for FSMap version 25 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 10 client.15942 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 20 client.15942 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223efa000 0x55a223efc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 4 client.15942 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a223de9dc0 con 0x55a223efa000 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.277+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223efa000 0x55a223efc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.278+0000 7f94dffff640 10 client.15942 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:10.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.278+0000 7f9504e4f640 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223efa000 0x55a223efc420 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b76ada1eb28d71f server_cookie=7cd64693bc0feb14 in_seq=0 out_seq=0 2024-05-09T09:20:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f94dffff640 1 -- 172.21.15.69:0/4014501823 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a223de9dc0 con 0x55a223efa000 2024-05-09T09:20:10.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f94dffff640 10 client.15942 handle_command_reply: tid=0 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 client.15942 shutdown 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 msgr2=0x55a223e8c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x55a223e96460 tx=0x7f94f4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 shutdown_connections 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a223e83db0 0x55a223ef7d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a223efa000 0x55a223efc420 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a223e7ec40 0x55a223e8cde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f94dffff640 0 client.15942 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 --2- 172.21.15.69:0/4014501823 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a223e7e250 0x55a223e8c8a0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 >> 172.21.15.69:0/4014501823 conn(0x55a223e798c0 msgr2=0x55a223e7c870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 shutdown_connections 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 1 -- 172.21.15.69:0/4014501823 wait complete. 2024-05-09T09:20:10.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.432+0000 7f9509dc3740 20 client.15942 trim_cache size 0 max 16384 2024-05-09T09:20:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:20:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:20:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:20:10.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:20:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 msgr2=0x7f94f00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 0x7f94f00a50f0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f94f806ef20 tx=0x7f94f806ef50 comp rx=0 tx=0).stop 2024-05-09T09:20:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 msgr2=0x7f95001032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:10.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7f94f800eb10 tx=0x7f94f800c2d0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 shutdown_connections 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f94f00a2c30 0x7f94f00a50f0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9500102340 0x7f95001032a0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9500101980 0x7f9500102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 --2- 172.21.15.69:0/3908696420 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9500100790 0x7f9500104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 >> 172.21.15.69:0/3908696420 conn(0x7f95000fbd00 msgr2=0x7f95000fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 shutdown_connections 2024-05-09T09:20:10.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:10.433+0000 7f95070b3640 1 -- 172.21.15.69:0/3908696420 wait complete. 2024-05-09T09:20:10.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 21767 16.96 MB/sec warmup 8 sec latency 101.596 ms 2024-05-09T09:20:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:10 smithi069 ceph-mon[30123]: pgmap v403: 129 pgs: 129 active+clean; 6.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 634 KiB/s wr, 76 op/s 2024-05-09T09:20:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:10 smithi163 ceph-mon[34354]: pgmap v403: 129 pgs: 129 active+clean; 6.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 634 KiB/s wr, 76 op/s 2024-05-09T09:20:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:10 smithi103 ceph-mon[34421]: pgmap v403: 129 pgs: 129 active+clean; 6.6 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 269 KiB/s rd, 634 KiB/s wr, 76 op/s 2024-05-09T09:20:11.469 INFO:tasks.workunit.client.0.smithi069.stdout: 1 24950 17.05 MB/sec warmup 9 sec latency 15.677 ms 2024-05-09T09:20:12.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 27316 16.20 MB/sec warmup 10 sec latency 15.471 ms 2024-05-09T09:20:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:12 smithi069 ceph-mon[30123]: pgmap v404: 129 pgs: 129 active+clean; 7.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 713 KiB/s wr, 89 op/s 2024-05-09T09:20:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:12 smithi163 ceph-mon[34354]: pgmap v404: 129 pgs: 129 active+clean; 7.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 713 KiB/s wr, 89 op/s 2024-05-09T09:20:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:12 smithi103 ceph-mon[34421]: pgmap v404: 129 pgs: 129 active+clean; 7.3 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 268 KiB/s rd, 713 KiB/s wr, 89 op/s 2024-05-09T09:20:13.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29133 15.84 MB/sec warmup 11 sec latency 19.462 ms 2024-05-09T09:20:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:13 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:13 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:13 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:14.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29138 14.52 MB/sec warmup 12 sec latency 980.060 ms 2024-05-09T09:20:14.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:14 smithi069 ceph-mon[30123]: pgmap v405: 129 pgs: 129 active+clean; 7.5 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.1 MiB/s wr, 196 op/s 2024-05-09T09:20:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:14 smithi163 ceph-mon[34354]: pgmap v405: 129 pgs: 129 active+clean; 7.5 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.1 MiB/s wr, 196 op/s 2024-05-09T09:20:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:14 smithi103 ceph-mon[34421]: pgmap v405: 129 pgs: 129 active+clean; 7.5 MiB data, 386 MiB used, 1.0 TiB / 1.0 TiB avail; 519 KiB/s rd, 1.1 MiB/s wr, 196 op/s 2024-05-09T09:20:15.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29139 13.40 MB/sec warmup 13 sec latency 1740.297 ms 2024-05-09T09:20:16.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29139 12.44 MB/sec warmup 14 sec latency 1239.922 ms 2024-05-09T09:20:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:16 smithi069 ceph-mon[30123]: pgmap v406: 129 pgs: 129 active+clean; 8.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T09:20:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:16 smithi163 ceph-mon[34354]: pgmap v406: 129 pgs: 129 active+clean; 8.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T09:20:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:16 smithi103 ceph-mon[34421]: pgmap v406: 129 pgs: 129 active+clean; 8.3 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 521 KiB/s rd, 1.3 MiB/s wr, 252 op/s 2024-05-09T09:20:17.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29139 11.61 MB/sec warmup 15 sec latency 2240.022 ms 2024-05-09T09:20:18.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29139 10.89 MB/sec warmup 16 sec latency 3240.134 ms 2024-05-09T09:20:18.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:18 smithi069 ceph-mon[30123]: pgmap v407: 129 pgs: 129 active+clean; 9.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.4 MiB/s wr, 286 op/s 2024-05-09T09:20:18.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:18 smithi163 ceph-mon[34354]: pgmap v407: 129 pgs: 129 active+clean; 9.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.4 MiB/s wr, 286 op/s 2024-05-09T09:20:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:18 smithi103 ceph-mon[34421]: pgmap v407: 129 pgs: 129 active+clean; 9.6 MiB data, 412 MiB used, 1.0 TiB / 1.0 TiB avail; 666 KiB/s rd, 1.4 MiB/s wr, 286 op/s 2024-05-09T09:20:19.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29139 10.25 MB/sec warmup 17 sec latency 4240.219 ms 2024-05-09T09:20:20.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29395 9.71 MB/sec warmup 18 sec latency 5002.117 ms 2024-05-09T09:20:20.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:20 smithi069 ceph-mon[30123]: pgmap v408: 129 pgs: 129 active+clean; 9.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 234 op/s 2024-05-09T09:20:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:20 smithi163 ceph-mon[34354]: pgmap v408: 129 pgs: 129 active+clean; 9.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 234 op/s 2024-05-09T09:20:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:20 smithi103 ceph-mon[34421]: pgmap v408: 129 pgs: 129 active+clean; 9.9 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 1.0 MiB/s wr, 234 op/s 2024-05-09T09:20:21.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29395 9.20 MB/sec warmup 19 sec latency 1055.670 ms 2024-05-09T09:20:22.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29395 8.74 MB/sec warmup 20 sec latency 2055.782 ms 2024-05-09T09:20:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:22 smithi069 ceph-mon[30123]: pgmap v409: 129 pgs: 129 active+clean; 9.8 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 828 KiB/s wr, 223 op/s 2024-05-09T09:20:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:22 smithi163 ceph-mon[34354]: pgmap v409: 129 pgs: 129 active+clean; 9.8 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 828 KiB/s wr, 223 op/s 2024-05-09T09:20:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:22 smithi103 ceph-mon[34421]: pgmap v409: 129 pgs: 129 active+clean; 9.8 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 424 KiB/s rd, 828 KiB/s wr, 223 op/s 2024-05-09T09:20:23.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29395 8.33 MB/sec warmup 21 sec latency 3055.858 ms 2024-05-09T09:20:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 -- 172.21.15.69:0/1083320294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c100790 msgr2=0x7fab0c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 --2- 172.21.15.69:0/1083320294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c100790 0x7fab0c100bb0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fab08009900 tx=0x7fab0802a410 comp rx=0 tx=0).stop 2024-05-09T09:20:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 -- 172.21.15.69:0/1083320294 shutdown_connections 2024-05-09T09:20:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 --2- 172.21.15.69:0/1083320294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c102340 0x7fab0c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 --2- 172.21.15.69:0/1083320294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c101980 0x7fab0c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 --2- 172.21.15.69:0/1083320294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c100790 0x7fab0c100bb0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 -- 172.21.15.69:0/1083320294 >> 172.21.15.69:0/1083320294 conn(0x7fab0c0fbd20 msgr2=0x7fab0c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 -- 172.21.15.69:0/1083320294 shutdown_connections 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.520+0000 7fab138a5640 1 -- 172.21.15.69:0/1083320294 wait complete. 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 Processor -- start 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 -- start start 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 0x7fab0c197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c101980 0x7fab0c198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c10d530 con 0x7fab0c102340 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c10d3b0 con 0x7fab0c101980 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab138a5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c10d6b0 con 0x7fab0c100790 2024-05-09T09:20:23.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52034/0 (socket says 172.21.15.69:52034) 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 -- 172.21.15.69:0/450571307 learned_addr learned my addr 172.21.15.69:0/450571307 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab10e40640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c101980 0x7fab0c198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11641640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 0x7fab0c197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 -- 172.21.15.69:0/450571307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 msgr2=0x7fab0c197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 0x7fab0c197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 -- 172.21.15.69:0/450571307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c101980 msgr2=0x7fab0c198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c101980 0x7fab0c198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 -- 172.21.15.69:0/450571307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fab0c19f770 con 0x7fab0c102340 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11641640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 0x7fab0c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.521+0000 7fab11e42640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fab0400cad0 tx=0x7fab0400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.522+0000 7faafe7fc640 1 -- 172.21.15.69:0/450571307 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0400ae20 con 0x7fab0c102340 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.522+0000 7faafe7fc640 1 -- 172.21.15.69:0/450571307 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab04005020 con 0x7fab0c102340 2024-05-09T09:20:23.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.522+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fab0c19fa60 con 0x7fab0c102340 2024-05-09T09:20:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.522+0000 7faafe7fc640 1 -- 172.21.15.69:0/450571307 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0400f6f0 con 0x7fab0c102340 2024-05-09T09:20:23.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.522+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fab0c19ff60 con 0x7fab0c102340 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7faafe7fc640 1 -- 172.21.15.69:0/450571307 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fab040051d0 con 0x7fab0c102340 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 Processor -- start 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 -- start start 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de51fd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de5202db0 0x560de5205240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560de5161340 con 0x560de51fdc40 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560de5109290 con 0x560de51fd250 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7fab165b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560de515d300 con 0x560de5202db0 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.524+0000 7faafe7fc640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 0x7faaf80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52036/0 (socket says 172.21.15.69:52036) 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 -- 172.21.15.69:0/3232082663 learned_addr learned my addr 172.21.15.69:0/3232082663 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab10e40640 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de5202db0 0x560de5205240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11e42640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 0x7faaf80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab10e40640 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de51fd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7faafe7fc640 1 -- 172.21.15.69:0/450571307 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fab040cf360 con 0x7fab0c102340 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 -- 172.21.15.69:0/3232082663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de5202db0 msgr2=0x560de5205240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de5202db0 0x560de5205240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 -- 172.21.15.69:0/3232082663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 msgr2=0x560de51fd670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de51fd670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 -- 172.21.15.69:0/3232082663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de512d8e0 con 0x560de51fdc40 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11641640 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fab08031440 tx=0x7fab08031860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5539317266d7409d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7fab11e42640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 0x7faaf80a50f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fab04097d40 tx=0x7fab04086040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7faaf77fe640 1 -- 172.21.15.69:0/3232082663 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab0803f070 con 0x560de51fdc40 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7faaf77fe640 1 -- 172.21.15.69:0/3232082663 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab080035f0 con 0x560de51fdc40 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.525+0000 7faaf77fe640 1 -- 172.21.15.69:0/3232082663 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08035070 con 0x560de51fdc40 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- 172.21.15.69:0/3232082663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 msgr2=0x560de5202740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fab08031440 tx=0x7fab08031860 comp rx=0 tx=0).stop 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- 172.21.15.69:0/3232082663 shutdown_connections 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de5202db0 0x560de5205240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de5202740 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 --2- 172.21.15.69:0/3232082663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de51fd670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- 172.21.15.69:0/3232082663 >> 172.21.15.69:0/3232082663 conn(0x560de51f88c0 msgr2=0x560de51f8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- 172.21.15.69:0/3232082663 shutdown_connections 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- 172.21.15.69:0/3232082663 wait complete. 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 -- start start 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.526+0000 7fab165b9740 1 Processor -- start 2024-05-09T09:20:23.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de5207430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de5207970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560de5205d00 con 0x560de5202db0 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560de511c790 con 0x560de51fd250 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560de5205e80 con 0x560de51fdc40 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab165b9740 10 client.0 _ll_register_callbacks cb 0x560de5157a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52040/0 (socket says 172.21.15.69:52040) 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 -- 172.21.15.69:0/2719038210 learned_addr learned my addr 172.21.15.69:0/2719038210 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab11641640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de5207970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de5207430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 -- 172.21.15.69:0/2719038210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 msgr2=0x560de5207970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de5207970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 -- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 msgr2=0x560de5207430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de5207430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 -- 172.21.15.69:0/2719038210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de4e75210 con 0x560de5202db0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7fab10e40640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fab00012fc0 tx=0x7fab000046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=280692efef17f5bb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7faaf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00023020 con 0x560de5202db0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.527+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fab00004020 con 0x560de5202db0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab080035f0 con 0x560de5202db0 2024-05-09T09:20:23.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fab00004220 con 0x560de5202db0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 10 client.15954 fetch_fsmap learned FSMap version 26 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560de4f41620 con 0x560de5202db0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab080035f0 con 0x560de5202db0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fab000047a0 con 0x560de5202db0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 10 client.15954 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 10 client.15954 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 20 client.15954 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de5212f70 0x560de5215410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 4 client.15954 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560de516ea70 con 0x560de5212f70 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.528+0000 7fab11641640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de5212f70 0x560de5215410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7faaf4ff9640 10 client.15954 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab11641640 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de5212f70 0x560de5215410 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22feb2e2732c931 server_cookie=cf84b6fce606d55 in_seq=0 out_seq=0 2024-05-09T09:20:23.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7faaf4ff9640 1 -- 172.21.15.69:0/2719038210 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560de516ea70 con 0x560de5212f70 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7faaf4ff9640 10 client.15954 handle_command_reply: tid=0 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 client.15954 shutdown 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 msgr2=0x560de520e9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fab00012fc0 tx=0x7fab000046c0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 shutdown_connections 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de5202db0 0x560de520e9b0 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de5212f70 0x560de5215410 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de5207970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 --2- 172.21.15.69:0/2719038210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de51fd250 0x560de5207430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7faaf4ff9640 0 client.15954 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.529+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 >> 172.21.15.69:0/2719038210 conn(0x560de51f88c0 msgr2=0x560de51f7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.530+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 shutdown_connections 2024-05-09T09:20:23.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.530+0000 7fab165b9740 1 -- 172.21.15.69:0/2719038210 wait complete. 2024-05-09T09:20:23.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.530+0000 7fab165b9740 20 client.15954 trim_cache size 0 max 16384 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 Processor -- start 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 -- start start 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de521b260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1acc20 con 0x560de51fd250 2024-05-09T09:20:23.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1aa4f0 con 0x560de5202db0 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab165b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1a8340 con 0x560de51fdc40 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab10e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.544+0000 7fab10e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52042/0 (socket says 172.21.15.69:52042) 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 -- 172.21.15.69:0/3137130992 learned_addr learned my addr 172.21.15.69:0/3137130992 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 -- 172.21.15.69:0/3137130992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 msgr2=0x560de521b260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de521b260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 -- 172.21.15.69:0/3137130992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 msgr2=0x560de520e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 -- 172.21.15.69:0/3137130992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560de51f6f00 con 0x560de51fd250 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab10e40640 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fab00000c00 tx=0x7fab000128a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4714db3a3871d42 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7faaf6ffd640 1 -- 172.21.15.69:0/3137130992 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00034070 con 0x560de51fd250 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7faaf6ffd640 1 -- 172.21.15.69:0/3137130992 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab000043c0 con 0x560de51fd250 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7faaf6ffd640 1 -- 172.21.15.69:0/3137130992 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab00023030 con 0x560de51fd250 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab165b9740 1 -- 172.21.15.69:0/3137130992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 msgr2=0x560de521ad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.545+0000 7fab165b9740 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fab00000c00 tx=0x7fab000128a0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- 172.21.15.69:0/3137130992 shutdown_connections 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fdc40 0x560de521b260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- 172.21.15.69:0/3137130992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fd250 0x560de521ad20 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- 172.21.15.69:0/3137130992 >> 172.21.15.69:0/3137130992 conn(0x560de51f88c0 msgr2=0x560de5202460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- 172.21.15.69:0/3137130992 shutdown_connections 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- 172.21.15.69:0/3137130992 wait complete. 2024-05-09T09:20:23.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- start start 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 Processor -- start 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 0x560de520c750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520dca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1a89b0 con 0x560de51fdc40 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1a9e20 con 0x560de5202db0 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fab0c1a9080 con 0x560de51fd250 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab165b9740 10 client.0 _ll_register_callbacks cb 0x7fab0000ac30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab11641640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab11641640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52058/0 (socket says 172.21.15.69:52058) 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.546+0000 7fab11641640 1 -- 172.21.15.69:0/3005853682 learned_addr learned my addr 172.21.15.69:0/3005853682 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab10e40640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 0x560de520c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab10e40640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 -- 172.21.15.69:0/3005853682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 msgr2=0x560de520c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 0x560de520c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 -- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 msgr2=0x560de520dca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520dca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 -- 172.21.15.69:0/3005853682 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560de520e290 con 0x560de51fdc40 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab10e40640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 0x560de520c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab11641640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fab08009900 tx=0x7fab080058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9be49688091d86c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fab08052020 con 0x560de51fdc40 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fab000043c0 con 0x560de51fdc40 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(93..93 src has 1..93) ==== 9129+0+0 (secure 0 0 0) 0x7fab080319d0 con 0x560de51fdc40 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=26) ==== 24+0+0 (secure 0 0 0) 0x7fab0805a050 con 0x560de51fdc40 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab165b9740 10 client.15960 fetch_fsmap learned FSMap version 26 2024-05-09T09:20:23.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=26,monmap=4+}) -- 0x560de5279c30 con 0x560de51fdc40 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fab000043c0 con 0x560de51fdc40 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.547+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 26) ==== 8354+0+0 (secure 0 0 0) 0x7fab08003960 con 0x560de51fdc40 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 10 client.15960 fetch_fsmap finished waiting for FSMap version 26 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 10 client.15960 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 20 client.15960 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de527a2e0 0x560de527c700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 4 client.15960 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560de5168dc0 con 0x560de527a2e0 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab10e40640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de527a2e0 0x560de527c700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7faaf4ff9640 10 client.15960 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:23.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.548+0000 7fab10e40640 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de527a2e0 0x560de527c700 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc35ec297f735556 server_cookie=785fdd6cda068c3a in_seq=0 out_seq=0 2024-05-09T09:20:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7faaf4ff9640 1 -- 172.21.15.69:0/3005853682 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560de5168dc0 con 0x560de527a2e0 2024-05-09T09:20:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7faaf4ff9640 10 client.15960 handle_command_reply: tid=0 2024-05-09T09:20:23.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 client.15960 shutdown 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 msgr2=0x560de520cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fab08009900 tx=0x7fab080058b0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 shutdown_connections 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560de5202db0 0x560de520dca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560de51fdc40 0x560de520cc90 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560de527a2e0 0x560de527c700 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 --2- 172.21.15.69:0/3005853682 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560de51fd250 0x560de520c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7faaf4ff9640 0 client.15960 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 >> 172.21.15.69:0/3005853682 conn(0x560de51f88c0 msgr2=0x560de51fb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 shutdown_connections 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 1 -- 172.21.15.69:0/3005853682 wait complete. 2024-05-09T09:20:23.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab165b9740 20 client.15960 trim_cache size 0 max 16384 2024-05-09T09:20:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:20:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:20:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:20:23.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 msgr2=0x7faaf80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.638+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 0x7faaf80a50f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fab04097d40 tx=0x7fab04086040 comp rx=0 tx=0).stop 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 msgr2=0x7fab0c19f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fab0400cad0 tx=0x7fab0400cf90 comp rx=0 tx=0).stop 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 shutdown_connections 2024-05-09T09:20:23.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faaf80a2c30 0x7faaf80a50f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fab0c102340 0x7fab0c19f100 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fab0c101980 0x7fab0c198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 --2- 172.21.15.69:0/450571307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fab0c100790 0x7fab0c197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 >> 172.21.15.69:0/450571307 conn(0x7fab0c0fbd20 msgr2=0x7fab0c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 shutdown_connections 2024-05-09T09:20:23.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:23.639+0000 7fab138a5640 1 -- 172.21.15.69:0/450571307 wait complete. 2024-05-09T09:20:24.250 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:20:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:20:24.251 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:20:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:20:24.251 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:20:23 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/636042839 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5638087d5a40 con 0x5638087a8c00 2024-05-09T09:20:24.251 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:20:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:20:24.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29395 7.95 MB/sec warmup 22 sec latency 4055.958 ms 2024-05-09T09:20:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:24 smithi069 ceph-mon[30123]: pgmap v410: 129 pgs: 129 active+clean; 10 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 794 KiB/s wr, 218 op/s 2024-05-09T09:20:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:24 smithi069 ceph-mon[30123]: daemon mds.c restarted 2024-05-09T09:20:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:24 smithi163 ceph-mon[34354]: pgmap v410: 129 pgs: 129 active+clean; 10 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 794 KiB/s wr, 218 op/s 2024-05-09T09:20:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:24 smithi163 ceph-mon[34354]: daemon mds.c restarted 2024-05-09T09:20:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:24 smithi103 ceph-mon[34421]: pgmap v410: 129 pgs: 129 active+clean; 10 MiB data, 418 MiB used, 1.0 TiB / 1.0 TiB avail; 440 KiB/s rd, 794 KiB/s wr, 218 op/s 2024-05-09T09:20:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:24 smithi103 ceph-mon[34421]: daemon mds.c restarted 2024-05-09T09:20:25.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 29940 7.79 MB/sec warmup 23 sec latency 4837.232 ms 2024-05-09T09:20:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:25 smithi069 ceph-mon[30123]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T09:20:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:25 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/3564740613,v1:172.21.15.163:6837/3564740613] up:boot 2024-05-09T09:20:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:25 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:20:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:20:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:25 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:25 smithi163 ceph-mon[34354]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T09:20:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:25 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/3564740613,v1:172.21.15.163:6837/3564740613] up:boot 2024-05-09T09:20:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:25 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:20:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:20:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:25 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:25 smithi103 ceph-mon[34421]: osdmap e94: 12 total, 12 up, 12 in 2024-05-09T09:20:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:25 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/3564740613,v1:172.21.15.163:6837/3564740613] up:boot 2024-05-09T09:20:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:25 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:20:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:20:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:25 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:26.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 32381 8.04 MB/sec warmup 24 sec latency 9.796 ms 2024-05-09T09:20:26.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:26 smithi069 ceph-mon[30123]: pgmap v412: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-05-09T09:20:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:26 smithi163 ceph-mon[34354]: pgmap v412: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-05-09T09:20:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:26 smithi103 ceph-mon[34421]: pgmap v412: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 296 KiB/s rd, 1.9 MiB/s wr, 99 op/s 2024-05-09T09:20:26.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:20:26.673 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:20:26.791 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 -- 172.21.15.69:0/1246008357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 msgr2=0x7f6af0102f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 --2- 172.21.15.69:0/1246008357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af0102f00 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009900 tx=0x7f6ad802a410 comp rx=0 tx=0).stop 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 -- 172.21.15.69:0/1246008357 shutdown_connections 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 --2- 172.21.15.69:0/1246008357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af0109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 --2- 172.21.15.69:0/1246008357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af0102f00 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 --2- 172.21.15.69:0/1246008357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 0x7f6af0101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 -- 172.21.15.69:0/1246008357 >> 172.21.15.69:0/1246008357 conn(0x7f6af00fce00 msgr2=0x7f6af00ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.792 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 -- 172.21.15.69:0/1246008357 shutdown_connections 2024-05-09T09:20:26.793 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.791+0000 7f6af489e640 1 -- 172.21.15.69:0/1246008357 wait complete. 2024-05-09T09:20:26.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 Processor -- start 2024-05-09T09:20:26.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 -- start start 2024-05-09T09:20:26.803 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 0x7f6af0197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010d510 con 0x7f6af0102a80 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010d390 con 0x7f6af0101890 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6af489e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6af010d690 con 0x7f6af0103440 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6aedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6aedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52086/0 (socket says 172.21.15.69:52086) 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6aedd9b640 1 -- 172.21.15.69:0/3157261552 learned_addr learned my addr 172.21.15.69:0/3157261552 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.792+0000 7f6aee59c640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 0x7f6af0197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 -- 172.21.15.69:0/3157261552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 msgr2=0x7f6af019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.804 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 -- 172.21.15.69:0/3157261552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 msgr2=0x7f6af0197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 0x7f6af0197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 -- 172.21.15.69:0/3157261552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6af019f7b0 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009230 tx=0x7f6ad8031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6ae77fe640 1 -- 172.21.15.69:0/3157261552 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad803f030 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6ae77fe640 1 -- 172.21.15.69:0/3157261552 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ad8003ad0 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6af019fa40 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6ae77fe640 1 -- 172.21.15.69:0/3157261552 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8035070 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.793+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6af019ff30 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.795+0000 7f6ae77fe640 1 -- 172.21.15.69:0/3157261552 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6ad8042030 con 0x7f6af0102a80 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.795+0000 7f6af75ae740 1 Processor -- start 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 -- start start 2024-05-09T09:20:26.805 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3cd760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 0x55925b3d5330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55925b32dd00 con 0x55925b3cdd30 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55925b2d54c0 con 0x55925b3d2ea0 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6af75ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55925b329cc0 con 0x55925b3cd340 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aeed9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3cd760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52092/0 (socket says 172.21.15.69:52092) 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 -- 172.21.15.69:0/2741998714 learned_addr learned my addr 172.21.15.69:0/2741998714 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6ae77fe640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 0x7f6adc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 0x55925b3d5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 0x7f6adc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.806 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 -- 172.21.15.69:0/2741998714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 msgr2=0x55925b3cd760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3cd760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 -- 172.21.15.69:0/2741998714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 msgr2=0x55925b3d5330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 0x55925b3d5330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aee59c640 1 -- 172.21.15.69:0/2741998714 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925b10b7d0 con 0x55925b3cdd30 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.796+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 0x55925b3d5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6ae77fe640 1 -- 172.21.15.69:0/3157261552 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6ad80ef980 con 0x7f6af0102a80 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000b7d0 tx=0x7f6ae0004610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e70fe311ebd2fb0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6ae4ff9640 1 -- 172.21.15.69:0/2741998714 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0003dd0 con 0x55925b3cdd30 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 0x7f6adc0a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55925b3ce590 tx=0x7f6ae8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6ae4ff9640 1 -- 172.21.15.69:0/2741998714 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae0015070 con 0x55925b3cdd30 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6ae4ff9640 1 -- 172.21.15.69:0/2741998714 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae0011730 con 0x55925b3cdd30 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6af75ae740 1 -- 172.21.15.69:0/2741998714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 msgr2=0x55925b3d2830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.807 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.797+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7f6ae000b7d0 tx=0x7f6ae0004610 comp rx=0 tx=0).stop 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- 172.21.15.69:0/2741998714 shutdown_connections 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3d2ea0 0x55925b3d5330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3d2830 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2741998714 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3cd760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- 172.21.15.69:0/2741998714 >> 172.21.15.69:0/2741998714 conn(0x55925b3c89b0 msgr2=0x55925b3c8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- 172.21.15.69:0/2741998714 shutdown_connections 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- 172.21.15.69:0/2741998714 wait complete. 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- start start 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 Processor -- start 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3d77c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 0x55925b3d7d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55925b3d5e50 con 0x55925b3d2ea0 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55925b2e8b90 con 0x55925b3cd340 2024-05-09T09:20:26.808 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55925b3d5fd0 con 0x55925b3cdd30 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6af75ae740 10 client.0 _ll_register_callbacks cb 0x55925b324da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6aeed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6aeed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52108/0 (socket says 172.21.15.69:52108) 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6aeed9d640 1 -- 172.21.15.69:0/2661003089 learned_addr learned my addr 172.21.15.69:0/2661003089 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.798+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 0x55925b3d7d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3d77c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 -- 172.21.15.69:0/2661003089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 msgr2=0x55925b3d7d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 0x55925b3d7d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 -- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 msgr2=0x55925b3d77c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3d77c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 -- 172.21.15.69:0/2661003089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925b0413f0 con 0x55925b3d2ea0 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 0x55925b3d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:26.809 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8011b00 tx=0x7f6ae8004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee674175405b917f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8019070 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae0015070 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6ae8020430 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ae8013020 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6af75ae740 10 client.15972 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55925b0010f0 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae0015070 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6ae8028490 con 0x55925b3d2ea0 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.799+0000 7f6af75ae740 10 client.15972 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6af75ae740 10 client.15972 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:26.810 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6af75ae740 20 client.15972 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b3e36e0 0x55925b3e5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6af75ae740 4 client.15972 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55925b33b430 con 0x55925b3e36e0 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b3e36e0 0x55925b3e5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6aee59c640 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b3e36e0 0x55925b3e5ba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=854f7daa364aa56d server_cookie=e7b2bf2bf1828685 in_seq=0 out_seq=0 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.800+0000 7f6ad5ffb640 10 client.15972 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/2661003089 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55925b33b430 con 0x55925b3e36e0 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6ad5ffb640 10 client.15972 handle_command_reply: tid=0 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 client.15972 shutdown 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 msgr2=0x55925b3ded40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8011b00 tx=0x7f6ae8004a40 comp rx=0 tx=0).stop 2024-05-09T09:20:26.811 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 shutdown_connections 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3ded40 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b3e36e0 0x55925b3e5ba0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cdd30 0x55925b3d7d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 --2- 172.21.15.69:0/2661003089 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3d77c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6ad5ffb640 0 client.15972 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 >> 172.21.15.69:0/2661003089 conn(0x55925b3c89b0 msgr2=0x55925b3c37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 shutdown_connections 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 1 -- 172.21.15.69:0/2661003089 wait complete. 2024-05-09T09:20:26.812 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.801+0000 7f6af75ae740 20 client.15972 trim_cache size 0 max 16384 2024-05-09T09:20:26.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 Processor -- start 2024-05-09T09:20:26.824 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 -- start start 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3d5290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cdd30 0x55925b3ded00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01aab50 con 0x55925b3d2ea0 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01a83c0 con 0x55925b3cdd30 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6af75ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01a6210 con 0x55925b3cd340 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.818+0000 7f6aeed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52120/0 (socket says 172.21.15.69:52120) 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 -- 172.21.15.69:0/4063450843 learned_addr learned my addr 172.21.15.69:0/4063450843 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cdd30 0x55925b3ded00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aee59c640 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3d5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 -- 172.21.15.69:0/4063450843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 msgr2=0x55925b3d5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3d5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.825 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 -- 172.21.15.69:0/4063450843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cdd30 msgr2=0x55925b3ded00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cdd30 0x55925b3ded00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 -- 172.21.15.69:0/4063450843 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55925b3cc8d0 con 0x55925b3d2ea0 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8013550 tx=0x7f6ae8005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d17fd353f81a79aa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6ad7fff640 1 -- 172.21.15.69:0/4063450843 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae803a020 con 0x55925b3d2ea0 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6ad7fff640 1 -- 172.21.15.69:0/4063450843 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae8013e30 con 0x55925b3d2ea0 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.819+0000 7f6ad7fff640 1 -- 172.21.15.69:0/4063450843 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ae8019070 con 0x55925b3d2ea0 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- 172.21.15.69:0/4063450843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 msgr2=0x55925b3d63d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8013550 tx=0x7f6ae8005c80 comp rx=0 tx=0).stop 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- 172.21.15.69:0/4063450843 shutdown_connections 2024-05-09T09:20:26.826 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3d2ea0 0x55925b3d63d0 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cdd30 0x55925b3ded00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- 172.21.15.69:0/4063450843 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3cd340 0x55925b3d5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- 172.21.15.69:0/4063450843 >> 172.21.15.69:0/4063450843 conn(0x55925b3c89b0 msgr2=0x55925b3c8d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- 172.21.15.69:0/4063450843 shutdown_connections 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- 172.21.15.69:0/4063450843 wait complete. 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- start start 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 Processor -- start 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3dbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 0x55925b3dd520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01a6880 con 0x55925b3cdd30 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01a7cf0 con 0x55925b3cd340 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6af01a6f50 con 0x55925b3d2ea0 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6af75ae740 10 client.0 _ll_register_callbacks cb 0x7f6ae8013480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:26.827 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52128/0 (socket says 172.21.15.69:52128) 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 -- 172.21.15.69:0/1625087842 learned_addr learned my addr 172.21.15.69:0/1625087842 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 0x55925b3dd520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aee59c640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3dbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 -- 172.21.15.69:0/1625087842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 msgr2=0x55925b3dd520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 0x55925b3dd520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 -- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 msgr2=0x55925b3dbfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3dbfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aedd9b640 1 -- 172.21.15.69:0/1625087842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55925b3ddb10 con 0x55925b3cdd30 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aeed9d640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 0x55925b3dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.820+0000 7f6aee59c640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3dbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:26.828 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6aedd9b640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80bbdf0 tx=0x7f6ad80bc620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5d4fcd50cf9bba server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6ad8049040 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6ad80b84d0 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6ae8013e30 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f6ad80c4ad0 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 10 client.15978 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55925b3deef0 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6ae8013e30 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f6ad80bc0c0 con 0x55925b3cdd30 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 10 client.15978 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:26.829 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 10 client.15978 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 20 client.15978 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b44a450 0x55925b44c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 4 client.15978 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55925b335780 con 0x55925b44a450 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.821+0000 7f6aee59c640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b44a450 0x55925b44c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6aee59c640 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b44a450 0x55925b44c870 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30705aeb7f017f8f server_cookie=ea6e5cc33d625bf8 in_seq=0 out_seq=0 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6ad5ffb640 10 client.15978 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6ad5ffb640 1 -- 172.21.15.69:0/1625087842 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55925b335780 con 0x55925b44a450 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6ad5ffb640 10 client.15978 handle_command_reply: tid=0 2024-05-09T09:20:26.830 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 client.15978 shutdown 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 msgr2=0x55925b3dc510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7f6ad80bbdf0 tx=0x7f6ad80bc620 comp rx=0 tx=0).stop 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 shutdown_connections 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55925b44a450 0x55925b44c870 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55925b3d2ea0 0x55925b3dd520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55925b3cdd30 0x55925b3dc510 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 --2- 172.21.15.69:0/1625087842 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55925b3cd340 0x55925b3dbfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6ad5ffb640 0 client.15978 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 >> 172.21.15.69:0/1625087842 conn(0x55925b3c89b0 msgr2=0x55925b3cf190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 shutdown_connections 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 1 -- 172.21.15.69:0/1625087842 wait complete. 2024-05-09T09:20:26.831 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.822+0000 7f6af75ae740 20 client.15978 trim_cache size 0 max 16384 2024-05-09T09:20:26.832 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:20:26.832 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-05-09T09:20:26.832 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-05-09T09:20:26.832 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:20:26.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 msgr2=0x7f6adc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 0x7f6adc0a5050 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55925b3ce590 tx=0x7f6ae8004ab0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 msgr2=0x7f6af01980d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7f6ad8009230 tx=0x7f6ad8031b60 comp rx=0 tx=0).stop 2024-05-09T09:20:26.834 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 shutdown_connections 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6adc0a2b90 0x7f6adc0a5050 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6af0103440 0x7f6af019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6af0102a80 0x7f6af01980d0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 --2- 172.21.15.69:0/3157261552 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6af0101890 0x7f6af0197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 >> 172.21.15.69:0/3157261552 conn(0x7f6af00fce00 msgr2=0x7f6af0105da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 shutdown_connections 2024-05-09T09:20:26.835 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.823+0000 7f6af489e640 1 -- 172.21.15.69:0/3157261552 wait complete. 2024-05-09T09:20:26.836 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:20:26.836 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 -- 172.21.15.69:0/3532167483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec1018a0 msgr2=0x7fefec101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 --2- 172.21.15.69:0/3532167483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec1018a0 0x7fefec101cc0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7fefe8009900 tx=0x7fefe802a3f0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 -- 172.21.15.69:0/3532167483 shutdown_connections 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 --2- 172.21.15.69:0/3532167483 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 0x7fefec109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 --2- 172.21.15.69:0/3532167483 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec102a90 0x7fefec102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 --2- 172.21.15.69:0/3532167483 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec1018a0 0x7fefec101cc0 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 -- 172.21.15.69:0/3532167483 >> 172.21.15.69:0/3532167483 conn(0x7fefec0fce10 msgr2=0x7fefec0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 -- 172.21.15.69:0/3532167483 shutdown_connections 2024-05-09T09:20:26.943 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.937+0000 7feff3eee640 1 -- 172.21.15.69:0/3532167483 wait complete. 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 Processor -- start 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 -- start start 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec1018a0 0x7fefec0755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 0x7fefec074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fefec10d520 con 0x7fefec102a90 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fefec10d3a0 con 0x7fefec1018a0 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff3eee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fefec10d6a0 con 0x7fefec103450 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52150/0 (socket says 172.21.15.69:52150) 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 -- 172.21.15.69:0/1562028087 learned_addr learned my addr 172.21.15.69:0/1562028087 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1c8a640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec1018a0 0x7fefec0755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.944 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff248b640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 0x7fefec074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 -- 172.21.15.69:0/1562028087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 msgr2=0x7fefec074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 0x7fefec074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 -- 172.21.15.69:0/1562028087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec1018a0 msgr2=0x7fefec0755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.938+0000 7feff1489640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec1018a0 0x7fefec0755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7feff1489640 1 -- 172.21.15.69:0/1562028087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fefec0749f0 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7feff1489640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9f0 tx=0x7fefdc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc00cd80 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fefdc010910 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fefec1a1c90 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fefdc018990 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.939+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fefec1a2180 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.940+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fefec108bc0 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.941+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fefdc010ac0 con 0x7fefec102a90 2024-05-09T09:20:26.945 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.942+0000 7fefe2ffd640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 0x7fefd80a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:26.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.942+0000 7feff1c8a640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 0x7fefd80a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:26.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.942+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fefdc0cf640 con 0x7fefec102a90 2024-05-09T09:20:26.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.942+0000 7feff1c8a640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 0x7fefd80a50d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fefe8005ab0 tx=0x7fefe8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:26.946 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:26.947+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fefdc09bbd0 con 0x7fefec102a90 2024-05-09T09:20:27.160 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.159+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fefec062430 con 0x7fefec102a90 2024-05-09T09:20:27.161 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.160+0000 7fefe2ffd640 1 -- 172.21.15.69:0/1562028087 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) ==== 94+0+11882 (secure 0 0 0) 0x7fefdc09fed0 con 0x7fefec102a90 2024-05-09T09:20:27.161 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:20:27.161 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":28,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:20:24.258374+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":11,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":10,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25375":{"gid":25375,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3564740613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3564740613},{"type":"v1","addr":"172.21.15.163:6837","nonce":3564740613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24527,24491,24511]},"id":1}]} 2024-05-09T09:20:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 28 2024-05-09T09:20:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 msgr2=0x7fefd80a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 0x7fefd80a50d0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fefe8005ab0 tx=0x7fefe8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.162 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 msgr2=0x7fefec073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7fefdc00e9f0 tx=0x7fefdc00eeb0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 shutdown_connections 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fefd80a2c10 0x7fefd80a50d0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fefec103450 0x7fefec074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fefec102a90 0x7fefec073c20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 --2- 172.21.15.69:0/1562028087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fefec1018a0 0x7fefec0755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 >> 172.21.15.69:0/1562028087 conn(0x7fefec0fce10 msgr2=0x7fefec105d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 shutdown_connections 2024-05-09T09:20:27.163 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.162+0000 7feff3eee640 1 -- 172.21.15.69:0/1562028087 wait complete. 2024-05-09T09:20:27.173 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 -- 172.21.15.69:0/2621549558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88104cc0 msgr2=0x7efc881070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 --2- 172.21.15.69:0/2621549558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88104cc0 0x7efc881070b0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7efc7c009900 tx=0x7efc7c02a410 comp rx=0 tx=0).stop 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 -- 172.21.15.69:0/2621549558 shutdown_connections 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 --2- 172.21.15.69:0/2621549558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc881075f0 0x7efc881099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 --2- 172.21.15.69:0/2621549558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88104cc0 0x7efc881070b0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 --2- 172.21.15.69:0/2621549558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc88102390 0x7efc88104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.281 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 -- 172.21.15.69:0/2621549558 >> 172.21.15.69:0/2621549558 conn(0x7efc880fbd20 msgr2=0x7efc880fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 -- 172.21.15.69:0/2621549558 shutdown_connections 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.278+0000 7efc8e590640 1 -- 172.21.15.69:0/2621549558 wait complete. 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 Processor -- start 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 -- start start 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88104cc0 0x7efc881981c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 0x7efc8819f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d560 con 0x7efc88102390 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d3e0 con 0x7efc881075f0 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8e590640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d6e0 con 0x7efc88104cc0 2024-05-09T09:20:27.282 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52162/0 (socket says 172.21.15.69:52162) 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 -- 172.21.15.69:0/2482536161 learned_addr learned my addr 172.21.15.69:0/2482536161 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc877fe640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88104cc0 0x7efc881981c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 -- 172.21.15.69:0/2482536161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88104cc0 msgr2=0x7efc881981c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88104cc0 0x7efc881981c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc87fff640 1 -- 172.21.15.69:0/2482536161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 msgr2=0x7efc8819f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.279+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 0x7efc8819f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc87fff640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 0x7efc8819f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc87fff640 1 -- 172.21.15.69:0/2482536161 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc8819f890 con 0x7efc88102390 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 0x7efc8819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc87fff640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7efc7800e9f0 tx=0x7efc7800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc857fa640 1 -- 172.21.15.69:0/2482536161 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7800cd80 con 0x7efc88102390 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc8819fb80 con 0x7efc88102390 2024-05-09T09:20:27.283 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc857fa640 1 -- 172.21.15.69:0/2482536161 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc78010910 con 0x7efc88102390 2024-05-09T09:20:27.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc857fa640 1 -- 172.21.15.69:0/2482536161 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc780188f0 con 0x7efc88102390 2024-05-09T09:20:27.284 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.280+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc881a0120 con 0x7efc88102390 2024-05-09T09:20:27.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.282+0000 7efc857fa640 1 -- 172.21.15.69:0/2482536161 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efc78010ac0 con 0x7efc88102390 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.282+0000 7efc91378740 1 Processor -- start 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.282+0000 7efc91378740 1 -- start start 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x564179056840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179056eb0 0x564179059340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc857fa640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 0x7efc700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564178fb1c40 con 0x564179051350 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564178f59580 con 0x564179056eb0 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc91378740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564178fadc00 con 0x564179051d40 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52178/0 (socket says 172.21.15.69:52178) 2024-05-09T09:20:27.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2438777360 learned_addr learned my addr 172.21.15.69:0/2438777360 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc857fa640 1 -- 172.21.15.69:0/2482536161 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc780cf9c0 con 0x7efc88102390 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc87fff640 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x564179056840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 0x7efc700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc877fe640 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179056eb0 0x564179059340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2438777360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 msgr2=0x564179056840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x564179056840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2438777360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179056eb0 msgr2=0x564179059340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179056eb0 0x564179059340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2438777360 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564178cc53f0 con 0x564179051350 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efc8000ed70 tx=0x7efc8000c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c4e30c3b190d088 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 0x7efc700a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efc8000cc40 tx=0x7efc80019000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc6e7fc640 1 -- 172.21.15.69:0/2438777360 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80018b70 con 0x564179051350 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc6e7fc640 1 -- 172.21.15.69:0/2438777360 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80018d10 con 0x564179051350 2024-05-09T09:20:27.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.283+0000 7efc6e7fc640 1 -- 172.21.15.69:0/2438777360 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80023600 con 0x564179051350 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- 172.21.15.69:0/2438777360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 msgr2=0x564179051770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7efc8000ed70 tx=0x7efc8000c740 comp rx=0 tx=0).stop 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- 172.21.15.69:0/2438777360 shutdown_connections 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179056eb0 0x564179059340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x564179056840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 --2- 172.21.15.69:0/2438777360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051350 0x564179051770 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- 172.21.15.69:0/2438777360 >> 172.21.15.69:0/2438777360 conn(0x5641790485b0 msgr2=0x5641790489b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- 172.21.15.69:0/2438777360 shutdown_connections 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- 172.21.15.69:0/2438777360 wait complete. 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.284+0000 7efc91378740 1 -- start start 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 Processor -- start 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.294 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179062d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564179059e00 con 0x564179051d40 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564178f6cc40 con 0x564179051350 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564179059f80 con 0x564179056eb0 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc91378740 10 client.0 _ll_register_callbacks cb 0x564178cc4be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52188/0 (socket says 172.21.15.69:52188) 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 -- 172.21.15.69:0/3287094619 learned_addr learned my addr 172.21.15.69:0/3287094619 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc87fff640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 -- 172.21.15.69:0/3287094619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 msgr2=0x564179062d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179062d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179062d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 -- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 msgr2=0x56417905b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.295 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc877fe640 1 -- 172.21.15.69:0/3287094619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564178d8f7d0 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc87fff640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.285+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179062d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc877fe640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efc7c005ab0 tx=0x7efc7c031960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a09f99da99dffa9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c03f070 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc7c004450 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc80018d10 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc7c04c050 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 10 client.15996 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x5641790660e0 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80018d10 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7efc7c004c00 con 0x564179051d40 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 10 client.15996 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 10 client.15996 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:20:27.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 20 client.15996 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x564179067790 0x564179069c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 4 client.15996 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564178fbf370 con 0x564179067790 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.286+0000 7efc87fff640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x564179067790 0x564179069c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.287+0000 7efc87fff640 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x564179067790 0x564179069c30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=8561df5ca3993a99 server_cookie=96bf653e04246418 in_seq=0 out_seq=0 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.287+0000 7efc677fe640 10 client.15996 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.287+0000 7efc677fe640 1 -- 172.21.15.69:0/3287094619 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564178fbf370 con 0x564179067790 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.287+0000 7efc677fe640 10 client.15996 handle_command_reply: tid=0 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 client.15996 shutdown 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 msgr2=0x56417905bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7efc7c005ab0 tx=0x7efc7c031960 comp rx=0 tx=0).stop 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 shutdown_connections 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x564179067790 0x564179069c30 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.297 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179062d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc677fe640 0 client.15996 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 --2- 172.21.15.69:0/3287094619 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 >> 172.21.15.69:0/3287094619 conn(0x5641790485b0 msgr2=0x5641790477b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 shutdown_connections 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 1 -- 172.21.15.69:0/3287094619 wait complete. 2024-05-09T09:20:27.298 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.288+0000 7efc91378740 20 client.15996 trim_cache size 0 max 16384 2024-05-09T09:20:27.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 Processor -- start 2024-05-09T09:20:27.307 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 -- start start 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc881aad00 con 0x564179056eb0 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc881a8570 con 0x564179051350 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc91378740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc881a63c0 con 0x564179051d40 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52202/0 (socket says 172.21.15.69:52202) 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2354888657 learned_addr learned my addr 172.21.15.69:0/2354888657 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc87fff640 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.308 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2354888657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 msgr2=0x56417905bcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2354888657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 msgr2=0x56417905b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.301+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc8cb2d640 1 -- 172.21.15.69:0/2354888657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5641790441a0 con 0x564179056eb0 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efc8000f010 tx=0x7efc8000e660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb8678552475a8d3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc6dffb640 1 -- 172.21.15.69:0/2354888657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80018b40 con 0x564179056eb0 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc6dffb640 1 -- 172.21.15.69:0/2354888657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80028070 con 0x564179056eb0 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc6dffb640 1 -- 172.21.15.69:0/2354888657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80024650 con 0x564179056eb0 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 -- 172.21.15.69:0/2354888657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 msgr2=0x564179062cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7efc8000f010 tx=0x7efc8000e660 comp rx=0 tx=0).stop 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 -- 172.21.15.69:0/2354888657 shutdown_connections 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179056eb0 0x564179062cc0 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179051d40 0x56417905bcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.309 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 --2- 172.21.15.69:0/2354888657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x56417905b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 -- 172.21.15.69:0/2354888657 >> 172.21.15.69:0/2354888657 conn(0x5641790485b0 msgr2=0x564179048990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 -- 172.21.15.69:0/2354888657 shutdown_connections 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.302+0000 7efc91378740 1 -- 172.21.15.69:0/2354888657 wait complete. 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 -- start start 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 Processor -- start 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x564179060010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179061560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc881a6a30 con 0x564179051d40 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc881a7ea0 con 0x564179051350 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc881a7100 con 0x564179056eb0 2024-05-09T09:20:27.310 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc91378740 10 client.0 _ll_register_callbacks cb 0x7efc78005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52216/0 (socket says 172.21.15.69:52216) 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc877fe640 1 -- 172.21.15.69:0/735950640 learned_addr learned my addr 172.21.15.69:0/735950640 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc87fff640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x564179060010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.303+0000 7efc8cb2d640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179061560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 -- 172.21.15.69:0/735950640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 msgr2=0x564179061560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179061560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 -- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 msgr2=0x564179060010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x564179060010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 -- 172.21.15.69:0/735950640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564179061b50 con 0x564179051d40 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc877fe640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efc7c003640 tx=0x7efc7c003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ad6a2b5f74f7df9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:27.311 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7c03f070 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc7c004250 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc80028070 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7efc7c035070 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 10 client.16002 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x564179062e50 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80028070 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7efc7c0505c0 con 0x564179051d40 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 10 client.16002 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 10 client.16002 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 20 client.16002 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5641790ce3e0 0x5641790d0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 4 client.16002 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.304+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "damage ls"}) -- 0x564178fb96c0 con 0x5641790ce3e0 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc87fff640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5641790ce3e0 0x5641790d0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.312 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc677fe640 10 client.16002 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc87fff640 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5641790ce3e0 0x5641790d0800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=1c19976cb7e2eafc server_cookie=43ff33a8bf35748d in_seq=0 out_seq=0 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc677fe640 1 -- 172.21.15.69:0/735950640 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x564178fb96c0 con 0x5641790ce3e0 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc677fe640 10 client.16002 handle_command_reply: tid=0 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 client.16002 shutdown 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 msgr2=0x564179060550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7efc7c003640 tx=0x7efc7c003f80 comp rx=0 tx=0).stop 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 shutdown_connections 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x5641790ce3e0 0x5641790d0800 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564179056eb0 0x564179061560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564179051d40 0x564179060550 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc677fe640 0 client.16002 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 --2- 172.21.15.69:0/735950640 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564179051350 0x564179060010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 >> 172.21.15.69:0/735950640 conn(0x5641790485b0 msgr2=0x564179053240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 shutdown_connections 2024-05-09T09:20:27.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 1 -- 172.21.15.69:0/735950640 wait complete. 2024-05-09T09:20:27.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.305+0000 7efc91378740 20 client.16002 trim_cache size 0 max 16384 2024-05-09T09:20:27.314 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 msgr2=0x7efc700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 0x7efc700a5050 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7efc8000cc40 tx=0x7efc80019000 comp rx=0 tx=0).stop 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 msgr2=0x7efc88197c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x7efc7800e9f0 tx=0x7efc7800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 shutdown_connections 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc700a2b90 0x7efc700a5050 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881075f0 0x7efc8819f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88104cc0 0x7efc881981c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 --2- 172.21.15.69:0/2482536161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88102390 0x7efc88197c80 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 >> 172.21.15.69:0/2482536161 conn(0x7efc880fbd20 msgr2=0x7efc88102e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 shutdown_connections 2024-05-09T09:20:27.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.306+0000 7efc8e590640 1 -- 172.21.15.69:0/2482536161 wait complete. 2024-05-09T09:20:27.317 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:20:27.433 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- 172.21.15.69:0/3679062371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8102340 msgr2=0x7f2cd8108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 --2- 172.21.15.69:0/3679062371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8102340 0x7f2cd8108bd0 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0009ca0 tx=0x7f2cd002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- 172.21.15.69:0/3679062371 shutdown_connections 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 --2- 172.21.15.69:0/3679062371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8102340 0x7f2cd8108bd0 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 --2- 172.21.15.69:0/3679062371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8101980 0x7f2cd8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 --2- 172.21.15.69:0/3679062371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd8100790 0x7f2cd8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- 172.21.15.69:0/3679062371 >> 172.21.15.69:0/3679062371 conn(0x7f2cd80fbd20 msgr2=0x7f2cd80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- 172.21.15.69:0/3679062371 shutdown_connections 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- 172.21.15.69:0/3679062371 wait complete. 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 Processor -- start 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.419+0000 7f2ce029e640 1 -- start start 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8100790 0x7f2cd8197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.434 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd8102340 0x7f2cd819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810d530 con 0x7f2cd8101980 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810d3b0 con 0x7f2cd8102340 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd810d6b0 con 0x7f2cd8100790 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52244/0 (socket says 172.21.15.69:52244) 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 -- 172.21.15.69:0/4135711237 learned_addr learned my addr 172.21.15.69:0/4135711237 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cde03a640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8100790 0x7f2cd8197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 -- 172.21.15.69:0/4135711237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8100790 msgr2=0x7f2cd8197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8100790 0x7f2cd8197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 -- 172.21.15.69:0/4135711237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd8102340 msgr2=0x7f2cd819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd8102340 0x7f2cd819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 -- 172.21.15.69:0/4135711237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2cd819f740 con 0x7f2cd8101980 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2cdd839640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c970 tx=0x7f2cc800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ccf7fe640 1 -- 172.21.15.69:0/4135711237 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc8007bb0 con 0x7f2cd8101980 2024-05-09T09:20:27.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ccf7fe640 1 -- 172.21.15.69:0/4135711237 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cc8007d50 con 0x7f2cd8101980 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2cd819fa30 con 0x7f2cd8101980 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ccf7fe640 1 -- 172.21.15.69:0/4135711237 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cc800f450 con 0x7f2cd8101980 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.420+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2cd819ff30 con 0x7f2cd8101980 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.422+0000 7f2ccf7fe640 1 -- 172.21.15.69:0/4135711237 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2cc8005020 con 0x7f2cd8101980 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 Processor -- start 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 -- start start 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e6442770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e644a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a9e63a2c40 con 0x55a9e6442d40 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9e634a580 con 0x55a9e6447eb0 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a9e639ec00 con 0x55a9e6442350 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2ccf7fe640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 0x7f2cc40a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52248/0 (socket says 172.21.15.69:52248) 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.423+0000 7f2cde03a640 1 -- 172.21.15.69:0/2644248470 learned_addr learned my addr 172.21.15.69:0/2644248470 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde83b640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 0x7f2cc40a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde83b640 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e6442770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2ccf7fe640 1 -- 172.21.15.69:0/4135711237 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f2cc80cf1a0 con 0x7f2cd8101980 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cdd839640 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e644a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 -- 172.21.15.69:0/2644248470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 msgr2=0x55a9e6442770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e6442770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.437 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 -- 172.21.15.69:0/2644248470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 msgr2=0x55a9e644a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e644a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 -- 172.21.15.69:0/2644248470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e60b63f0 con 0x55a9e6442d40 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400abf0 tx=0x7f2cd400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=935211f814ed7c58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cde83b640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 0x7f2cc40a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55a9e64435a0 tx=0x7f2cd00095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cccbfe640 1 -- 172.21.15.69:0/2644248470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4004220 con 0x55a9e6442d40 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cccbfe640 1 -- 172.21.15.69:0/2644248470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd40043c0 con 0x55a9e6442d40 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.424+0000 7f2cccbfe640 1 -- 172.21.15.69:0/2644248470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4018910 con 0x55a9e6442d40 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2644248470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 msgr2=0x55a9e6447840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f2cd400abf0 tx=0x7f2cd400f790 comp rx=0 tx=0).stop 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2644248470 shutdown_connections 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e644a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442d40 0x55a9e6447840 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2644248470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e6442770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2644248470 >> 172.21.15.69:0/2644248470 conn(0x55a9e64395b0 msgr2=0x55a9e64399b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2644248470 shutdown_connections 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2644248470 wait complete. 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 -- start start 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.425+0000 7f2ce2fab740 1 Processor -- start 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e644c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6442d40 0x55a9e644cc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a9e644ae60 con 0x55a9e6447eb0 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9e635dc40 con 0x55a9e6442d40 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a9e644afe0 con 0x55a9e6442350 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 10 client.0 _ll_register_callbacks cb 0x55a9e60b5be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e644c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52264/0 (socket says 172.21.15.69:52264) 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e644c6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:43706/0 (socket says 172.21.15.69:43706) 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 -- 172.21.15.69:0/1298028710 learned_addr learned my addr 172.21.15.69:0/1298028710 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cdd839640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6442d40 0x55a9e644cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 -- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 msgr2=0x55a9e644c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e644c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 -- 172.21.15.69:0/1298028710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6442d40 msgr2=0x55a9e644cc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6442d40 0x55a9e644cc20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 -- 172.21.15.69:0/1298028710 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e61807d0 con 0x55a9e6447eb0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cde83b640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0035000 tx=0x7f2cd0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef78974e6e4caff7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd004f070 con 0x55a9e6447eb0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f2cd0004520 con 0x55a9e6447eb0 2024-05-09T09:20:27.440 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd40043c0 con 0x55a9e6447eb0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cd0004720 con 0x55a9e6447eb0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 10 client.16014 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.426+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9e6456fb0 con 0x55a9e6447eb0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd40043c0 con 0x55a9e6447eb0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2cd004a070 con 0x55a9e6447eb0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 10 client.16014 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 10 client.16014 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 20 client.16014 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e6458690 0x55a9e645aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 4 client.16014 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e63b0370 con 0x55a9e6458690 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2cde03a640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e6458690 0x55a9e645aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2cc1ffb640 10 client.16014 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.427+0000 7f2cde03a640 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e6458690 0x55a9e645aaf0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=9950f118bcc9b0a server_cookie=964e8f9e2635c364 in_seq=0 out_seq=0 2024-05-09T09:20:27.441 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/1298028710 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9e63b0370 con 0x55a9e6458690 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2cc1ffb640 10 client.16014 handle_command_reply: tid=0 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 client.16014 shutdown 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 msgr2=0x55a9e6453c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7f2cd0035000 tx=0x7f2cd0004a40 comp rx=0 tx=0).stop 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 shutdown_connections 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e6458690 0x55a9e645aaf0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c60 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6442d40 0x55a9e644cc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2cc1ffb640 0 client.16014 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/1298028710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442350 0x55a9e644c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.442 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 >> 172.21.15.69:0/1298028710 conn(0x55a9e64395b0 msgr2=0x55a9e64387b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 shutdown_connections 2024-05-09T09:20:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 1 -- 172.21.15.69:0/1298028710 wait complete. 2024-05-09T09:20:27.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.428+0000 7f2ce2fab740 20 client.16014 trim_cache size 0 max 16384 2024-05-09T09:20:27.449 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 Processor -- start 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 -- start start 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6438b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81aaae0 con 0x55a9e6442350 2024-05-09T09:20:27.450 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81a8350 con 0x55a9e6447eb0 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81a61a0 con 0x55a9e6442d40 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52276/0 (socket says 172.21.15.69:52276) 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 -- 172.21.15.69:0/2419661541 learned_addr learned my addr 172.21.15.69:0/2419661541 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cdd839640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6438b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 -- 172.21.15.69:0/2419661541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 msgr2=0x55a9e6438b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6438b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 -- 172.21.15.69:0/2419661541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 msgr2=0x55a9e6453c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde83b640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 -- 172.21.15.69:0/2419661541 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e642ad20 con 0x55a9e6442350 2024-05-09T09:20:27.451 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cdd839640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6438b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde83b640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.442+0000 7f2cde03a640 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4004140 tx=0x7f2cd400a780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bfce38b3f8a351f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2cc3fff640 1 -- 172.21.15.69:0/2419661541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4004270 con 0x55a9e6442350 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2cc3fff640 1 -- 172.21.15.69:0/2419661541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd4004410 con 0x55a9e6442350 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2cc3fff640 1 -- 172.21.15.69:0/2419661541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4004c10 con 0x55a9e6442350 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2419661541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 msgr2=0x55a9e6438630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f2cd4004140 tx=0x7f2cd400a780 comp rx=0 tx=0).stop 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2419661541 shutdown_connections 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6453c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6438b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/2419661541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e6438630 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2419661541 >> 172.21.15.69:0/2419661541 conn(0x55a9e64395b0 msgr2=0x55a9e6437f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.452 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2419661541 shutdown_connections 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- 172.21.15.69:0/2419661541 wait complete. 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.443+0000 7f2ce2fab740 1 -- start start 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 Processor -- start 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6451f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6452ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81a6810 con 0x55a9e6442350 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81a7c80 con 0x55a9e6447eb0 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2cd81a6ee0 con 0x55a9e6442d40 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 10 client.0 _ll_register_callbacks cb 0x7f2cd0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.453 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52280/0 (socket says 172.21.15.69:52280) 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 -- 172.21.15.69:0/3724128307 learned_addr learned my addr 172.21.15.69:0/3724128307 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cdd839640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6451f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde83b640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6452ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 -- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 msgr2=0x55a9e6451f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6451f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 -- 172.21.15.69:0/3724128307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 msgr2=0x55a9e6452ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6452ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 -- 172.21.15.69:0/3724128307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e64534b0 con 0x55a9e6442350 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cdd839640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6451f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cde03a640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55a9e644b870 tx=0x7f2cd400a7f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d3b861a38d464d6c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cc1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd4035070 con 0x55a9e6442350 2024-05-09T09:20:27.454 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f2cd4004da0 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.444+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd4004410 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f2cd4041050 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 10 client.16020 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a9e64bf110 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd4004410 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f2cd402c070 con 0x55a9e6442350 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 10 client.16020 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.455 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 10 client.16020 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 20 client.16020 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e64bf910 0x55a9e64c1d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 4 client.16020 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55a9e63aa6c0 con 0x55a9e64bf910 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cdd839640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e64bf910 0x55a9e64c1d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cc1ffb640 10 client.16020 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:27.456 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.445+0000 7f2cdd839640 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e64bf910 0x55a9e64c1d30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=ed487c0b2e7141b5 server_cookie=1f1d5ce805083647 in_seq=0 out_seq=0 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2cc1ffb640 1 -- 172.21.15.69:0/3724128307 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55a9e63aa6c0 con 0x55a9e64bf910 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2cc1ffb640 10 client.16020 handle_command_reply: tid=0 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 client.16020 shutdown 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 msgr2=0x55a9e64519c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x55a9e644b870 tx=0x7f2cd400a7f0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 shutdown_connections 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x55a9e64bf910 0x55a9e64c1d30 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.457 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e6447eb0 0x55a9e6452ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2cc1ffb640 0 client.16020 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e6442d40 0x55a9e6451f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 --2- 172.21.15.69:0/3724128307 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e6442350 0x55a9e64519c0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 >> 172.21.15.69:0/3724128307 conn(0x55a9e64395b0 msgr2=0x55a9e64387b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 shutdown_connections 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 1 -- 172.21.15.69:0/3724128307 wait complete. 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce2fab740 20 client.16020 trim_cache size 0 max 16384 2024-05-09T09:20:27.458 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:20:27.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 msgr2=0x7f2cc40a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 0x7f2cc40a5050 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x55a9e64435a0 tx=0x7f2cd00095c0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.459 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 msgr2=0x7f2cd8198060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7f2cc800c970 tx=0x7f2cc800ce30 comp rx=0 tx=0).stop 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 shutdown_connections 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cc40a2b90 0x7f2cc40a5050 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2cd8102340 0x7f2cd819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2cd8101980 0x7f2cd8198060 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 --2- 172.21.15.69:0/4135711237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2cd8100790 0x7f2cd8197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 >> 172.21.15.69:0/4135711237 conn(0x7f2cd80fbd20 msgr2=0x7f2cd8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 shutdown_connections 2024-05-09T09:20:27.460 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.446+0000 7f2ce029e640 1 -- 172.21.15.69:0/4135711237 wait complete. 2024-05-09T09:20:27.460 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:27 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1562028087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:20:27.462 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:20:27.470 INFO:tasks.workunit.client.0.smithi069.stdout: 1 32687 7.73 MB/sec warmup 25 sec latency 720.786 ms 2024-05-09T09:20:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:27 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1562028087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3556715371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8102340 msgr2=0x7fcfa8108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3556715371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8102340 0x7fcfa8108bd0 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf9002a410 comp rx=0 tx=0).stop 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3556715371 shutdown_connections 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3556715371 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8102340 0x7fcfa8108bd0 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3556715371 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8101980 0x7fcfa8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3556715371 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 0x7fcfa8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3556715371 >> 172.21.15.69:0/3556715371 conn(0x7fcfa80fbd00 msgr2=0x7fcfa80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3556715371 shutdown_connections 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3556715371 wait complete. 2024-05-09T09:20:27.577 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 Processor -- start 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.564+0000 7fcfaf11d640 1 -- start start 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 0x7fcfa8197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8102340 0x7fcfa819f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810c410 con 0x7fcfa8101980 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810c290 con 0x7fcfa8102340 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa810c590 con 0x7fcfa8100790 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39082/0 (socket says 172.21.15.69:39082) 2024-05-09T09:20:27.578 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 -- 172.21.15.69:0/3422372051 learned_addr learned my addr 172.21.15.69:0/3422372051 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 0x7fcfa8197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8102340 0x7fcfa819f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 -- 172.21.15.69:0/3422372051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 msgr2=0x7fcfa8197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 0x7fcfa8197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 -- 172.21.15.69:0/3422372051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8102340 msgr2=0x7fcfa819f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8102340 0x7fcfa819f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 -- 172.21.15.69:0/3422372051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcfa819f7d0 con 0x7fcfa8101980 2024-05-09T09:20:27.579 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa7fff640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ebc0 tx=0x7fcf9800c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa5ffb640 1 -- 172.21.15.69:0/3422372051 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98010630 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa5ffb640 1 -- 172.21.15.69:0/3422372051 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf98010cb0 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfa5ffb640 1 -- 172.21.15.69:0/3422372051 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf98018bd0 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.565+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcfa819fac0 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.566+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcfa81a0010 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfa5ffb640 1 -- 172.21.15.69:0/3422372051 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcf980107d0 con 0x7fcfa8101980 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 Processor -- start 2024-05-09T09:20:27.580 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 -- start start 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f4840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfa5ffb640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 0x7fcf940a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603f7340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fd6034fc40 con 0x55fd603ef350 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fd602f7580 con 0x55fd603f4eb0 2024-05-09T09:20:27.581 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fd6034bc00 con 0x55fd603efd40 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfa7fff640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 0x7fcf940a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39098/0 (socket says 172.21.15.69:39098) 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.568+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3164391460 learned_addr learned my addr 172.21.15.69:0/3164391460 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f4840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfa5ffb640 1 -- 172.21.15.69:0/3422372051 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fcf980cf9e0 con 0x7fcfa8101980 2024-05-09T09:20:27.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3164391460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 msgr2=0x55fd603f4840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f4840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3164391460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 msgr2=0x55fd603f7340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603f7340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3164391460 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd600633f0 con 0x55fd603ef350 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfa7fff640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 0x7fcf940a5050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cd1b0 tx=0x7fcf98067090 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf90005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fc2761790a8d73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcf9effd640 1 -- 172.21.15.69:0/3164391460 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90035040 con 0x55fd603ef350 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcf9effd640 1 -- 172.21.15.69:0/3164391460 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf900319b0 con 0x55fd603ef350 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.569+0000 7fcf9effd640 1 -- 172.21.15.69:0/3164391460 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf900049e0 con 0x55fd603ef350 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3164391460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 msgr2=0x55fd603ef770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf90005e00 comp rx=0 tx=0).stop 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3164391460 shutdown_connections 2024-05-09T09:20:27.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603f7340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f4840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3164391460 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603ef770 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3164391460 >> 172.21.15.69:0/3164391460 conn(0x55fd603e65b0 msgr2=0x55fd603e69b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3164391460 shutdown_connections 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3164391460 wait complete. 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 -- start start 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.570+0000 7fcfb1e21740 1 Processor -- start 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603ef350 0x55fd603f9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 0x55fd60400cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fd603f7e00 con 0x55fd603f4eb0 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fd6030ac40 con 0x55fd603ef350 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fd603f7f80 con 0x55fd603efd40 2024-05-09T09:20:27.585 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 10 client.0 _ll_register_callbacks cb 0x55fd60062be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 0x55fd60400cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 0x55fd60400cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39114/0 (socket says 172.21.15.69:39114) 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3044478201 learned_addr learned my addr 172.21.15.69:0/3044478201 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603ef350 0x55fd603f9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 -- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603ef350 msgr2=0x55fd603f9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603ef350 0x55fd603f9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 -- 172.21.15.69:0/3044478201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 msgr2=0x55fd60400cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 0x55fd60400cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 -- 172.21.15.69:0/3044478201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd6012d7d0 con 0x55fd603efd40 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f9c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55fd603f0140 tx=0x7fcfa0012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=873de3abc366e857 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcf87fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0004050 con 0x55fd603efd40 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fcfa00041f0 con 0x55fd603efd40 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.571+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf900319b0 con 0x55fd603efd40 2024-05-09T09:20:27.587 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcfa0025050 con 0x55fd603efd40 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 10 client.25429 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd60404110 con 0x55fd603efd40 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf900319b0 con 0x55fd603efd40 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcfa0003a40 con 0x55fd603efd40 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 10 client.25429 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 10 client.25429 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 20 client.25429 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd604057f0 0x55fd60407c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 4 client.25429 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fd6035d370 con 0x55fd604057f0 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.572+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd604057f0 0x55fd60407c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.573+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd604057f0 0x55fd60407c10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b68de1d95fbf9a1 server_cookie=d25a07dfe0114f02 in_seq=0 out_seq=0 2024-05-09T09:20:27.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.573+0000 7fcf87fff640 10 client.25429 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.573+0000 7fcf87fff640 1 -- 172.21.15.69:0/3044478201 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fd6035d370 con 0x55fd604057f0 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.573+0000 7fcf87fff640 10 client.25429 handle_command_reply: tid=0 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.573+0000 7fcfb1e21740 1 client.25429 shutdown 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 msgr2=0x55fd603f9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f9c90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x55fd603f0140 tx=0x7fcfa0012b60 comp rx=0 tx=0).stop 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 shutdown_connections 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603f4eb0 0x55fd60400cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd604057f0 0x55fd60407c10 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f9c90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcf87fff640 0 client.25429 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3044478201 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603ef350 0x55fd603f9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 >> 172.21.15.69:0/3044478201 conn(0x55fd603e65b0 msgr2=0x55fd603e57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 shutdown_connections 2024-05-09T09:20:27.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3044478201 wait complete. 2024-05-09T09:20:27.590 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.574+0000 7fcfb1e21740 20 client.25429 trim_cache size 0 max 16384 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 Processor -- start 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 -- start start 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f1640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd60400c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81aab70 con 0x55fd603ef350 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81a83e0 con 0x55fd603f4eb0 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81a6230 con 0x55fd603efd40 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39116/0 (socket says 172.21.15.69:39116) 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3539654989 learned_addr learned my addr 172.21.15.69:0/3539654989 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3539654989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 msgr2=0x55fd603f1640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.587+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f1640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3539654989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 msgr2=0x55fd60400c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd60400c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfad6ba640 1 -- 172.21.15.69:0/3539654989 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fd60408290 con 0x55fd603ef350 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf9002a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7e6544a05b90537 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcf9e7fc640 1 -- 172.21.15.69:0/3539654989 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90035040 con 0x55fd603ef350 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcf9e7fc640 1 -- 172.21.15.69:0/3539654989 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf90005550 con 0x55fd603ef350 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcf9e7fc640 1 -- 172.21.15.69:0/3539654989 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf900050d0 con 0x55fd603ef350 2024-05-09T09:20:27.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3539654989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 msgr2=0x55fd603f1100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcf90009900 tx=0x7fcf9002a410 comp rx=0 tx=0).stop 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3539654989 shutdown_connections 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd60400c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603f1640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/3539654989 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603f1100 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3539654989 >> 172.21.15.69:0/3539654989 conn(0x55fd603e65b0 msgr2=0x55fd603f9750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3539654989 shutdown_connections 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.588+0000 7fcfb1e21740 1 -- 172.21.15.69:0/3539654989 wait complete. 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 -- start start 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 Processor -- start 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603fe750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603ff710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81a68a0 con 0x55fd603ef350 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81a7d10 con 0x55fd603f4eb0 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcfa81a6f70 con 0x55fd603efd40 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfb1e21740 10 client.0 _ll_register_callbacks cb 0x7fcfa0009830 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfad6ba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39120/0 (socket says 172.21.15.69:39120) 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfad6ba640 1 -- 172.21.15.69:0/859381092 learned_addr learned my addr 172.21.15.69:0/859381092 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603fe750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.589+0000 7fcfad6ba640 1 -- 172.21.15.69:0/859381092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 msgr2=0x55fd603fe750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603fe750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfad6ba640 1 -- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 msgr2=0x55fd603ff710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603ff710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfad6ba640 1 -- 172.21.15.69:0/859381092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fd603ffd00 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfad6ba640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005c40 tx=0x7fcf90031a50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=39d4847f773c49f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcf90035040 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fcf90004e50 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcf90005550 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7fcf90005050 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 10 client.16038 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55fd60401000 con 0x55fd603ef350 2024-05-09T09:20:27.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcf90005550 con 0x55fd603ef350 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7fcf9005e030 con 0x55fd603ef350 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 10 client.16038 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 10 client.16038 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 20 client.16038 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd6046c0b0 0x55fd6046e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 4 client.16038 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55fd603576c0 con 0x55fd6046c0b0 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.590+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd6046c0b0 0x55fd6046e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcf87fff640 10 client.16038 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfaceb9640 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd6046c0b0 0x55fd6046e4d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0b90f30e8478bb1 server_cookie=ef42ebc7680305ba in_seq=0 out_seq=0 2024-05-09T09:20:27.600 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcf87fff640 1 -- 172.21.15.69:0/859381092 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x55fd603576c0 con 0x55fd6046c0b0 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcf87fff640 10 client.16038 handle_command_reply: tid=0 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 client.16038 shutdown 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 msgr2=0x55fd603fe210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fcf90005c40 tx=0x7fcf90031a50 comp rx=0 tx=0).stop 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 shutdown_connections 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fd603f4eb0 0x55fd603ff710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fd6046c0b0 0x55fd6046e4d0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fd603efd40 0x55fd603fe750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 --2- 172.21.15.69:0/859381092 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fd603ef350 0x55fd603fe210 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcf87fff640 0 client.16038 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 >> 172.21.15.69:0/859381092 conn(0x55fd603e65b0 msgr2=0x55fd603e9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 shutdown_connections 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 1 -- 172.21.15.69:0/859381092 wait complete. 2024-05-09T09:20:27.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.591+0000 7fcfb1e21740 20 client.16038 trim_cache size 0 max 16384 2024-05-09T09:20:27.602 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:20:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 msgr2=0x7fcf940a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 0x7fcf940a5050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fcf980cd1b0 tx=0x7fcf98067090 comp rx=0 tx=0).stop 2024-05-09T09:20:27.603 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 msgr2=0x7fcfa81980f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800ebc0 tx=0x7fcf9800c6a0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 shutdown_connections 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcf940a2b90 0x7fcf940a5050 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcfa8102340 0x7fcfa819f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcfa8101980 0x7fcfa81980f0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 --2- 172.21.15.69:0/3422372051 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcfa8100790 0x7fcfa8197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 >> 172.21.15.69:0/3422372051 conn(0x7fcfa80fbd00 msgr2=0x7fcfa8069e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 shutdown_connections 2024-05-09T09:20:27.604 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:27.592+0000 7fcfaf11d640 1 -- 172.21.15.69:0/3422372051 wait complete. 2024-05-09T09:20:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:27 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1562028087' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:20:28.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 32687 7.44 MB/sec warmup 26 sec latency 1720.849 ms 2024-05-09T09:20:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:28 smithi069 ceph-mon[30123]: pgmap v413: 129 pgs: 129 active+clean; 25 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 3.5 MiB/s wr, 137 op/s 2024-05-09T09:20:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:28 smithi163 ceph-mon[34354]: pgmap v413: 129 pgs: 129 active+clean; 25 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 3.5 MiB/s wr, 137 op/s 2024-05-09T09:20:28.605 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-05-09T09:20:28.606 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:20:28.606 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-05-09T09:20:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:28 smithi103 ceph-mon[34421]: pgmap v413: 129 pgs: 129 active+clean; 25 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 138 KiB/s rd, 3.5 MiB/s wr, 137 op/s 2024-05-09T09:20:28.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 -- 172.21.15.69:0/390858363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4102340 msgr2=0x7f8ed4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.725 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/390858363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4102340 0x7f8ed4108bd0 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc009ca0 tx=0x7f8ecc02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 -- 172.21.15.69:0/390858363 shutdown_connections 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/390858363 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4102340 0x7f8ed4108bd0 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/390858363 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/390858363 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4100790 0x7f8ed4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 -- 172.21.15.69:0/390858363 >> 172.21.15.69:0/390858363 conn(0x7f8ed40fbd20 msgr2=0x7f8ed40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 -- 172.21.15.69:0/390858363 shutdown_connections 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.723+0000 7f8ed8db7640 1 -- 172.21.15.69:0/390858363 wait complete. 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 Processor -- start 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 -- start start 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4102340 0x7f8ed419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.726 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed410d530 con 0x7f8ed4100790 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed410d3b0 con 0x7f8ed4101980 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed8db7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed410d6b0 con 0x7f8ed4102340 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39152/0 (socket says 172.21.15.69:39152) 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 -- 172.21.15.69:0/1613157574 learned_addr learned my addr 172.21.15.69:0/1613157574 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4102340 0x7f8ed419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 -- 172.21.15.69:0/1613157574 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4102340 msgr2=0x7f8ed419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4102340 0x7f8ed419f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 -- 172.21.15.69:0/1613157574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 msgr2=0x7f8ed4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 -- 172.21.15.69:0/1613157574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8ed419f770 con 0x7f8ed4100790 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.724+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400d950 tx=0x7f8ec400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.725+0000 7f8ecb7fe640 1 -- 172.21.15.69:0/1613157574 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4017070 con 0x7f8ed4100790 2024-05-09T09:20:28.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.725+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8ed419fa60 con 0x7f8ed4100790 2024-05-09T09:20:28.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.725+0000 7f8ecb7fe640 1 -- 172.21.15.69:0/1613157574 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4009c30 con 0x7f8ed4100790 2024-05-09T09:20:28.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.725+0000 7f8ecb7fe640 1 -- 172.21.15.69:0/1613157574 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec400f600 con 0x7f8ed4100790 2024-05-09T09:20:28.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.725+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8ed419ffb0 con 0x7f8ed4100790 2024-05-09T09:20:28.734 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.727+0000 7f8ecb7fe640 1 -- 172.21.15.69:0/1613157574 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8ec401a070 con 0x7f8ed4100790 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.727+0000 7f8edbba5740 1 Processor -- start 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 -- start start 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949a89b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949ab4b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ecb7fe640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 0x7f8ec00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a894907990 con 0x55a8949a34c0 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a8948af5b0 con 0x55a8949a3eb0 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8edbba5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a894903950 con 0x55a8949a9020 2024-05-09T09:20:28.735 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 0x7f8ec00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39166/0 (socket says 172.21.15.69:39166) 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 -- 172.21.15.69:0/1556947796 learned_addr learned my addr 172.21.15.69:0/1556947796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ecb7fe640 1 -- 172.21.15.69:0/1613157574 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f8ec40cef00 con 0x7f8ed4100790 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949a89b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 -- 172.21.15.69:0/1556947796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 msgr2=0x55a8949ab4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949ab4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 -- 172.21.15.69:0/1556947796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 msgr2=0x55a8949a89b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949a89b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 -- 172.21.15.69:0/1556947796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8946e37e0 con 0x55a8949a34c0 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 0x7f8ec00a5050 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4199060 tx=0x7f8ebc005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.728+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc005cc0 tx=0x7f8ecc009800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fd5ac53eaf52b5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8ec8ff9640 1 -- 172.21.15.69:0/1556947796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc031450 con 0x55a8949a34c0 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8ec8ff9640 1 -- 172.21.15.69:0/1556947796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc036050 con 0x55a8949a34c0 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8ec8ff9640 1 -- 172.21.15.69:0/1556947796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ecc004460 con 0x55a8949a34c0 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 -- 172.21.15.69:0/1556947796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 msgr2=0x55a8949a38e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7f8ecc005cc0 tx=0x7f8ecc009800 comp rx=0 tx=0).stop 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 -- 172.21.15.69:0/1556947796 shutdown_connections 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949ab4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949a89b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1556947796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949a38e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 -- 172.21.15.69:0/1556947796 >> 172.21.15.69:0/1556947796 conn(0x55a89496b0a0 msgr2=0x55a89499ef40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 -- 172.21.15.69:0/1556947796 shutdown_connections 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.729+0000 7f8edbba5740 1 -- 172.21.15.69:0/1556947796 wait complete. 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 -- start start 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 Processor -- start 2024-05-09T09:20:28.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949ade80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a8949abf70 con 0x55a8949a34c0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a8948c2b40 con 0x55a8949a9020 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a8949ac0f0 con 0x55a8949a3eb0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8edbba5740 10 client.0 _ll_register_callbacks cb 0x55a8948b70f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39176/0 (socket says 172.21.15.69:39176) 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 -- 172.21.15.69:0/3277524386 learned_addr learned my addr 172.21.15.69:0/3277524386 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949ade80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 -- 172.21.15.69:0/3277524386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 msgr2=0x55a8949ade80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949ade80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 -- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 msgr2=0x55a8949b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed259c640 1 -- 172.21.15.69:0/3277524386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a894743f70 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.730+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8ed259c640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40cc680 tx=0x7f8ec4009de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9938b10c4d44bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4021070 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f8ec40a3bc0 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ecc036050 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ec409f790 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 10 client.16050 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a8945db170 con 0x55a8949a34c0 2024-05-09T09:20:28.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ecc036050 con 0x55a8949a34c0 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8ec40ccc60 con 0x55a8949a34c0 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 10 client.16050 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 10 client.16050 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 20 client.16050 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a8949b9360 0x55a8949bb820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 4 client.16050 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.731+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a8949150c0 con 0x55a8949b9360 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.732+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a8949b9360 0x55a8949bb820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.732+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a8949b9360 0x55a8949bb820 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=26a03d11299204d5 server_cookie=738531a18ce7bbec in_seq=0 out_seq=0 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.732+0000 7f8eb9ffb640 10 client.16050 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/3277524386 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a8949150c0 con 0x55a8949b9360 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8eb9ffb640 10 client.16050 handle_command_reply: tid=0 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 client.16050 shutdown 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 msgr2=0x55a8949ad940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f8ec40cc680 tx=0x7f8ec4009de0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 shutdown_connections 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a8949b9360 0x55a8949bb820 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949ade80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 --2- 172.21.15.69:0/3277524386 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949ad940 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8eb9ffb640 0 client.16050 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 >> 172.21.15.69:0/3277524386 conn(0x55a89496b0a0 msgr2=0x55a89499e080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 shutdown_connections 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 1 -- 172.21.15.69:0/3277524386 wait complete. 2024-05-09T09:20:28.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.733+0000 7f8edbba5740 20 client.16050 trim_cache size 0 max 16384 2024-05-09T09:20:28.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.750+0000 7f8edbba5740 1 Processor -- start 2024-05-09T09:20:28.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 -- start start 2024-05-09T09:20:28.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949a2540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41acc20 con 0x55a8949a34c0 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41aa490 con 0x55a8949a3eb0 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8edbba5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41a82e0 con 0x55a8949a9020 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39180/0 (socket says 172.21.15.69:39180) 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 -- 172.21.15.69:0/2904608604 learned_addr learned my addr 172.21.15.69:0/2904608604 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 -- 172.21.15.69:0/2904608604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 msgr2=0x55a8949a2540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949a2540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 -- 172.21.15.69:0/2904608604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 msgr2=0x55a8949b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 -- 172.21.15.69:0/2904608604 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a8949bbea0 con 0x55a8949a34c0 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ed259c640 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009610 tx=0x7f8ec40998f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83597acb41b5b1f4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ebbfff640 1 -- 172.21.15.69:0/2904608604 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4021070 con 0x55a8949a34c0 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ebbfff640 1 -- 172.21.15.69:0/2904608604 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4015070 con 0x55a8949a34c0 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.751+0000 7f8ebbfff640 1 -- 172.21.15.69:0/2904608604 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4097bb0 con 0x55a8949a34c0 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- 172.21.15.69:0/2904608604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 msgr2=0x55a8949adde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f8ec4009610 tx=0x7f8ec40998f0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- 172.21.15.69:0/2904608604 shutdown_connections 2024-05-09T09:20:28.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a9020 0x55a8949a2540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a3eb0 0x55a8949b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- 172.21.15.69:0/2904608604 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949adde0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- 172.21.15.69:0/2904608604 >> 172.21.15.69:0/2904608604 conn(0x55a89496b0a0 msgr2=0x55a8949a1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- 172.21.15.69:0/2904608604 shutdown_connections 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- 172.21.15.69:0/2904608604 wait complete. 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 -- start start 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 Processor -- start 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949b4de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.752+0000 7f8edbba5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b2f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41a8950 con 0x55a8949a34c0 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41a9dc0 con 0x55a8949a9020 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8ed41a9020 con 0x55a8949a3eb0 2024-05-09T09:20:28.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 10 client.0 _ll_register_callbacks cb 0x7f8ec4095f50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39184/0 (socket says 172.21.15.69:39184) 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 -- 172.21.15.69:0/1164481064 learned_addr learned my addr 172.21.15.69:0/1164481064 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b2f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 -- 172.21.15.69:0/1164481064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 msgr2=0x55a8949b4de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949b4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 -- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 msgr2=0x55a8949b2f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b2f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 -- 172.21.15.69:0/1164481064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a8949b3750 con 0x55a8949a34c0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed2d9d640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8ed259c640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8ec409b370 tx=0x7f8ec409b1b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=241565e3e1538692 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8eb9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8ec4021070 con 0x55a8949a34c0 2024-05-09T09:20:28.761 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f8ec40ccb40 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8ec4015070 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f8ec40ccd40 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 10 client.16056 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x55a8949b3750 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.753+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8ec4015070 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f8ec409a9c0 con 0x55a8949a34c0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 10 client.16056 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 10 client.16056 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 20 client.16056 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a894a20b10 0x55a894a22f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 4 client.16056 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) -- 0x55a89490f410 con 0x55a894a20b10 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a894a20b10 0x55a894a22f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8eb9ffb640 10 client.16056 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.762 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.754+0000 7f8ed1d9b640 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a894a20b10 0x55a894a22f30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe9774aa98a5890 server_cookie=ee0f123b06b33c61 in_seq=0 out_seq=0 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8eb9ffb640 1 -- 172.21.15.69:0/1164481064 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+110 (crc 0 0 0) 0x55a89490f410 con 0x55a894a20b10 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8eb9ffb640 10 client.16056 handle_command_reply: tid=0 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 client.16056 shutdown 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 msgr2=0x55a8949b48a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8ec409b370 tx=0x7f8ec409b1b0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 shutdown_connections 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a8949a9020 0x55a8949b2f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a894a20b10 0x55a894a22f30 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a8949a3eb0 0x55a8949b4de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 --2- 172.21.15.69:0/1164481064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a8949a34c0 0x55a8949b48a0 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 >> 172.21.15.69:0/1164481064 conn(0x55a89496b0a0 msgr2=0x55a8949adb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8eb9ffb640 0 client.16056 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 shutdown_connections 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.755+0000 7f8edbba5740 1 -- 172.21.15.69:0/1164481064 wait complete. 2024-05-09T09:20:28.763 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8edbba5740 20 client.16056 trim_cache size 0 max 16384 2024-05-09T09:20:28.764 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:20:28.764 INFO:teuthology.orchestra.run.smithi069.stdout: "return_code": 0, 2024-05-09T09:20:28.764 INFO:teuthology.orchestra.run.smithi069.stdout: "scrub_tag": "fd017bb8-2ab1-498d-a446-41e81feec31c", 2024-05-09T09:20:28.764 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": "asynchronous" 2024-05-09T09:20:28.764 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 msgr2=0x7f8ec00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 0x7f8ec00a5050 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f8ed4199060 tx=0x7f8ebc005d30 comp rx=0 tx=0).stop 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 msgr2=0x7f8ed4197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7f8ec400d950 tx=0x7f8ec400de10 comp rx=0 tx=0).stop 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 shutdown_connections 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8ec00a2b90 0x7f8ec00a5050 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8ed4102340 0x7f8ed419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8ed4101980 0x7f8ed4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.765 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 --2- 172.21.15.69:0/1613157574 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8ed4100790 0x7f8ed4197b50 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 >> 172.21.15.69:0/1613157574 conn(0x7f8ed40fbd20 msgr2=0x7f8ed4104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 shutdown_connections 2024-05-09T09:20:28.766 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.756+0000 7f8ed8db7640 1 -- 172.21.15.69:0/1613157574 wait complete. 2024-05-09T09:20:28.768 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:20:28.768 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:20:28.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- 172.21.15.69:0/2075764927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 msgr2=0x7f3470101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 --2- 172.21.15.69:0/2075764927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f3470101de0 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f3458009900 tx=0x7f345802a410 comp rx=0 tx=0).stop 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- 172.21.15.69:0/2075764927 shutdown_connections 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 --2- 172.21.15.69:0/2075764927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 0x7f3470108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 --2- 172.21.15.69:0/2075764927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f3470101de0 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 --2- 172.21.15.69:0/2075764927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 0x7f3470100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- 172.21.15.69:0/2075764927 >> 172.21.15.69:0/2075764927 conn(0x7f34700fbce0 msgr2=0x7f34700fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- 172.21.15.69:0/2075764927 shutdown_connections 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- 172.21.15.69:0/2075764927 wait complete. 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 Processor -- start 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.875+0000 7f347675c640 1 -- start start 2024-05-09T09:20:28.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 0x7f3470197c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 0x7f347019f210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f347010d570 con 0x7f3470101960 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f347010d3f0 con 0x7f3470102320 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f347010d6f0 con 0x7f3470100770 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39206/0 (socket says 172.21.15.69:39206) 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 -- 172.21.15.69:0/2432894521 learned_addr learned my addr 172.21.15.69:0/2432894521 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f346ffff640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 0x7f3470197c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 0x7f347019f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 -- 172.21.15.69:0/2432894521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 msgr2=0x7f3470197c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 0x7f3470197c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 -- 172.21.15.69:0/2432894521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 msgr2=0x7f347019f210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 0x7f347019f210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 -- 172.21.15.69:0/2432894521 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f347019f8b0 con 0x7f3470101960 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f3467fff640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3458009230 tx=0x7f3458004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f346dffb640 1 -- 172.21.15.69:0/2432894521 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f345803f030 con 0x7f3470101960 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f347019fb40 con 0x7f3470101960 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f346dffb640 1 -- 172.21.15.69:0/2432894521 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3458003ce0 con 0x7f3470101960 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f346dffb640 1 -- 172.21.15.69:0/2432894521 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458035070 con 0x7f3470101960 2024-05-09T09:20:28.880 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.876+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34701a0080 con 0x7f3470101960 2024-05-09T09:20:28.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f346dffb640 1 -- 172.21.15.69:0/2432894521 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3458042030 con 0x7f3470101960 2024-05-09T09:20:28.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 Processor -- start 2024-05-09T09:20:28.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 -- start start 2024-05-09T09:20:28.888 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd2d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd2da90 0x556bfcd2ff20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556bfcc8cd00 con 0x556bfcd27f30 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bfcc344c0 con 0x556bfcd2da90 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3479460740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556bfcc88cc0 con 0x556bfcd28920 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3474cf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3474cf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39214/0 (socket says 172.21.15.69:39214) 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.879+0000 7f3474cf9640 1 -- 172.21.15.69:0/424137567 learned_addr learned my addr 172.21.15.69:0/424137567 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f346dffb640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 0x7f345c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f346ffff640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd2d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3467fff640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd2da90 0x556bfcd2ff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 -- 172.21.15.69:0/424137567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 msgr2=0x556bfcd2d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd2d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 -- 172.21.15.69:0/424137567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd2da90 msgr2=0x556bfcd2ff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.889 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd2da90 0x556bfcd2ff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 -- 172.21.15.69:0/424137567 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bfca6a7d0 con 0x556bfcd27f30 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f346ffff640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd2d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f346dffb640 1 -- 172.21.15.69:0/2432894521 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f34580efec0 con 0x7f3470101960 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 0x7f345c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f3474cf9640 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f346800f4e0 tx=0x7f346800f9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=623d6b34d39e87b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f34667fc640 1 -- 172.21.15.69:0/424137567 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34680124b0 con 0x556bfcd27f30 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f34667fc640 1 -- 172.21.15.69:0/424137567 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346801c8e0 con 0x556bfcd27f30 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.880+0000 7f34667fc640 1 -- 172.21.15.69:0/424137567 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346801b740 con 0x556bfcd27f30 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- 172.21.15.69:0/424137567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 msgr2=0x556bfcd28350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 0x7f345c0a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3468020000 tx=0x7f346801d000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f346800f4e0 tx=0x7f346800f9a0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- 172.21.15.69:0/424137567 shutdown_connections 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd2da90 0x556bfcd2ff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd2d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 --2- 172.21.15.69:0/424137567 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd27f30 0x556bfcd28350 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- 172.21.15.69:0/424137567 >> 172.21.15.69:0/424137567 conn(0x556bfcd235a0 msgr2=0x556bfcd239a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- 172.21.15.69:0/424137567 shutdown_connections 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- 172.21.15.69:0/424137567 wait complete. 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 -- start start 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.881+0000 7f3479460740 1 Processor -- start 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd323b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd39930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556bfcd30a40 con 0x556bfcd28920 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556bfcc47b90 con 0x556bfcd27f30 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556bfcd30bc0 con 0x556bfcd2da90 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 10 client.0 _ll_register_callbacks cb 0x556bfcc83da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:28.891 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39230/0 (socket says 172.21.15.69:39230) 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 -- 172.21.15.69:0/2603710474 learned_addr learned my addr 172.21.15.69:0/2603710474 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd39930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f346ffff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd323b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 -- 172.21.15.69:0/2603710474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 msgr2=0x556bfcd39930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd39930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 -- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 msgr2=0x556bfcd323b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd323b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 -- 172.21.15.69:0/2603710474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bfc9a03f0 con 0x556bfcd28920 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f346ffff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd39930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3467fff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3458004650 tx=0x7f34580bc110 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dfd76615ee7e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f344f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458049040 con 0x556bfcd28920 2024-05-09T09:20:28.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f34580c0420 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346801c8e0 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f34580bca40 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 10 client.16068 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.882+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bfc9600f0 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346801c8e0 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f34580b89b0 con 0x556bfcd28920 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 10 client.16068 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 10 client.16068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 20 client.16068 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcd3e130 0x556bfcd40590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 4 client.16068 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556bfcc9a430 con 0x556bfcd3e130 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f346ffff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcd3e130 0x556bfcd40590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f344f7fe640 10 client.16068 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.883+0000 7f346ffff640 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcd3e130 0x556bfcd40590 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=488b44cebc78e033 server_cookie=db06375b7e23441a in_seq=0 out_seq=0 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f344f7fe640 1 -- 172.21.15.69:0/2603710474 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556bfcc9a430 con 0x556bfcd3e130 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f344f7fe640 10 client.16068 handle_command_reply: tid=0 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 client.16068 shutdown 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 msgr2=0x556bfcd328f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f3458004650 tx=0x7f34580bc110 comp rx=0 tx=0).stop 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 shutdown_connections 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcd3e130 0x556bfcd40590 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd39930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f344f7fe640 0 client.16068 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd328f0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 --2- 172.21.15.69:0/2603710474 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd323b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 >> 172.21.15.69:0/2603710474 conn(0x556bfcd235a0 msgr2=0x556bfcd227e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 shutdown_connections 2024-05-09T09:20:28.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 1 -- 172.21.15.69:0/2603710474 wait complete. 2024-05-09T09:20:28.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.884+0000 7f3479460740 20 client.16068 trim_cache size 0 max 16384 2024-05-09T09:20:28.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 Processor -- start 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 -- start start 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd29d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd398f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34701acdb0 con 0x556bfcd2da90 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34701aa620 con 0x556bfcd27f30 2024-05-09T09:20:28.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3479460740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34701a8470 con 0x556bfcd28920 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39238/0 (socket says 172.21.15.69:39238) 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 -- 172.21.15.69:0/2883075654 learned_addr learned my addr 172.21.15.69:0/2883075654 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3467fff640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd398f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f346ffff640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd29d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 -- 172.21.15.69:0/2883075654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 msgr2=0x556bfcd398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd398f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 -- 172.21.15.69:0/2883075654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 msgr2=0x556bfcd29d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd29d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f3474cf9640 1 -- 172.21.15.69:0/2883075654 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556bfcd40c10 con 0x556bfcd2da90 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.902+0000 7f346ffff640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd29d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3474cf9640 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f34680098f0 tx=0x7f3468027000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa154b320fa50e1b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3465ffb640 1 -- 172.21.15.69:0/2883075654 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346801baa0 con 0x556bfcd2da90 2024-05-09T09:20:28.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3465ffb640 1 -- 172.21.15.69:0/2883075654 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346802a030 con 0x556bfcd2da90 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3465ffb640 1 -- 172.21.15.69:0/2883075654 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f346801b720 con 0x556bfcd2da90 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 -- 172.21.15.69:0/2883075654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 msgr2=0x556bfcd2a2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f34680098f0 tx=0x7f3468027000 comp rx=0 tx=0).stop 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 -- 172.21.15.69:0/2883075654 shutdown_connections 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd2da90 0x556bfcd2a2c0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd28920 0x556bfcd398f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 --2- 172.21.15.69:0/2883075654 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd29d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 -- 172.21.15.69:0/2883075654 >> 172.21.15.69:0/2883075654 conn(0x556bfcd235a0 msgr2=0x556bfcd323b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 -- 172.21.15.69:0/2883075654 shutdown_connections 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.903+0000 7f3479460740 1 -- 172.21.15.69:0/2883075654 wait complete. 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 -- start start 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 Processor -- start 2024-05-09T09:20:28.914 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd36d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd38250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34701a8ae0 con 0x556bfcd28920 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34701a9f50 con 0x556bfcd27f30 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34701a91b0 con 0x556bfcd2da90 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3479460740 10 client.0 _ll_register_callbacks cb 0x7f34580b6d40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3467fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39254/0 (socket says 172.21.15.69:39254) 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3467fff640 1 -- 172.21.15.69:0/1508590907 learned_addr learned my addr 172.21.15.69:0/1508590907 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f346ffff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd36d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.915 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.904+0000 7f3474cf9640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd38250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 -- 172.21.15.69:0/1508590907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 msgr2=0x556bfcd38250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd38250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 -- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 msgr2=0x556bfcd36d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd36d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 -- 172.21.15.69:0/1508590907 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556bfcd38840 con 0x556bfcd28920 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3474cf9640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd38250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3467fff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3458009a20 tx=0x7f34580095e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=915e5d7dcf20e114 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3458049070 con 0x556bfcd28920 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f34580c4b20 con 0x556bfcd28920 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f346802a030 con 0x556bfcd28920 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) ==== 24+0+0 (secure 0 0 0) 0x7f34580bc800 con 0x556bfcd28920 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 10 client.16074 fetch_fsmap learned FSMap version 28 2024-05-09T09:20:28.916 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) -- 0x556bfcd39b50 con 0x556bfcd28920 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f346802a030 con 0x556bfcd28920 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 28) ==== 8354+0+0 (secure 0 0 0) 0x7f34580bb790 con 0x556bfcd28920 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 10 client.16074 fetch_fsmap finished waiting for FSMap version 28 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 10 client.16074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 20 client.16074 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcda4c20 0x556bfcda7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 4 client.16074 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.905+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x556bfcc94780 con 0x556bfcda4c20 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f346ffff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcda4c20 0x556bfcda7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f344f7fe640 10 client.16074 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f346ffff640 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcda4c20 0x556bfcda7040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=402ca97567a1233e server_cookie=f43afcc0c8851e in_seq=0 out_seq=0 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f344f7fe640 1 -- 172.21.15.69:0/1508590907 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+259 (crc 0 0 0) 0x556bfcc94780 con 0x556bfcda4c20 2024-05-09T09:20:28.917 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f344f7fe640 10 client.16074 handle_command_reply: tid=0 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 client.16074 shutdown 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 msgr2=0x556bfcd37240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f3458009a20 tx=0x7f34580095e0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 shutdown_connections 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556bfcda4c20 0x556bfcda7040 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556bfcd2da90 0x556bfcd38250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556bfcd28920 0x556bfcd37240 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f344f7fe640 0 client.16074 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 --2- 172.21.15.69:0/1508590907 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556bfcd27f30 0x556bfcd36d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 >> 172.21.15.69:0/1508590907 conn(0x556bfcd235a0 msgr2=0x556bfcd32280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 shutdown_connections 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 1 -- 172.21.15.69:0/1508590907 wait complete. 2024-05-09T09:20:28.918 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.906+0000 7f3479460740 20 client.16074 trim_cache size 0 max 16384 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "IDLE (waiting for more scrubs)", 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": { 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "fd017bb8-2ab1-498d-a446-41e81feec31c": { 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/", 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "tag": "fd017bb8-2ab1-498d-a446-41e81feec31c", 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: "options": "recursive,force" 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:20:28.919 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 msgr2=0x7f345c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 0x7f345c0a5050 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f3468020000 tx=0x7f346801d000 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 msgr2=0x7f34701981b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f3458009230 tx=0x7f3458004360 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 shutdown_connections 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f345c0a2b90 0x7f345c0a5050 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3470102320 0x7f347019f210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3470101960 0x7f34701981b0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 --2- 172.21.15.69:0/2432894521 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3470100770 0x7f3470197c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 >> 172.21.15.69:0/2432894521 conn(0x7f34700fbce0 msgr2=0x7f3470104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:28.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 shutdown_connections 2024-05-09T09:20:28.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:28.907+0000 7f347675c640 1 -- 172.21.15.69:0/2432894521 wait complete. 2024-05-09T09:20:28.922 INFO:tasks.cephfs.filesystem:scrub status for tag:fd017bb8-2ab1-498d-a446-41e81feec31c - {'path': '/', 'tag': 'fd017bb8-2ab1-498d-a446-41e81feec31c', 'options': 'recursive,force'} 2024-05-09T09:20:29.056 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.767+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022c [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04 auth v938 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee67b700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022c:[<0x10000000229/COURSES.X04 v6>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022c:[<0x10000000229/COURSES.X04 v938>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.056 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.768+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000231 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0 auth v987 pv1117 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.675323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.675323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee642000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000231:[<0x10000000229/STUDENTS.YG0 v16>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000231:[<0x10000000229/STUDENTS.YG0 v987>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.056 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.768+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000237 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX auth v999 pv1119 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.703323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.703323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee642580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000237:[<0x10000000229/ENROLL.PX v28>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000237:[<0x10000000229/ENROLL.PX v999>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.056 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.768+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022f [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX auth v983 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee642b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022f:[<0x10000000229/STUDENTS.PX v12>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022f:[<0x10000000229/STUDENTS.PX v983>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.768+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022b [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX auth v936 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee67b180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022b:[<0x10000000229/COURSES.PX v4>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022b:[<0x10000000229/COURSES.PX v936>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.769+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023b [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV auth v1073 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.722322+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee643080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023b:[<0x10000000229/STUDENTS.TV v36>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000023b:[<0x10000000229/STUDENTS.TV v1073>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.769+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023c [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL auth v1075 pv1121 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.727322+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.727322+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee643600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023c:[<0x10000000229/STUDENTS.FSL v38>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000023c:[<0x10000000229/STUDENTS.FSL v1075>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.769+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000233 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1 auth v991 pv1123 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.684323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.684323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee643b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000233:[<0x10000000229/STUDENTS.YG1 v20>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000233:[<0x10000000229/STUDENTS.YG1 v991>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.770+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000239 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03 auth v1003 pv1125 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.713322+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.713322+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee644100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000239:[<0x10000000229/ENROLL.Y03 v32>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000239:[<0x10000000229/ENROLL.Y03 v1003>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.770+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023d [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL auth v1077 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.732322+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee644680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023d:[<0x10000000229/COURSES.FSL v40>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000023d:[<0x10000000229/COURSES.FSL v1077>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.770+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000234 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL auth v993 pv1127 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.689323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.689323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee644c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000234:[<0x10000000229/STUDENTS.VAL v22>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000234:[<0x10000000229/STUDENTS.VAL v993>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.770+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022d [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04 auth v940 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee645700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022d:[<0x10000000229/COURSES.Y04 v8>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022d:[<0x10000000229/COURSES.Y04 v940>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.771+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022a [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB auth v934 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee645180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022a:[<0x10000000229/COURSES.DB v2>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022a:[<0x10000000229/COURSES.DB v934>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.771+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000235 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB auth v995 pv1131 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.693323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.693323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f2580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000235:[<0x10000000229/STUDENTS.MB v24>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000235:[<0x10000000229/STUDENTS.MB v995>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.771+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000236 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB auth v997 pv1135 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.698323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.698323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f3080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000236:[<0x10000000229/ENROLL.DB v26>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000236:[<0x10000000229/ENROLL.DB v997>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.057 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.772+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000238 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03 auth v1001 pv1129 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.708322+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.708322+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f2000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000238:[<0x10000000229/ENROLL.X03 v30>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000238:[<0x10000000229/ENROLL.X03 v1001>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.058 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.772+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000232 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1 auth v989 pv1137 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.680323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.680323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f3600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000232:[<0x10000000229/STUDENTS.XG1 v18>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000232:[<0x10000000229/STUDENTS.XG1 v989>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.058 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.772+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000023a [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL auth v1005 pv1133 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.718322+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.718322+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f2b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000023a:[<0x10000000229/ENROLL.VAL v34>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000023a:[<0x10000000229/ENROLL.VAL v1005>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.058 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.772+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000230 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0 auth v985 pv1139 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.671323+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.671323+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f3b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000230:[<0x10000000229/STUDENTS.XG0 v14>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000230:[<0x10000000229/STUDENTS.XG0 v985>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.058 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:20:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:20:28.772+0000 7fee7194e700 -1 mds.1.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000022e [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB auth v981 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.367101+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@30},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x55d5ee6f4100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000022e:[<0x10000000229/STUDENTS.DB v10>,<0x10000000009/PARADOX v4036>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000022e:[<0x10000000229/STUDENTS.DB v981>,<0x10000000009/PARADOX v6459>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.118 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.764+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021f [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT auth v653 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.614325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c42580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021f:[<0x10000000215/ROSE.CPT v20>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021f:[<0x10000000215/ROSE.CPT v653>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.764+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021b [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT auth v619 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.593325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c42b00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v12>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021b:[<0x10000000215/TEAPOT2.CPT v619>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.764+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021a [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT auth v617 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.588325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c43080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v10>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021a:[<0x10000000215/MAILBOX2.CPT v617>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.764+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000220 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT auth v600 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.618325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c43600]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000220:[<0x10000000215/CORELDRW.CDT v22>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000220:[<0x10000000215/CORELDRW.CDT v600>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.765+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021e [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR auth v638 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.609325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c43b80]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v18>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021e:[<0x10000000215/GRAPHIC2.CDR v638>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.765+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000218 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL auth v593 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.576325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c44100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000218:[<0x10000000215/ARTISTIC.ACL v6>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000218:[<0x10000000215/ARTISTIC.ACL v593>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.119 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.765+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021c [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT auth v621 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.599325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c44680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021c:[<0x10000000215/SIGN2.CPT v14>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021c:[<0x10000000215/SIGN2.CPT v621>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.766+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000217 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG auth v591 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.910088+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c44c00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000217:[<0x10000000215/CDRBARS.CFG v4>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000217:[<0x10000000215/CDRBARS.CFG v591>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.767+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x1000000021d [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR auth v623 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.604325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c45180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x1000000021d:[<0x10000000215/TRAIN.CDR v16>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x1000000021d:[<0x10000000215/TRAIN.CDR v623>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.767+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000219 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR auth v602 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.582325+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c45700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v8>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000219:[<0x10000000215/GRAPHIC1.CDR v602>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.768+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000216 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG auth v589 ap=1 s=260096 n(v0 rc2024-05-09T09:20:12.919088+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@36},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c5c000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000216:[<0x10000000215/CDRROLS.CFG v2>,<0x10000000009/COREL v4284>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000216:[<0x10000000215/CDRROLS.CFG v589>,<0x10000000009/COREL v6437>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.768+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000226 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS auth v1340 pv1440 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.633324+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.633324+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c5c580]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000226:[<0x10000000225/T1.XLS v2>,<0x10000000009/PM v4374>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000226:[<0x10000000225/T1.XLS v1340>,<0x10000000009/PM v6415>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.769+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000227 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC auth v1432 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.638324+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@136},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c5cb00]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000227:[<0x10000000225/EVENTS.DOC v4>,<0x10000000009/PM v4374>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000227:[<0x10000000225/EVENTS.DOC v1432>,<0x10000000009/PM v6415>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.769+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000228 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC auth v1434 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.642324+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@136},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923c5d080]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000228:[<0x10000000225/MOVED.DOC v6>,<0x10000000009/PM v4374>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000228:[<0x10000000225/MOVED.DOC v1434>,<0x10000000009/PM v6415>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.769+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000210 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT auth v1097 ap=1 s=260096 n(v0 rc2024-05-09T09:20:26.689772+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@33},l=15444 | ptrwaiter=0 request=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x555923bf6100]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000210:[<0x1000000020f/PCBENCHM.PPT v2>,<0x10000000009/PWRPNT v3880>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000210:[<0x1000000020f/PCBENCHM.PPT v1097>,<0x10000000009/PWRPNT v6479>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.770+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000211 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP auth{0=1} v1118 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.547326+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@32},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 waiter=0 authpin=1 scrubqueue=0 0x555923bf6680]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000211:[<0x1000000020f/ZD16.BMP v4>,<0x10000000009/PWRPNT v3880>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000211:[<0x1000000020f/ZD16.BMP v1118>,<0x10000000009/PWRPNT v6479>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.120 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.770+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000214 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT auth v1061 pv1130 ap=2 s=260096 n(v0 rc2024-05-09T09:20:02.561326+0000 b260096 1=1+0)->n(v0 rc2024-05-09T09:20:02.561326+0000 b260096 1=1+0) (ifile lock w=1) cr={15444=0-4194304@1} | ptrwaiter=1 lock=1 importingcaps=0 caps=0 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923bf7180]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000214:[<0x1000000020f/TRIDOTS.POT v10>,<0x10000000009/PWRPNT v3880>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000214:[<0x1000000020f/TRIDOTS.POT v1061>,<0x10000000009/PWRPNT v6479>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.121 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.771+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000213 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT auth v1059 ap=1 s=260096 n(v0 rc2024-05-09T09:20:20.346918+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@28},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=0 dirty=1 authpin=1 scrubqueue=0 0x555923bf7700]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000213:[<0x1000000020f/TIPS.PPT v8>,<0x10000000009/PWRPNT v3880>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000213:[<0x1000000020f/TIPS.PPT v1059>,<0x10000000009/PWRPNT v6479>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.121 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:20:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:20:28.771+0000 7f194ed14700 -1 mds.2.scrubstack _validate_inode_done scrub error on inode [inode 0x10000000212 [2,head] /volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA auth{0=1} v1067 ap=1 s=260096 n(v0 rc2024-05-09T09:20:02.552326+0000 b260096 1=1+0) (iauth excl) (ifile excl) (ixattr excl) cr={15444=0-4194304@1} caps={15444=pAsxLsXsxFsxcrwb/pAsxXsxFsxcrwb@28},l=15444 | ptrwaiter=0 lock=0 importingcaps=0 caps=1 replicated=1 dirty=1 authpin=1 scrubqueue=0 0x555923c42000]: {"performed_validation":true,"passed_validation":false,"backtrace":{"checked":true,"passed":false,"read_ret_val":0,"ondisk_value":"(4)0x10000000212:[<0x1000000020f/PPTOOLS1.PPA v6>,<0x10000000009/PWRPNT v3880>,<0x10000000008/~dmtmp v44>,<0x10000000007/client0 v28>,<0x10000000006/clients v18>,<0x10000000005/tmp v18>,<0x100000001fb/client.0 v20>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v36>,<0x10000000001/sv_1 v59>,<0x10000000000/qa v47>,<0x1/volumes v45>]//[3]","memoryvalue":"(4)0x10000000212:[<0x1000000020f/PPTOOLS1.PPA v1067>,<0x10000000009/PWRPNT v6479>,<0x10000000008/~dmtmp v212>,<0x10000000007/client0 v58>,<0x10000000006/clients v39>,<0x10000000005/tmp v39>,<0x100000001fb/client.0 v41>,<0x100000001fa/84be3150-10b2-4ab3-ab07-1a2632c1c47d v53>,<0x10000000001/sv_1 v73>,<0x10000000000/qa v25>,<0x1/volumes v23>]//[3]","error_str":"On-disk backtrace is divergent or newer"},"raw_stats":{"checked":false,"passed":false,"read_ret_val":0,"ondisk_value.dirstat":"f()","ondisk_value.rstat":"n()","memory_value.dirstat":"f()","memory_value.rstat":"n()","error_str":""},"return_code":0} 2024-05-09T09:20:29.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 32755 7.17 MB/sec warmup 27 sec latency 2029.952 ms 2024-05-09T09:20:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub queued for path: / 2024-05-09T09:20:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: active paths [/] 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:29 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub queued for path: / 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: active paths [/] 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.620 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:29 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub queued for path: / 2024-05-09T09:20:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000220 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000218 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: active paths [/] 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000217 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000021d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000231 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000237 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000219 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022f (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000216 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000023b (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000226 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000023c (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000227 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000233 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000228 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000239 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000210 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000023d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000211 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000234 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022d (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000214 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000213 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000235 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000212 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA) see mds.h log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000236 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:29.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000238 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000232 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000023a (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x10000000230 (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: Scrub error on inode 0x1000000022e (/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB) see mds.i log and `damage ls` output for details 2024-05-09T09:20:29.764 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:29 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:30.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 33367 7.08 MB/sec warmup 28 sec latency 1396.540 ms 2024-05-09T09:20:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:30 smithi163 ceph-mon[34354]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:30 smithi163 ceph-mon[34354]: pgmap v414: 129 pgs: 129 active+clean; 23 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.4 MiB/s wr, 201 op/s 2024-05-09T09:20:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:30 smithi103 ceph-mon[34421]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:30 smithi103 ceph-mon[34421]: pgmap v414: 129 pgs: 129 active+clean; 23 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.4 MiB/s wr, 201 op/s 2024-05-09T09:20:30.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:30 smithi069 ceph-mon[30123]: scrub summary: idle+waiting paths [/] 2024-05-09T09:20:30.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:30 smithi069 ceph-mon[30123]: pgmap v414: 129 pgs: 129 active+clean; 23 MiB data, 426 MiB used, 1.0 TiB / 1.0 TiB avail; 140 KiB/s rd, 4.4 MiB/s wr, 201 op/s 2024-05-09T09:20:31.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 35899 7.32 MB/sec warmup 29 sec latency 11.564 ms 2024-05-09T09:20:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:31 smithi069 ceph-mon[30123]: pgmap v415: 129 pgs: 129 active+clean; 24 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 4.5 MiB/s wr, 224 op/s 2024-05-09T09:20:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:31 smithi163 ceph-mon[34354]: pgmap v415: 129 pgs: 129 active+clean; 24 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 4.5 MiB/s wr, 224 op/s 2024-05-09T09:20:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:31 smithi103 ceph-mon[34421]: pgmap v415: 129 pgs: 129 active+clean; 24 MiB data, 425 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 4.5 MiB/s wr, 224 op/s 2024-05-09T09:20:32.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36174 7.08 MB/sec warmup 30 sec latency 705.727 ms 2024-05-09T09:20:32.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:32 smithi103 ceph-mon[34421]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:32.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:32 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:20:32.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:32 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:32 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:20:32.537+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:32 smithi069 ceph-mon[30123]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:32 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:20:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:32 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:32 smithi163 ceph-mon[34354]: Health check failed: 1 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:32 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] up:active 2024-05-09T09:20:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:32 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:33.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36174 6.85 MB/sec warmup 31 sec latency 1705.826 ms 2024-05-09T09:20:33.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:33 smithi103 ceph-mon[34421]: pgmap v416: 129 pgs: 129 active+clean; 24 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-05-09T09:20:34.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:33 smithi069 ceph-mon[30123]: pgmap v416: 129 pgs: 129 active+clean; 24 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-05-09T09:20:34.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:33 smithi163 ceph-mon[34354]: pgmap v416: 129 pgs: 129 active+clean; 24 MiB data, 430 MiB used, 1.0 TiB / 1.0 TiB avail; 469 KiB/s rd, 4.6 MiB/s wr, 238 op/s 2024-05-09T09:20:34.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36174 6.63 MB/sec warmup 32 sec latency 2705.874 ms 2024-05-09T09:20:35.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36230 6.44 MB/sec warmup 33 sec latency 3464.186 ms 2024-05-09T09:20:36.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36230 6.25 MB/sec warmup 34 sec latency 1187.755 ms 2024-05-09T09:20:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:36 smithi069 ceph-mon[30123]: pgmap v417: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-05-09T09:20:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:36 smithi163 ceph-mon[34354]: pgmap v417: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-05-09T09:20:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:36 smithi103 ceph-mon[34421]: pgmap v417: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 586 KiB/s rd, 2.9 MiB/s wr, 219 op/s 2024-05-09T09:20:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- 172.21.15.69:0/195298786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 msgr2=0x7efc88109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 --2- 172.21.15.69:0/195298786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88109d00 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x7efc80009900 tx=0x7efc8002a410 comp rx=0 tx=0).stop 2024-05-09T09:20:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- 172.21.15.69:0/195298786 shutdown_connections 2024-05-09T09:20:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 --2- 172.21.15.69:0/195298786 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88109d00 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 --2- 172.21.15.69:0/195298786 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc88102ab0 0x7efc88102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 --2- 172.21.15.69:0/195298786 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc881018c0 0x7efc88101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- 172.21.15.69:0/195298786 >> 172.21.15.69:0/195298786 conn(0x7efc880fce50 msgr2=0x7efc880ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- 172.21.15.69:0/195298786 shutdown_connections 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- 172.21.15.69:0/195298786 wait complete. 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 Processor -- start 2024-05-09T09:20:36.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.736+0000 7efc8da53640 1 -- start start 2024-05-09T09:20:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 0x7efc880755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88102ab0 0x7efc88073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d520 con 0x7efc88103470 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d3a0 con 0x7efc881018c0 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc8810d6a0 con 0x7efc88102ab0 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39264/0 (socket says 172.21.15.69:39264) 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 -- 172.21.15.69:0/3406331931 learned_addr learned my addr 172.21.15.69:0/3406331931 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc867fc640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88102ab0 0x7efc88073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc86ffd640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 0x7efc880755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 -- 172.21.15.69:0/3406331931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88102ab0 msgr2=0x7efc88073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88102ab0 0x7efc88073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 -- 172.21.15.69:0/3406331931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 msgr2=0x7efc880755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 0x7efc880755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 -- 172.21.15.69:0/3406331931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efc880749f0 con 0x7efc88103470 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc86ffd640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 0x7efc880755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc877fe640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efc800314e0 tx=0x7efc800043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8ca51640 1 -- 172.21.15.69:0/3406331931 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc8003f030 con 0x7efc88103470 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8ca51640 1 -- 172.21.15.69:0/3406331931 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc80003ce0 con 0x7efc88103470 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.737+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efc881a1b30 con 0x7efc88103470 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.738+0000 7efc8ca51640 1 -- 172.21.15.69:0/3406331931 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc80035070 con 0x7efc88103470 2024-05-09T09:20:36.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.738+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efc881a2080 con 0x7efc88103470 2024-05-09T09:20:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.740+0000 7efc8ca51640 1 -- 172.21.15.69:0/3406331931 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efc80042030 con 0x7efc88103470 2024-05-09T09:20:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.740+0000 7efc90835740 1 Processor -- start 2024-05-09T09:20:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.740+0000 7efc90835740 1 -- start start 2024-05-09T09:20:36.751 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.740+0000 7efc90835740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cc6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.740+0000 7efc90835740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47ccb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc90835740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc90835740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bd47c2a340 con 0x55bd47ccbdb0 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc90835740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bd47bd2290 con 0x55bd47cc6250 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc90835740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bd47c26300 con 0x55bd47cc6c40 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39268/0 (socket says 172.21.15.69:39268) 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 -- 172.21.15.69:0/3006170531 learned_addr learned my addr 172.21.15.69:0/3006170531 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc86ffd640 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47ccb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc8ca51640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 0x7efc740a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cc6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc877fe640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 0x7efc740a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 -- 172.21.15.69:0/3006170531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 msgr2=0x55bd47ccb740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.752 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47ccb740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 -- 172.21.15.69:0/3006170531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 msgr2=0x55bd47cc6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cc6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 -- 172.21.15.69:0/3006170531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd47bf68e0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc8ca51640 1 -- 172.21.15.69:0/3406331931 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc800ef960 con 0x7efc88103470 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc867fc640 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efc70012540 tx=0x7efc7000e610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adcf6593def0d842 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc7dffb640 1 -- 172.21.15.69:0/3006170531 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70018070 con 0x55bd47ccbdb0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc7dffb640 1 -- 172.21.15.69:0/3006170531 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc700047a0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc877fe640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 0x7efc740a50f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efc800b69a0 tx=0x7efc800e69d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.741+0000 7efc7dffb640 1 -- 172.21.15.69:0/3006170531 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70010990 con 0x55bd47ccbdb0 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 -- 172.21.15.69:0/3006170531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 msgr2=0x55bd47cce240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7efc70012540 tx=0x7efc7000e610 comp rx=0 tx=0).stop 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 -- 172.21.15.69:0/3006170531 shutdown_connections 2024-05-09T09:20:36.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cce240 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47ccb740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 --2- 172.21.15.69:0/3006170531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cc6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 -- 172.21.15.69:0/3006170531 >> 172.21.15.69:0/3006170531 conn(0x55bd47cc18c0 msgr2=0x55bd47cc1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 -- 172.21.15.69:0/3006170531 shutdown_connections 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.742+0000 7efc90835740 1 -- 172.21.15.69:0/3006170531 wait complete. 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 -- start start 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 Processor -- start 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cd04c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47cd0a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bd47cced60 con 0x55bd47ccbdb0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bd47be5790 con 0x55bd47cc6250 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bd47cceee0 con 0x55bd47cc6c40 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc90835740 10 client.0 _ll_register_callbacks cb 0x55bd47c20a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc867fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc867fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39278/0 (socket says 172.21.15.69:39278) 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc867fc640 1 -- 172.21.15.69:0/2824776996 learned_addr learned my addr 172.21.15.69:0/2824776996 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc867fc640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cd04c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.743+0000 7efc86ffd640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47cd0a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 -- 172.21.15.69:0/2824776996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 msgr2=0x55bd47cd0a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47cd0a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 -- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 msgr2=0x55bd47cd04c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cd04c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 -- 172.21.15.69:0/2824776996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd4793e210 con 0x55bd47ccbdb0 2024-05-09T09:20:36.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc86ffd640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47cd0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc867fc640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efc70004e50 tx=0x7efc7000ebc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad2ea5f96e6d6574 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc70020020 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc70010940 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc700047a0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efc70033050 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 10 client.16086 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bd47a0a620 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc700047a0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7efc7001cab0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 10 client.16086 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 10 client.16086 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 20 client.16086 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:36.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.744+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47cdbd60 0x55bd47cde180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.745+0000 7efc90835740 4 client.16086 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.745+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bd47c37a70 con 0x55bd47cdbd60 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.745+0000 7efc86ffd640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47cdbd60 0x55bd47cde180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.745+0000 7efc86ffd640 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47cdbd60 0x55bd47cde180 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=88eb009a553f5280 server_cookie=b23b87237b5efec6 in_seq=0 out_seq=0 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.745+0000 7efc66ffd640 10 client.16086 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc66ffd640 1 -- 172.21.15.69:0/2824776996 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bd47c37a70 con 0x55bd47cdbd60 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc66ffd640 10 client.16086 handle_command_reply: tid=0 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 client.16086 shutdown 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 msgr2=0x55bd47cd7a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7efc70004e50 tx=0x7efc7000ebc0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 shutdown_connections 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd7a40 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47cdbd60 0x55bd47cde180 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6c40 0x55bd47cd0a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 --2- 172.21.15.69:0/2824776996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47cc6250 0x55bd47cd04c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc66ffd640 0 client.16086 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 >> 172.21.15.69:0/2824776996 conn(0x55bd47cc18c0 msgr2=0x55bd47cc0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 shutdown_connections 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.746+0000 7efc90835740 1 -- 172.21.15.69:0/2824776996 wait complete. 2024-05-09T09:20:36.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.747+0000 7efc90835740 20 client.16086 trim_cache size 0 max 16384 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 Processor -- start 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 -- start start 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cc1ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47ce3a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc881aee60 con 0x55bd47cc6c40 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc881ac6d0 con 0x55bd47ccbdb0 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.761+0000 7efc90835740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc881aa520 con 0x55bd47cc6250 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39284/0 (socket says 172.21.15.69:39284) 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 -- 172.21.15.69:0/315369803 learned_addr learned my addr 172.21.15.69:0/315369803 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc867fc640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cc1ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc867fc640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47ce3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 -- 172.21.15.69:0/315369803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 msgr2=0x55bd47cc1ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cc1ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 -- 172.21.15.69:0/315369803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 msgr2=0x55bd47ce3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47ce3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 -- 172.21.15.69:0/315369803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bd47cc7910 con 0x55bd47cc6c40 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc867fc640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47ce3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:36.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc86ffd640 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x55bd47cc76b0 tx=0x7efc78011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ca57376f1f52ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc7d7fa640 1 -- 172.21.15.69:0/315369803 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7801d700 con 0x55bd47cc6c40 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc7d7fa640 1 -- 172.21.15.69:0/315369803 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc78021030 con 0x55bd47cc6c40 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.762+0000 7efc7d7fa640 1 -- 172.21.15.69:0/315369803 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc78025890 con 0x55bd47cc6c40 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- 172.21.15.69:0/315369803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 msgr2=0x55bd47cd7960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x55bd47cc76b0 tx=0x7efc78011a30 comp rx=0 tx=0).stop 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- 172.21.15.69:0/315369803 shutdown_connections 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47ce3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd7960 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 --2- 172.21.15.69:0/315369803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cc1ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- 172.21.15.69:0/315369803 >> 172.21.15.69:0/315369803 conn(0x55bd47cc18c0 msgr2=0x55bd47cbc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- 172.21.15.69:0/315369803 shutdown_connections 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- 172.21.15.69:0/315369803 wait complete. 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.763+0000 7efc90835740 1 -- start start 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 Processor -- start 2024-05-09T09:20:36.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cd4f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efc881aab90 con 0x55bd47cc6c40 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efc881ac000 con 0x55bd47ccbdb0 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efc881ab260 con 0x55bd47cc6250 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc90835740 10 client.0 _ll_register_callbacks cb 0x7efc78008590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39294/0 (socket says 172.21.15.69:39294) 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 -- 172.21.15.69:0/1827065855 learned_addr learned my addr 172.21.15.69:0/1827065855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc867fc640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cd4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc867fc640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 -- 172.21.15.69:0/1827065855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 msgr2=0x55bd47cd4f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cd4f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 -- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 msgr2=0x55bd47cd6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:36.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 -- 172.21.15.69:0/1827065855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bd47cd6a50 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc867fc640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc86ffd640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efc7800f010 tx=0x7efc78004b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=262b110b122c38a1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc66ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.764+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efc7801d700 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7efc7801d8a0 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7efc78021030 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7efc78011020 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 10 client.16092 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bd47d422c0 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efc78021030 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7efc780256a0 con 0x55bd47cc6c40 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 10 client.16092 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:36.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 10 client.16092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 20 client.16092 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47d42be0 0x55bd47d45000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 4 client.16092 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55bd47c31dc0 con 0x55bd47d42be0 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.765+0000 7efc867fc640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47d42be0 0x55bd47d45000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.766+0000 7efc867fc640 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47d42be0 0x55bd47d45000 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=51f3d9ddf8faeed5 server_cookie=472d01a11ad92b25 in_seq=0 out_seq=0 2024-05-09T09:20:36.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.766+0000 7efc66ffd640 10 client.16092 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:36.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc66ffd640 1 -- 172.21.15.69:0/1827065855 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55bd47c31dc0 con 0x55bd47d42be0 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc66ffd640 10 client.16092 handle_command_reply: tid=0 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 client.16092 shutdown 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 msgr2=0x55bd47cd54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7efc7800f010 tx=0x7efc78004b30 comp rx=0 tx=0).stop 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 shutdown_connections 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bd47ccbdb0 0x55bd47cd6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bd47cc6c40 0x55bd47cd54a0 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bd47d42be0 0x55bd47d45000 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 --2- 172.21.15.69:0/1827065855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bd47cc6250 0x55bd47cd4f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc66ffd640 0 client.16092 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 >> 172.21.15.69:0/1827065855 conn(0x55bd47cc18c0 msgr2=0x55bd47cc1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 shutdown_connections 2024-05-09T09:20:36.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 1 -- 172.21.15.69:0/1827065855 wait complete. 2024-05-09T09:20:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.810+0000 7efc90835740 20 client.16092 trim_cache size 0 max 16384 2024-05-09T09:20:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:20:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:20:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:20:36.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 msgr2=0x7efc740a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 0x7efc740a50f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7efc800b69a0 tx=0x7efc800e69d0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 msgr2=0x7efc88074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7efc800314e0 tx=0x7efc800043c0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 shutdown_connections 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efc740a2c30 0x7efc740a50f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efc88103470 0x7efc88074160 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efc88102ab0 0x7efc88073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 --2- 172.21.15.69:0/3406331931 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efc881018c0 0x7efc880755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 >> 172.21.15.69:0/3406331931 conn(0x7efc880fce50 msgr2=0x7efc88105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 shutdown_connections 2024-05-09T09:20:36.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:36.811+0000 7efc8da53640 1 -- 172.21.15.69:0/3406331931 wait complete. 2024-05-09T09:20:37.471 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36230 6.07 MB/sec warmup 35 sec latency 2187.847 ms 2024-05-09T09:20:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:37 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:20:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:37 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:37 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:20:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:37 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:20:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:37 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:20:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:37 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:37 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:20:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:37 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:20:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:37 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] up:active 2024-05-09T09:20:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:37 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:20:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:37 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:20:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:37 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:20:38.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36230 5.90 MB/sec warmup 36 sec latency 3187.978 ms 2024-05-09T09:20:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:38 smithi069 ceph-mon[30123]: scrub summary: idle 2024-05-09T09:20:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:38 smithi069 ceph-mon[30123]: pgmap v418: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 2.5 MiB/s wr, 195 op/s 2024-05-09T09:20:38.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:38 smithi103 ceph-mon[34421]: scrub summary: idle 2024-05-09T09:20:38.519 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:38 smithi103 ceph-mon[34421]: pgmap v418: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 2.5 MiB/s wr, 195 op/s 2024-05-09T09:20:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:38 smithi163 ceph-mon[34354]: scrub summary: idle 2024-05-09T09:20:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:38 smithi163 ceph-mon[34354]: pgmap v418: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 537 KiB/s rd, 2.5 MiB/s wr, 195 op/s 2024-05-09T09:20:39.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36230 5.74 MB/sec warmup 37 sec latency 4188.079 ms 2024-05-09T09:20:40.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 36688 5.64 MB/sec warmup 38 sec latency 4944.269 ms 2024-05-09T09:20:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:40 smithi069 ceph-mon[30123]: pgmap v419: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-05-09T09:20:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:40 smithi163 ceph-mon[34354]: pgmap v419: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-05-09T09:20:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:40 smithi103 ceph-mon[34421]: pgmap v419: 129 pgs: 129 active+clean; 25 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 531 KiB/s rd, 1.0 MiB/s wr, 136 op/s 2024-05-09T09:20:41.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39422 5.93 MB/sec warmup 39 sec latency 10.856 ms 2024-05-09T09:20:42.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:41 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:20:41.538+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:42.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39722 5.79 MB/sec warmup 40 sec latency 688.558 ms 2024-05-09T09:20:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:42 smithi069 ceph-mon[30123]: pgmap v420: 129 pgs: 129 active+clean; 25 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 292 KiB/s wr, 85 op/s 2024-05-09T09:20:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:42 smithi069 ceph-mon[30123]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:42 smithi163 ceph-mon[34354]: pgmap v420: 129 pgs: 129 active+clean; 25 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 292 KiB/s wr, 85 op/s 2024-05-09T09:20:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:42 smithi163 ceph-mon[34354]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:42 smithi103 ceph-mon[34421]: pgmap v420: 129 pgs: 129 active+clean; 25 MiB data, 421 MiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 292 KiB/s wr, 85 op/s 2024-05-09T09:20:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:42 smithi103 ceph-mon[34421]: Health check update: 2 MDSs report damaged metadata (MDS_DAMAGE) 2024-05-09T09:20:43.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39722 5.64 MB/sec warmup 41 sec latency 1688.668 ms 2024-05-09T09:20:44.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39722 5.51 MB/sec warmup 42 sec latency 2688.782 ms 2024-05-09T09:20:44.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:44 smithi069 ceph-mon[30123]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 298 KiB/s wr, 84 op/s 2024-05-09T09:20:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:44 smithi163 ceph-mon[34354]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 298 KiB/s wr, 84 op/s 2024-05-09T09:20:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:44 smithi103 ceph-mon[34421]: pgmap v421: 129 pgs: 129 active+clean; 26 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 298 KiB/s wr, 84 op/s 2024-05-09T09:20:45.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39778 5.39 MB/sec warmup 43 sec latency 3443.327 ms 2024-05-09T09:20:46.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39778 5.26 MB/sec warmup 44 sec latency 1196.340 ms 2024-05-09T09:20:46.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:46 smithi069 ceph-mon[30123]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 270 KiB/s wr, 80 op/s 2024-05-09T09:20:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:46 smithi163 ceph-mon[34354]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 270 KiB/s wr, 80 op/s 2024-05-09T09:20:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:46 smithi103 ceph-mon[34421]: pgmap v422: 129 pgs: 129 active+clean; 27 MiB data, 427 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 270 KiB/s wr, 80 op/s 2024-05-09T09:20:47.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39778 5.15 MB/sec warmup 45 sec latency 2196.433 ms 2024-05-09T09:20:48.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39778 5.03 MB/sec warmup 46 sec latency 3196.547 ms 2024-05-09T09:20:48.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:48 smithi069 ceph-mon[30123]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 204 KiB/s wr, 63 op/s 2024-05-09T09:20:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:48 smithi163 ceph-mon[34354]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 204 KiB/s wr, 63 op/s 2024-05-09T09:20:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:48 smithi103 ceph-mon[34421]: pgmap v423: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 204 KiB/s wr, 63 op/s 2024-05-09T09:20:49.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39778 4.93 MB/sec warmup 47 sec latency 4196.619 ms 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.907+0000 7f6a4894f640 1 -- 172.21.15.69:0/3097330847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 msgr2=0x7f6a44107090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 --2- 172.21.15.69:0/3097330847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44107090 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c009900 tx=0x7f6a2c02a410 comp rx=0 tx=0).stop 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 -- 172.21.15.69:0/3097330847 shutdown_connections 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 --2- 172.21.15.69:0/3097330847 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a441075d0 0x7f6a441099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 --2- 172.21.15.69:0/3097330847 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44107090 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 --2- 172.21.15.69:0/3097330847 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a44102370 0x7f6a44104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 -- 172.21.15.69:0/3097330847 >> 172.21.15.69:0/3097330847 conn(0x7f6a440fbd00 msgr2=0x7f6a440fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 -- 172.21.15.69:0/3097330847 shutdown_connections 2024-05-09T09:20:49.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 -- 172.21.15.69:0/3097330847 wait complete. 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.908+0000 7f6a4894f640 1 Processor -- start 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 -- start start 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a44102370 0x7f6a44197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a441075d0 0x7f6a4419f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d510 con 0x7f6a44104ca0 2024-05-09T09:20:49.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d390 con 0x7f6a441075d0 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4894f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a4410d690 con 0x7f6a44102370 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49852/0 (socket says 172.21.15.69:49852) 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 -- 172.21.15.69:0/650285777 learned_addr learned my addr 172.21.15.69:0/650285777 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a441075d0 0x7f6a4419f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a4259c640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a44102370 0x7f6a44197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 -- 172.21.15.69:0/650285777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a44102370 msgr2=0x7f6a44197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a44102370 0x7f6a44197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 -- 172.21.15.69:0/650285777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a441075d0 msgr2=0x7f6a4419f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a441075d0 0x7f6a4419f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 -- 172.21.15.69:0/650285777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6a4419f730 con 0x7f6a44104ca0 2024-05-09T09:20:49.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c031610 tx=0x7f6a2c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.909+0000 7f6a3b7fe640 1 -- 172.21.15.69:0/650285777 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c03f030 con 0x7f6a44104ca0 2024-05-09T09:20:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.910+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6a4419f9c0 con 0x7f6a44104ca0 2024-05-09T09:20:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.910+0000 7f6a3b7fe640 1 -- 172.21.15.69:0/650285777 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c003ce0 con 0x7f6a44104ca0 2024-05-09T09:20:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.910+0000 7f6a3b7fe640 1 -- 172.21.15.69:0/650285777 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c035070 con 0x7f6a44104ca0 2024-05-09T09:20:49.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.910+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6a4419feb0 con 0x7f6a44104ca0 2024-05-09T09:20:49.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.912+0000 7f6a3b7fe640 1 -- 172.21.15.69:0/650285777 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6a2c042030 con 0x7f6a44104ca0 2024-05-09T09:20:49.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.912+0000 7f6a4b65b740 1 Processor -- start 2024-05-09T09:20:49.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 -- start start 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872f050 0x563d78733b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d787341c0 0x563d78736650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563d7868e340 con 0x563d7872e660 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d78636290 con 0x563d7872f050 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563d7868a300 con 0x563d787341c0 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a3b7fe640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 0x7f6a300a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49864/0 (socket says 172.21.15.69:49864) 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 -- 172.21.15.69:0/3507989246 learned_addr learned my addr 172.21.15.69:0/3507989246 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a4259c640 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872f050 0x563d78733b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a3b7fe640 1 -- 172.21.15.69:0/650285777 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6a2c0ef8b0 con 0x7f6a44104ca0 2024-05-09T09:20:49.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d787341c0 0x563d78736650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 0x7f6a300a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 -- 172.21.15.69:0/3507989246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d787341c0 msgr2=0x563d78736650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d787341c0 0x563d78736650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 -- 172.21.15.69:0/3507989246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872f050 msgr2=0x563d78733b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872f050 0x563d78733b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.913+0000 7f6a42d9d640 1 -- 172.21.15.69:0/3507989246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d7865a8e0 con 0x563d7872e660 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.914+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00bda0 tx=0x7f6a3c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8370ebcf47516934 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.914+0000 7f6a38ff9640 1 -- 172.21.15.69:0/3507989246 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c00b7a0 con 0x563d7872e660 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.914+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 0x7f6a300a50f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c021000 tx=0x7f6a3c020000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.914+0000 7f6a38ff9640 1 -- 172.21.15.69:0/3507989246 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3c005570 con 0x563d7872e660 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.914+0000 7f6a38ff9640 1 -- 172.21.15.69:0/3507989246 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c01d3d0 con 0x563d7872e660 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3507989246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 msgr2=0x563d7872ea80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c00bda0 tx=0x7f6a3c0039c0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3507989246 shutdown_connections 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d787341c0 0x563d78736650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872f050 0x563d78733b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3507989246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872e660 0x563d7872ea80 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3507989246 >> 172.21.15.69:0/3507989246 conn(0x563d787258c0 msgr2=0x563d78725cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3507989246 shutdown_connections 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3507989246 wait complete. 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- start start 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 Processor -- start 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78738870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d7873fdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.915+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563d78737080 con 0x563d7872f050 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563d78649790 con 0x563d787341c0 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563d78737200 con 0x563d7872e660 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 10 client.0 _ll_register_callbacks cb 0x563d78684a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49870/0 (socket says 172.21.15.69:49870) 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 -- 172.21.15.69:0/4229106197 learned_addr learned my addr 172.21.15.69:0/4229106197 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78738870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d7873fdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 -- 172.21.15.69:0/4229106197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 msgr2=0x563d78738870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78738870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 -- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 msgr2=0x563d7873fdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d7873fdf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 -- 172.21.15.69:0/4229106197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d783a2210 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0314e0 tx=0x7f6a2c0b8210 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89b9d14c65d2fe6e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c049040 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a3c005570 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6a2c003800 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a2c0c0140 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 10 client.16104 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.916+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d7846e620 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a3c005570 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6a2c0bc0a0 con 0x563d7872f050 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 10 client.16104 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 10 client.16104 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 20 client.16104 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d78744460 0x563d787468a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 4 client.16104 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563d7869ba70 con 0x563d78744460 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d78744460 0x563d787468a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a29ffb640 10 client.16104 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.917+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d78744460 0x563d787468a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6c0cc431c44b0ba server_cookie=138ccada3280c85e in_seq=0 out_seq=0 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4229106197 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563d7869ba70 con 0x563d78744460 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a29ffb640 10 client.16104 handle_command_reply: tid=0 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 client.16104 shutdown 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 msgr2=0x563d78738db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c0314e0 tx=0x7f6a2c0b8210 comp rx=0 tx=0).stop 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 shutdown_connections 2024-05-09T09:20:49.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d7873fdf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d78738db0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d78744460 0x563d787468a0 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4229106197 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78738870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a29ffb640 0 client.16104 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 >> 172.21.15.69:0/4229106197 conn(0x563d787258c0 msgr2=0x563d78724a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 shutdown_connections 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.918+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4229106197 wait complete. 2024-05-09T09:20:49.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.919+0000 7f6a4b65b740 20 client.16104 trim_cache size 0 max 16384 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 Processor -- start 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 -- start start 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78728910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d78728e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441aaa80 con 0x563d7872f050 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a82f0 con 0x563d787341c0 2024-05-09T09:20:49.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a6140 con 0x563d7872e660 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a41d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49876/0 (socket says 172.21.15.69:49876) 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.933+0000 7f6a41d9b640 1 -- 172.21.15.69:0/3251145933 learned_addr learned my addr 172.21.15.69:0/3251145933 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d78728e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 -- 172.21.15.69:0/3251145933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 msgr2=0x563d78728910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78728910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 -- 172.21.15.69:0/3251145933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 msgr2=0x563d78728e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d78728e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 -- 172.21.15.69:0/3251145933 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563d78746f20 con 0x563d7872f050 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000c00 tx=0x7f6a2c005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b71ac621a14a6f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a2bfff640 1 -- 172.21.15.69:0/3251145933 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c049070 con 0x563d7872f050 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a2bfff640 1 -- 172.21.15.69:0/3251145933 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c0c4a00 con 0x563d7872f050 2024-05-09T09:20:49.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a2bfff640 1 -- 172.21.15.69:0/3251145933 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a2c0b84b0 con 0x563d7872f050 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3251145933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 msgr2=0x563d7873fdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c000c00 tx=0x7f6a2c005c00 comp rx=0 tx=0).stop 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3251145933 shutdown_connections 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d787341c0 0x563d78728e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d7872f050 0x563d7873fdb0 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/3251145933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872e660 0x563d78728910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3251145933 >> 172.21.15.69:0/3251145933 conn(0x563d787258c0 msgr2=0x563d78730230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.934+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3251145933 shutdown_connections 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 -- 172.21.15.69:0/3251145933 wait complete. 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 -- start start 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 Processor -- start 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872e660 0x563d7873d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872f050 0x563d7873dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.939 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a67b0 con 0x563d787341c0 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a7c20 con 0x563d7872e660 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6a441a6e80 con 0x563d7872f050 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4b65b740 10 client.0 _ll_register_callbacks cb 0x7f6a2c0095a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49886/0 (socket says 172.21.15.69:49886) 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 -- 172.21.15.69:0/4050329205 learned_addr learned my addr 172.21.15.69:0/4050329205 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872e660 0x563d7873d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 -- 172.21.15.69:0/4050329205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872f050 msgr2=0x563d7873dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a41d9b640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872f050 0x563d7873dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872f050 0x563d7873dcf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.935+0000 7f6a42d9d640 1 -- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872e660 msgr2=0x563d7873d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872e660 0x563d7873d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a42d9d640 1 -- 172.21.15.69:0/4050329205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563d7873f2f0 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a42d9d640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x563d7872f8b0 tx=0x7f6a3c0046f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8af274ed3bed6d93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6a3c037060 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f6a3c004b70 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6a2c0c4a00 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f6a3c03e050 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 10 client.16110 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x563d787aae10 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6a2c0c4a00 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f6a3c031070 con 0x563d787341c0 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 10 client.16110 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 10 client.16110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 20 client.16110 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:49.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d787ab6c0 0x563d787adae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 4 client.16110 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:20:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.936+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563d78695dc0 con 0x563d787ab6c0 2024-05-09T09:20:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.937+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d787ab6c0 0x563d787adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.937+0000 7f6a29ffb640 10 client.16110 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:49.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.937+0000 7f6a4259c640 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d787ab6c0 0x563d787adae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89c9b9ac0e550899 server_cookie=5bf5bd74d4fd07bf in_seq=0 out_seq=0 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.967+0000 7f6a29ffb640 1 -- 172.21.15.69:0/4050329205 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563d78695dc0 con 0x563d787ab6c0 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.967+0000 7f6a29ffb640 10 client.16110 handle_command_reply: tid=0 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.967+0000 7f6a4b65b740 1 client.16110 shutdown 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 msgr2=0x563d7873ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x563d7872f8b0 tx=0x7f6a3c0046f0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 shutdown_connections 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563d787341c0 0x563d7873ed00 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563d787ab6c0 0x563d787adae0 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563d7872f050 0x563d7873dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 --2- 172.21.15.69:0/4050329205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563d7872e660 0x563d7873d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 >> 172.21.15.69:0/4050329205 conn(0x563d787258c0 msgr2=0x563d78728910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a29ffb640 0 client.16110 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 shutdown_connections 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 1 -- 172.21.15.69:0/4050329205 wait complete. 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4b65b740 20 client.16110 trim_cache size 0 max 16384 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:20:49.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:20:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 msgr2=0x7f6a300a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 0x7f6a300a50f0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f6a3c021000 tx=0x7f6a3c020000 comp rx=0 tx=0).stop 2024-05-09T09:20:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 msgr2=0x7f6a44198050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f6a2c031610 tx=0x7f6a2c0043b0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 shutdown_connections 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6a300a2c30 0x7f6a300a50f0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6a441075d0 0x7f6a4419f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6a44104ca0 0x7f6a44198050 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 --2- 172.21.15.69:0/650285777 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6a44102370 0x7f6a44197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 >> 172.21.15.69:0/650285777 conn(0x7f6a440fbd00 msgr2=0x7f6a44102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.968+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 shutdown_connections 2024-05-09T09:20:49.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:20:49.969+0000 7f6a4894f640 1 -- 172.21.15.69:0/650285777 wait complete. 2024-05-09T09:20:50.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 39846 4.83 MB/sec warmup 48 sec latency 4951.506 ms 2024-05-09T09:20:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:50 smithi069 ceph-mon[30123]: pgmap v424: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 204 KiB/s wr, 61 op/s 2024-05-09T09:20:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:50 smithi163 ceph-mon[34354]: pgmap v424: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 204 KiB/s wr, 61 op/s 2024-05-09T09:20:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:50 smithi103 ceph-mon[34421]: pgmap v424: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 204 KiB/s wr, 61 op/s 2024-05-09T09:20:51.472 INFO:tasks.workunit.client.0.smithi069.stdout: 1 40045 4.74 MB/sec warmup 49 sec latency 1057.554 ms 2024-05-09T09:20:52.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 42847 5.00 MB/sec warmup 50 sec latency 14.857 ms 2024-05-09T09:20:52.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:52 smithi069 ceph-mon[30123]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 196 KiB/s wr, 57 op/s 2024-05-09T09:20:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:52 smithi163 ceph-mon[34354]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 196 KiB/s wr, 57 op/s 2024-05-09T09:20:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:52 smithi103 ceph-mon[34421]: pgmap v425: 129 pgs: 129 active+clean; 27 MiB data, 411 MiB used, 1.0 TiB / 1.0 TiB avail; 420 KiB/s rd, 196 KiB/s wr, 57 op/s 2024-05-09T09:20:53.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43264 4.91 MB/sec warmup 51 sec latency 514.068 ms 2024-05-09T09:20:54.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43264 4.82 MB/sec warmup 52 sec latency 1514.124 ms 2024-05-09T09:20:54.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:54 smithi069 ceph-mon[30123]: pgmap v426: 129 pgs: 129 active+clean; 29 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 645 KiB/s wr, 72 op/s 2024-05-09T09:20:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:54 smithi163 ceph-mon[34354]: pgmap v426: 129 pgs: 129 active+clean; 29 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 645 KiB/s wr, 72 op/s 2024-05-09T09:20:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:54 smithi103 ceph-mon[34421]: pgmap v426: 129 pgs: 129 active+clean; 29 MiB data, 420 MiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 645 KiB/s wr, 72 op/s 2024-05-09T09:20:55.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43388 4.73 MB/sec warmup 53 sec latency 2268.415 ms 2024-05-09T09:20:56.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43388 4.65 MB/sec warmup 54 sec latency 1130.283 ms 2024-05-09T09:20:56.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:56 smithi069 ceph-mon[30123]: pgmap v427: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 953 KiB/s wr, 90 op/s 2024-05-09T09:20:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:56 smithi163 ceph-mon[34354]: pgmap v427: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 953 KiB/s wr, 90 op/s 2024-05-09T09:20:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:56 smithi103 ceph-mon[34421]: pgmap v427: 129 pgs: 129 active+clean; 30 MiB data, 444 MiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 953 KiB/s wr, 90 op/s 2024-05-09T09:20:57.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43388 4.56 MB/sec warmup 55 sec latency 2130.399 ms 2024-05-09T09:20:58.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43388 4.48 MB/sec warmup 56 sec latency 3130.515 ms 2024-05-09T09:20:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:20:58 smithi163 ceph-mon[34354]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1012 KiB/s wr, 88 op/s 2024-05-09T09:20:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:20:58 smithi103 ceph-mon[34421]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1012 KiB/s wr, 88 op/s 2024-05-09T09:20:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:20:58 smithi069 ceph-mon[30123]: pgmap v428: 129 pgs: 129 active+clean; 27 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 423 KiB/s rd, 1012 KiB/s wr, 88 op/s 2024-05-09T09:20:58.923 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-05-09T09:20:58.923 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 -- 172.21.15.69:0/3848167134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b41021f0 msgr2=0x7f20b4108a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 --2- 172.21.15.69:0/3848167134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b41021f0 0x7f20b4108a80 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f20ac009ca0 tx=0x7f20ac02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 -- 172.21.15.69:0/3848167134 shutdown_connections 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 --2- 172.21.15.69:0/3848167134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b41021f0 0x7f20b4108a80 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 --2- 172.21.15.69:0/3848167134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b4101830 0x7f20b4101cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 --2- 172.21.15.69:0/3848167134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4100640 0x7f20b4100a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 -- 172.21.15.69:0/3848167134 >> 172.21.15.69:0/3848167134 conn(0x7f20b40fbbd0 msgr2=0x7f20b40fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 -- 172.21.15.69:0/3848167134 shutdown_connections 2024-05-09T09:20:59.042 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.041+0000 7f20ba79e640 1 -- 172.21.15.69:0/3848167134 wait complete. 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 Processor -- start 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 -- start start 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4101830 0x7f20b4102c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b41021f0 0x7f20b4103150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20b410c2a0 con 0x7f20b4100640 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20b410c120 con 0x7f20b4101830 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20ba79e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20b410c420 con 0x7f20b41021f0 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41238/0 (socket says 172.21.15.69:41238) 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 -- 172.21.15.69:0/1744460611 learned_addr learned my addr 172.21.15.69:0/1744460611 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4101830 0x7f20b4102c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b41021f0 0x7f20b4103150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 -- 172.21.15.69:0/1744460611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b41021f0 msgr2=0x7f20b4103150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.043 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b41021f0 0x7f20b4103150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 -- 172.21.15.69:0/1744460611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4101830 msgr2=0x7f20b4102c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4101830 0x7f20b4102c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 -- 172.21.15.69:0/1744460611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f20b41039e0 con 0x7f20b4100640 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.042+0000 7f20b979c640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f20b000d950 tx=0x7f20b000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.043+0000 7f20aa7fc640 1 -- 172.21.15.69:0/1744460611 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b0017020 con 0x7f20b4100640 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.043+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f20b41a1aa0 con 0x7f20b4100640 2024-05-09T09:20:59.044 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.043+0000 7f20aa7fc640 1 -- 172.21.15.69:0/1744460611 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20b0009c30 con 0x7f20b4100640 2024-05-09T09:20:59.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.043+0000 7f20aa7fc640 1 -- 172.21.15.69:0/1744460611 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b000f740 con 0x7f20b4100640 2024-05-09T09:20:59.045 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.043+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f20b41a1ff0 con 0x7f20b4100640 2024-05-09T09:20:59.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.045+0000 7f20aa7fc640 1 -- 172.21.15.69:0/1744460611 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f20b001a020 con 0x7f20b4100640 2024-05-09T09:20:59.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.045+0000 7f20be7ef740 1 Processor -- start 2024-05-09T09:20:59.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.045+0000 7f20be7ef740 1 -- start start 2024-05-09T09:20:59.051 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.045+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5a65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5a90d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20be7ef740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcba505d00 con 0x55bcba5a10e0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20be7ef740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcba4ad4c0 con 0x55bcba5a6c40 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20be7ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bcba501cc0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20aa7fc640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 0x7f20a00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41248/0 (socket says 172.21.15.69:41248) 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 -- 172.21.15.69:0/347437609 learned_addr learned my addr 172.21.15.69:0/347437609 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b979c640 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5a65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 0x7f20a00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5a90d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20aa7fc640 1 -- 172.21.15.69:0/1744460611 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f20b00cef00 con 0x7f20b4100640 2024-05-09T09:20:59.052 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 -- 172.21.15.69:0/347437609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 msgr2=0x55bcba5a65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5a65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 -- 172.21.15.69:0/347437609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 msgr2=0x55bcba5a90d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5a90d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 -- 172.21.15.69:0/347437609 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcba2e37d0 con 0x55bcba5a10e0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0316e0 tx=0x7f20ac031710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c989cc9bc58bd51 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 0x7f20a00a5050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20ac02f540 tx=0x7f20ac041000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.046+0000 7f209f7fe640 1 -- 172.21.15.69:0/347437609 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac004420 con 0x55bcba5a10e0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f209f7fe640 1 -- 172.21.15.69:0/347437609 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ac036040 con 0x55bcba5a10e0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f209f7fe640 1 -- 172.21.15.69:0/347437609 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20ac02fd50 con 0x55bcba5a10e0 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 -- 172.21.15.69:0/347437609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 msgr2=0x55bcba5a1500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f20ac0316e0 tx=0x7f20ac031710 comp rx=0 tx=0).stop 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 -- 172.21.15.69:0/347437609 shutdown_connections 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5a90d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5a65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.053 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 --2- 172.21.15.69:0/347437609 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5a1500 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 -- 172.21.15.69:0/347437609 >> 172.21.15.69:0/347437609 conn(0x55bcba59c750 msgr2=0x55bcba59cb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 -- 172.21.15.69:0/347437609 shutdown_connections 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.047+0000 7f20be7ef740 1 -- 172.21.15.69:0/347437609 wait complete. 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 -- start start 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 Processor -- start 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5ab530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5b2ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55bcba5a9b90 con 0x55bcba5a1ad0 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55bcba4c0b90 con 0x55bcba5a10e0 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55bcba5a9d10 con 0x55bcba5a6c40 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20be7ef740 10 client.0 _ll_register_callbacks cb 0x55bcba4fcda0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b8f9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.054 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b8f9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41252/0 (socket says 172.21.15.69:41252) 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b8f9b640 1 -- 172.21.15.69:0/3849035985 learned_addr learned my addr 172.21.15.69:0/3849035985 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5b2ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b979c640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5ab530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b8f9b640 1 -- 172.21.15.69:0/3849035985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 msgr2=0x55bcba5b2ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.048+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5b2ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20b8f9b640 1 -- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 msgr2=0x55bcba5ab530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5ab530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20b8f9b640 1 -- 172.21.15.69:0/3849035985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcba2193f0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f20a400cd60 tx=0x7f20a40039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c559172e22a09837 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4010900 con 0x55bcba5a1ad0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f20a4010aa0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.055 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20ac036040 con 0x55bcba5a1ad0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f20a4023050 con 0x55bcba5a1ad0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 10 client.16122 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bcba1d90f0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20ac036040 con 0x55bcba5a1ad0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f20a4018020 con 0x55bcba5a1ad0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 10 client.16122 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 10 client.16122 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 20 client.16122 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba5b9940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 4 client.16122 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.049+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55bcba513430 con 0x55bcba5b74e0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.050+0000 7f20b979c640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba5b9940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.050+0000 7f20b979c640 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba5b9940 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6dc56e5642899fb3 server_cookie=a426d176d06ae552 in_seq=0 out_seq=0 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.050+0000 7f209cff9640 10 client.16122 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.056 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f209cff9640 1 -- 172.21.15.69:0/3849035985 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55bcba513430 con 0x55bcba5b74e0 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f209cff9640 10 client.16122 handle_command_reply: tid=0 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 client.16122 shutdown 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 msgr2=0x55bcba5aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7f20a400cd60 tx=0x7f20a40039c0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 shutdown_connections 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba5b9940 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5b2ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f209cff9640 0 client.16122 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5aba70 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 --2- 172.21.15.69:0/3849035985 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5ab530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 >> 172.21.15.69:0/3849035985 conn(0x55bcba59c750 msgr2=0x55bcba59ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 shutdown_connections 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 1 -- 172.21.15.69:0/3849035985 wait complete. 2024-05-09T09:20:59.057 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.051+0000 7f20be7ef740 20 client.16122 trim_cache size 0 max 16384 2024-05-09T09:20:59.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 Processor -- start 2024-05-09T09:20:59.080 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 -- start start 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5a2f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5a3480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41aeca0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41ac510 con 0x55bcba5a10e0 2024-05-09T09:20:59.081 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20be7ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41aa360 con 0x55bcba5a6c40 2024-05-09T09:20:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41254/0 (socket says 172.21.15.69:41254) 2024-05-09T09:20:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 -- 172.21.15.69:0/589577768 learned_addr learned my addr 172.21.15.69:0/589577768 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5a3480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 -- 172.21.15.69:0/589577768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 msgr2=0x55bcba5a3480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5a3480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 -- 172.21.15.69:0/589577768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 msgr2=0x55bcba5a2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5a2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 -- 172.21.15.69:0/589577768 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55bcba59c580 con 0x55bcba5a1ad0 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.069+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f20a4005b90 tx=0x7f20a400c9a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15c39c67e394c60a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f209effd640 1 -- 172.21.15.69:0/589577768 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a402b020 con 0x55bcba5a1ad0 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f209effd640 1 -- 172.21.15.69:0/589577768 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a400e630 con 0x55bcba5a1ad0 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f209effd640 1 -- 172.21.15.69:0/589577768 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20a4010700 con 0x55bcba5a1ad0 2024-05-09T09:20:59.083 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- 172.21.15.69:0/589577768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 msgr2=0x55bcba5b2a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7f20a4005b90 tx=0x7f20a400c9a0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- 172.21.15.69:0/589577768 shutdown_connections 2024-05-09T09:20:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a6c40 0x55bcba5a3480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b2a70 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 --2- 172.21.15.69:0/589577768 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a10e0 0x55bcba5a2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- 172.21.15.69:0/589577768 >> 172.21.15.69:0/589577768 conn(0x55bcba59c750 msgr2=0x55bcba59cb30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- 172.21.15.69:0/589577768 shutdown_connections 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- 172.21.15.69:0/589577768 wait complete. 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.070+0000 7f20be7ef740 1 -- start start 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 Processor -- start 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b0b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5b1b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41aa9d0 con 0x55bcba5a10e0 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41abe40 con 0x55bcba5a6c40 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f20b41ab0a0 con 0x55bcba5a1ad0 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20be7ef740 10 client.0 _ll_register_callbacks cb 0x7f20a4004080 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41264/0 (socket says 172.21.15.69:41264) 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 -- 172.21.15.69:0/2178812691 learned_addr learned my addr 172.21.15.69:0/2178812691 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5b1b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 -- 172.21.15.69:0/2178812691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 msgr2=0x55bcba5b0b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b0b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 -- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 msgr2=0x55bcba5b1b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5b1b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 -- 172.21.15.69:0/2178812691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55bcba5b2110 con 0x55bcba5a10e0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b9f9d640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f20b979c640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f20b000bfa0 tx=0x7f20b001d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95742a37a530ff63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.071+0000 7f209cff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f20b00b4060 con 0x55bcba5a10e0 2024-05-09T09:20:59.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f20b0010a40 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f20a400e630 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f20b009b3e0 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 10 client.16128 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55bcba61e000 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f20a400e630 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f20b00b4200 con 0x55bcba5a10e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 10 client.16128 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 10 client.16128 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 20 client.16128 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba620aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 4 client.16128 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "scrub status"}) -- 0x55bcba50d780 con 0x55bcba5b74e0 2024-05-09T09:20:59.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.072+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba620aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20b8f9b640 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba620aa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f0b3aecd86809a8 server_cookie=e44009163f2c9e9f in_seq=0 out_seq=0 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f209cff9640 10 client.16128 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f209cff9640 1 -- 172.21.15.69:0/2178812691 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+63 (crc 0 0 0) 0x55bcba50d780 con 0x55bcba5b74e0 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f209cff9640 10 client.16128 handle_command_reply: tid=0 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 client.16128 shutdown 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 msgr2=0x55bcba5b05d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7f20b000bfa0 tx=0x7f20b001d000 comp rx=0 tx=0).stop 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 shutdown_connections 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55bcba5a6c40 0x55bcba5b1b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55bcba5b74e0 0x55bcba620aa0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55bcba5a1ad0 0x55bcba5b0b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 --2- 172.21.15.69:0/2178812691 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55bcba5a10e0 0x55bcba5b05d0 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f209cff9640 0 client.16128 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 >> 172.21.15.69:0/2178812691 conn(0x55bcba59c750 msgr2=0x55bcba5ab400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 shutdown_connections 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 1 -- 172.21.15.69:0/2178812691 wait complete. 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.073+0000 7f20be7ef740 20 client.16128 trim_cache size 0 max 16384 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stdout: "status": "no active scrubs running", 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stdout: "scrubs": {} 2024-05-09T09:20:59.091 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 msgr2=0x7f20a00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 0x7f20a00a5050 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f20ac02f540 tx=0x7f20ac041000 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 msgr2=0x7f20b41045d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f20b000d950 tx=0x7f20b000de10 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 shutdown_connections 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f20a00a2b90 0x7f20a00a5050 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f20b41021f0 0x7f20b4103150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f20b4101830 0x7f20b4102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 --2- 172.21.15.69:0/1744460611 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f20b4100640 0x7f20b41045d0 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 >> 172.21.15.69:0/1744460611 conn(0x7f20b40fbbd0 msgr2=0x7f20b40689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 shutdown_connections 2024-05-09T09:20:59.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.074+0000 7f20ba79e640 1 -- 172.21.15.69:0/1744460611 wait complete. 2024-05-09T09:20:59.094 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-05-09T09:20:59.094 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:20:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 -- 172.21.15.69:0/3141686389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 msgr2=0x7f4518101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 --2- 172.21.15.69:0/3141686389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f4518101e00 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7f450c009900 tx=0x7f450c02a410 comp rx=0 tx=0).stop 2024-05-09T09:20:59.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 -- 172.21.15.69:0/3141686389 shutdown_connections 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 --2- 172.21.15.69:0/3141686389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518102340 0x7f4518108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 --2- 172.21.15.69:0/3141686389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f4518101e00 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 --2- 172.21.15.69:0/3141686389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4518100790 0x7f4518100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 -- 172.21.15.69:0/3141686389 >> 172.21.15.69:0/3141686389 conn(0x7f45180fbd20 msgr2=0x7f45180fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 -- 172.21.15.69:0/3141686389 shutdown_connections 2024-05-09T09:20:59.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.196+0000 7f451c963640 1 -- 172.21.15.69:0/3141686389 wait complete. 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 Processor -- start 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 -- start start 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518100790 0x7f4518197c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4518102340 0x7f451819f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f451810d590 con 0x7f4518101980 2024-05-09T09:20:59.198 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f451810d410 con 0x7f4518100790 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451c963640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f451810d710 con 0x7f4518102340 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41270/0 (socket says 172.21.15.69:41270) 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 -- 172.21.15.69:0/3998648188 learned_addr learned my addr 172.21.15.69:0/3998648188 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f451659c640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518100790 0x7f4518197c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 -- 172.21.15.69:0/3998648188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4518102340 msgr2=0x7f451819f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4518102340 0x7f451819f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 -- 172.21.15.69:0/3998648188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518100790 msgr2=0x7f4518197c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518100790 0x7f4518197c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 -- 172.21.15.69:0/3998648188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f451819f870 con 0x7f4518101980 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.197+0000 7f4515d9b640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f450c009230 tx=0x7f450c004360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.198+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c03f030 con 0x7f4518101980 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.198+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f450c003ce0 con 0x7f4518101980 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.198+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f451819fb00 con 0x7f4518101980 2024-05-09T09:20:59.199 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.198+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f450c035070 con 0x7f4518101980 2024-05-09T09:20:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.198+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f45181a00c0 con 0x7f4518101980 2024-05-09T09:20:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.199+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f4518107b30 con 0x7f4518101980 2024-05-09T09:20:59.200 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.200+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f450c042030 con 0x7f4518101980 2024-05-09T09:20:59.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.200+0000 7f45077fe640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 0x7f45000a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.201+0000 7f451659c640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 0x7f45000a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.201+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f450c0efe70 con 0x7f4518101980 2024-05-09T09:20:59.201 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.201+0000 7f451659c640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 0x7f45000a50d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4518199130 tx=0x7f4508008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.203 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.204+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f450c0bca50 con 0x7f4518101980 2024-05-09T09:20:59.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.417+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f451802d1c0 con 0x7f4518101980 2024-05-09T09:20:59.416 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.418+0000 7f45077fe640 1 -- 172.21.15.69:0/3998648188 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) ==== 94+0+11882 (secure 0 0 0) 0x7f450c0bc350 con 0x7f4518101980 2024-05-09T09:20:59.417 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:20:59.417 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":30,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":30,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:20:35.223932+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":66,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":66,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[1,2],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25375":{"gid":25375,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3564740613","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3564740613},{"type":"v1","addr":"172.21.15.163:6837","nonce":3564740613}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24491,24511,24527]},"id":1}]} 2024-05-09T09:20:59.417 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 30 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 msgr2=0x7f45000a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 0x7f45000a50d0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f4518199130 tx=0x7f4508008040 comp rx=0 tx=0).stop 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 msgr2=0x7f45181981a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7f450c009230 tx=0x7f450c004360 comp rx=0 tx=0).stop 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 shutdown_connections 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f45000a2c10 0x7f45000a50d0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4518102340 0x7f451819f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4518101980 0x7f45181981a0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.419 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 --2- 172.21.15.69:0/3998648188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4518100790 0x7f4518197c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 >> 172.21.15.69:0/3998648188 conn(0x7f45180fbd20 msgr2=0x7f4518104d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 shutdown_connections 2024-05-09T09:20:59.420 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.420+0000 7f451c963640 1 -- 172.21.15.69:0/3998648188 wait complete. 2024-05-09T09:20:59.430 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-05-09T09:20:59.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43388 4.40 MB/sec warmup 57 sec latency 4130.561 ms 2024-05-09T09:20:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 -- 172.21.15.69:0/1775255215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28101880 msgr2=0x7f7a28101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1775255215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28101880 0x7f7a28101ca0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7f7a18009ca0 tx=0x7f7a1802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.536 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 -- 172.21.15.69:0/1775255215 shutdown_connections 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1775255215 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28103430 0x7f7a28109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1775255215 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1775255215 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28101880 0x7f7a28101ca0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 -- 172.21.15.69:0/1775255215 >> 172.21.15.69:0/1775255215 conn(0x7f7a280fce10 msgr2=0x7f7a280ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.534+0000 7f7a3014f640 1 -- 172.21.15.69:0/1775255215 shutdown_connections 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 -- 172.21.15.69:0/1775255215 wait complete. 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 Processor -- start 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 -- start start 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28101880 0x7f7a280755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.537 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810d5d0 con 0x7f7a28103430 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810d450 con 0x7f7a28102a70 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a3014f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a2810d750 con 0x7f7a28101880 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a2e6ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a2e6ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41282/0 (socket says 172.21.15.69:41282) 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.535+0000 7f7a2e6ec640 1 -- 172.21.15.69:0/1901851891 learned_addr learned my addr 172.21.15.69:0/1901851891 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.538 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 -- 172.21.15.69:0/1901851891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28101880 msgr2=0x7f7a280755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28101880 0x7f7a280755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28101880 0x7f7a280755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 -- 172.21.15.69:0/1901851891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 msgr2=0x7f7a28073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 -- 172.21.15.69:0/1901851891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7a280749f0 con 0x7f7a28103430 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a2e6ec640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009fd0 tx=0x7f7a2000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a26ffd640 1 -- 172.21.15.69:0/1901851891 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a20011830 con 0x7f7a28103430 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a26ffd640 1 -- 172.21.15.69:0/1901851891 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a20016070 con 0x7f7a28103430 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7a281a1e50 con 0x7f7a28103430 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a26ffd640 1 -- 172.21.15.69:0/1901851891 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a2000f2e0 con 0x7f7a28103430 2024-05-09T09:20:59.539 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.536+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7a281a2330 con 0x7f7a28103430 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a26ffd640 1 -- 172.21.15.69:0/1901851891 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7a200119d0 con 0x7f7a28103430 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 Processor -- start 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 -- start start 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd377430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.547 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd377aa0 0x55b9fd379f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd2d6c40 con 0x55b9fd371f40 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd27e580 con 0x55b9fd372930 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.539+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd2d2c00 con 0x55b9fd377aa0 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41290/0 (socket says 172.21.15.69:41290) 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1839648612 learned_addr learned my addr 172.21.15.69:0/1839648612 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a26ffd640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 0x7f7a100a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd377430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd377aa0 0x55b9fd379f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1839648612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd377aa0 msgr2=0x55b9fd379f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd377aa0 0x55b9fd379f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.548 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1839648612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 msgr2=0x55b9fd377430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2e6ec640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 0x7f7a100a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd377430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1839648612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9fcfea3f0 con 0x55b9fd371f40 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00af40 tx=0x7f7a1c00a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13b3d8cab9ece18b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a26ffd640 1 -- 172.21.15.69:0/1901851891 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f7a200cea40 con 0x7f7a28103430 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a0ffff640 1 -- 172.21.15.69:0/1839648612 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c01b070 con 0x55b9fd371f40 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a0ffff640 1 -- 172.21.15.69:0/1839648612 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c0168d0 con 0x55b9fd371f40 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.540+0000 7f7a0ffff640 1 -- 172.21.15.69:0/1839648612 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c00f040 con 0x55b9fd371f40 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1839648612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 msgr2=0x55b9fd372360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a2e6ec640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 0x7f7a100a5050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7a20011fd0 tx=0x7f7a20065f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.549 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00af40 tx=0x7f7a1c00a410 comp rx=0 tx=0).stop 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1839648612 shutdown_connections 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd377aa0 0x55b9fd379f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd377430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1839648612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd372360 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1839648612 >> 172.21.15.69:0/1839648612 conn(0x55b9fd36d5b0 msgr2=0x55b9fd36d9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1839648612 shutdown_connections 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1839648612 wait complete. 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 -- start start 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.541+0000 7f7a32e5a740 1 Processor -- start 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd371f40 0x55b9fd37c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd37c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.550 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd37a9f0 con 0x55b9fd377aa0 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd291c40 con 0x55b9fd372930 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b9fd37ab70 con 0x55b9fd371f40 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a32e5a740 10 client.0 _ll_register_callbacks cb 0x55b9fcfe9be0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41298/0 (socket says 172.21.15.69:41298) 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/204012929 learned_addr learned my addr 172.21.15.69:0/204012929 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd371f40 0x55b9fd37c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd37c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/204012929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd371f40 msgr2=0x55b9fd37c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.551 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd371f40 0x55b9fd37c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 msgr2=0x55b9fd37c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd37c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/204012929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9fd0b47d0 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00ea90 tx=0x7f7a1c0133a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6340f7caefcedc1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.542+0000 7f7a0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c02a060 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f7a1c013680 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a1c0168d0 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7a1c01b030 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 10 client.16146 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b9fd386bd0 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c0168d0 con 0x55b9fd377aa0 2024-05-09T09:20:59.552 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7a1c021070 con 0x55b9fd377aa0 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 10 client.16146 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 10 client.16146 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 20 client.16146 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd388130 0x55b9fd38a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 4 client.16146 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b9fd2e4370 con 0x55b9fd388130 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.543+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd388130 0x55b9fd38a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.544+0000 7f7a0d7fa640 10 client.16146 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.544+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd388130 0x55b9fd38a520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b169087d07b72e4d server_cookie=87639ab11af5a095 in_seq=0 out_seq=0 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/204012929 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b9fd2e4370 con 0x55b9fd388130 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a0d7fa640 10 client.16146 handle_command_reply: tid=0 2024-05-09T09:20:59.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 client.16146 shutdown 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 msgr2=0x55b9fd383860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00ea90 tx=0x7f7a1c0133a0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 shutdown_connections 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd388130 0x55b9fd38a520 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd383860 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd372930 0x55b9fd37c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/204012929 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd371f40 0x55b9fd37c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a0d7fa640 0 client.16146 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 >> 172.21.15.69:0/204012929 conn(0x55b9fd36d5b0 msgr2=0x55b9fd36c7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 shutdown_connections 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 1 -- 172.21.15.69:0/204012929 wait complete. 2024-05-09T09:20:59.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.545+0000 7f7a32e5a740 20 client.16146 trim_cache size 0 max 16384 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 Processor -- start 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 -- start start 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd383820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd377aa0 0x55b9fd370a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281af1c0 con 0x55b9fd371f40 2024-05-09T09:20:59.564 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281aca30 con 0x55b9fd377aa0 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281aa880 con 0x55b9fd372930 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.558+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41308/0 (socket says 172.21.15.69:41308) 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/795691932 learned_addr learned my addr 172.21.15.69:0/795691932 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/795691932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 msgr2=0x55b9fd383820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd383820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/795691932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd377aa0 msgr2=0x55b9fd370a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd377aa0 0x55b9fd370a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/795691932 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b9fd35ed20 con 0x55b9fd371f40 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00af40 tx=0x7f7a1c00ad60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23c3da838f6f1c7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a0f7fe640 1 -- 172.21.15.69:0/795691932 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c02a040 con 0x55b9fd371f40 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a0f7fe640 1 -- 172.21.15.69:0/795691932 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c01c040 con 0x55b9fd371f40 2024-05-09T09:20:59.565 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.559+0000 7f7a0f7fe640 1 -- 172.21.15.69:0/795691932 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c005de0 con 0x55b9fd371f40 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- 172.21.15.69:0/795691932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 msgr2=0x55b9fd37af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c00af40 tx=0x7f7a1c00ad60 comp rx=0 tx=0).stop 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- 172.21.15.69:0/795691932 shutdown_connections 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd377aa0 0x55b9fd370a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd383820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/795691932 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd371f40 0x55b9fd37af10 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- 172.21.15.69:0/795691932 >> 172.21.15.69:0/795691932 conn(0x55b9fd36d5b0 msgr2=0x55b9fd370330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- 172.21.15.69:0/795691932 shutdown_connections 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- 172.21.15.69:0/795691932 wait complete. 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- start start 2024-05-09T09:20:59.566 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 Processor -- start 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd371f40 0x55b9fd3811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd381710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281aaef0 con 0x55b9fd377aa0 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281ac360 con 0x55b9fd371f40 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7a281ab5c0 con 0x55b9fd372930 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a32e5a740 10 client.0 _ll_register_callbacks cb 0x7f7a1c019070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a2d6ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41322/0 (socket says 172.21.15.69:41322) 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.560+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1672794041 learned_addr learned my addr 172.21.15.69:0/1672794041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd371f40 0x55b9fd3811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd381710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.567 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1672794041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 msgr2=0x55b9fd381710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd381710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd371f40 msgr2=0x55b9fd3811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd371f40 0x55b9fd3811d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 -- 172.21.15.69:0/1672794041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b9fd382d10 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a2d6ea640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c013e50 tx=0x7f7a1c027040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e074e9feb58fab4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7a1c006610 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f7a1c0067b0 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7a1c01c040 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f7a1c0134e0 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a32e5a740 10 client.16152 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55b9fd3ee870 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7a1c01c040 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.561+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f7a1c034070 con 0x55b9fd377aa0 2024-05-09T09:20:59.568 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 10 client.16152 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 10 client.16152 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.h 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 20 client.16152 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd3ef100 0x55b9fd3f1520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 4 client.16152 mds_command: new command op to 24491 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 --> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] -- command(tid 0: {"prefix": "damage ls"}) -- 0x55b9fd2de6c0 con 0x55b9fd3ef100 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd3ef100 0x55b9fd3f1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a0d7fa640 10 client.16152 ms_handle_connect on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.562+0000 7f7a2deeb640 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd3ef100 0x55b9fd3f1520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=6e033b73745d02fa server_cookie=dcb3e43c59ce61dd in_seq=0 out_seq=0 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a0d7fa640 1 -- 172.21.15.69:0/1672794041 <== mds.2 v2:172.21.15.103:6836/1084447179 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4371 (crc 0 0 0) 0x55b9fd2de6c0 con 0x55b9fd3ef100 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a0d7fa640 10 client.16152 handle_command_reply: tid=0 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 client.16152 shutdown 2024-05-09T09:20:59.569 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 msgr2=0x55b9fd382720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f7a1c013e50 tx=0x7f7a1c027040 comp rx=0 tx=0).stop 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 shutdown_connections 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:6836/1084447179,v1:172.21.15.103:6837/1084447179] conn(0x55b9fd3ef100 0x55b9fd3f1520 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b9fd377aa0 0x55b9fd382720 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b9fd372930 0x55b9fd381710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a0d7fa640 0 client.16152 ms_handle_reset on v2:172.21.15.103:6836/1084447179 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 --2- 172.21.15.69:0/1672794041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b9fd371f40 0x55b9fd3811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 >> 172.21.15.69:0/1672794041 conn(0x55b9fd36d5b0 msgr2=0x55b9fd370bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 shutdown_connections 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 1 -- 172.21.15.69:0/1672794041 wait complete. 2024-05-09T09:20:59.570 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a32e5a740 20 client.16152 trim_cache size 0 max 16384 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 146283825, 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628327, 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC" 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 414764831, 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628308, 2024-05-09T09:20:59.571 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT" 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 717784298, 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628313, 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR" 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.572 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1290319711, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628307, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT" 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1399979270, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628317, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR" 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1502547320, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628318, 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR" 2024-05-09T09:20:59.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1772988225, 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628311, 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG" 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1914427505, 2024-05-09T09:20:59.574 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628320, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT" 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2047076189, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628305, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP" 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2264206023, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628310, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG" 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.575 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2347558041, 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628328, 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC" 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2404976457, 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628304, 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT" 2024-05-09T09:20:59.576 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2544295677, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628312, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL" 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3068605026, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628326, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS" 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.577 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3219078762, 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628315, 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT" 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3641844568, 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628319, 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT" 2024-05-09T09:20:59.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3823753704, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628314, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT" 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4110299333, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628306, 2024-05-09T09:20:59.579 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA" 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4290404541, 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628316, 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT" 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:20:59.580 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 msgr2=0x7f7a100a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 0x7f7a100a5050 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f7a20011fd0 tx=0x7f7a20065f90 comp rx=0 tx=0).stop 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 msgr2=0x7f7a28074160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f7a20009fd0 tx=0x7f7a2000b8e0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 shutdown_connections 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.563+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7a100a2b90 0x7f7a100a5050 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7a28103430 0x7f7a28074160 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.582 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7a28102a70 0x7f7a28073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 --2- 172.21.15.69:0/1901851891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7a28101880 0x7f7a280755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 >> 172.21.15.69:0/1901851891 conn(0x7f7a280fce10 msgr2=0x7f7a28105e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 shutdown_connections 2024-05-09T09:20:59.583 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.564+0000 7f7a3014f640 1 -- 172.21.15.69:0/1901851891 wait complete. 2024-05-09T09:20:59.584 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-05-09T09:20:59.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 -- 172.21.15.69:0/2296787570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870101960 msgr2=0x7f1870101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2296787570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870101960 0x7f1870101de0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f1858009900 tx=0x7f185802a410 comp rx=0 tx=0).stop 2024-05-09T09:20:59.727 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 -- 172.21.15.69:0/2296787570 shutdown_connections 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2296787570 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f1870108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2296787570 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870101960 0x7f1870101de0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2296787570 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870100770 0x7f1870100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 -- 172.21.15.69:0/2296787570 >> 172.21.15.69:0/2296787570 conn(0x7f18700fbce0 msgr2=0x7f18700fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 -- 172.21.15.69:0/2296787570 shutdown_connections 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.726+0000 7f1874ae1640 1 -- 172.21.15.69:0/2296787570 wait complete. 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 Processor -- start 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 -- start start 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870101960 0x7f1870198170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f187019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f187010c450 con 0x7f1870100770 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f187010c2d0 con 0x7f1870102320 2024-05-09T09:20:59.728 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1874ae1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f187010c5d0 con 0x7f1870101960 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41344/0 (socket says 172.21.15.69:41344) 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 -- 172.21.15.69:0/2567161783 learned_addr learned my addr 172.21.15.69:0/2567161783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f1867fff640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870101960 0x7f1870198170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186ed9d640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f187019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 -- 172.21.15.69:0/2567161783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870101960 msgr2=0x7f1870198170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870101960 0x7f1870198170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 -- 172.21.15.69:0/2567161783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 msgr2=0x7f187019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f187019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 -- 172.21.15.69:0/2567161783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f187019f840 con 0x7f1870100770 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186ed9d640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f187019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.727+0000 7f186e59c640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f186000c970 tx=0x7f186000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.728+0000 7f18677fe640 1 -- 172.21.15.69:0/2567161783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1860004220 con 0x7f1870100770 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.728+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f187019fb30 con 0x7f1870100770 2024-05-09T09:20:59.729 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.728+0000 7f18677fe640 1 -- 172.21.15.69:0/2567161783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f18600043c0 con 0x7f1870100770 2024-05-09T09:20:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.728+0000 7f18677fe640 1 -- 172.21.15.69:0/2567161783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186000f450 con 0x7f1870100770 2024-05-09T09:20:59.730 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.728+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f18701a00d0 con 0x7f1870100770 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.730+0000 7f18677fe640 1 -- 172.21.15.69:0/2567161783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f18600079b0 con 0x7f1870100770 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 Processor -- start 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 -- start start 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204cd8360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cddaa0 0x561204cdff30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561204c3cc40 con 0x561204cd8930 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561204be4580 con 0x561204cddaa0 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18777f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561204c38c00 con 0x561204cd7f40 2024-05-09T09:20:59.736 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41360/0 (socket says 172.21.15.69:41360) 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186e59c640 1 -- 172.21.15.69:0/243845356 learned_addr learned my addr 172.21.15.69:0/243845356 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186ed9d640 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204cd8360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f1867fff640 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cddaa0 0x561204cdff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f18677fe640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 0x7f185c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186e59c640 1 -- 172.21.15.69:0/243845356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 msgr2=0x561204cd8360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.731+0000 7f186e59c640 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204cd8360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186e59c640 1 -- 172.21.15.69:0/243845356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cddaa0 msgr2=0x561204cdff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186e59c640 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cddaa0 0x561204cdff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186e59c640 1 -- 172.21.15.69:0/243845356 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5612049503f0 con 0x561204cd8930 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186ed9d640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 0x7f185c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f18677fe640 1 -- 172.21.15.69:0/2567161783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f18600cf2a0 con 0x7f1870100770 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186e59c640 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f18600cc960 tx=0x7f18600662c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4007359632b3c43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f1864ff9640 1 -- 172.21.15.69:0/243845356 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186001d070 con 0x561204cd8930 2024-05-09T09:20:59.737 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f186ed9d640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 0x7f185c0a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x561204cd9190 tx=0x7f1868004b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f1864ff9640 1 -- 172.21.15.69:0/243845356 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1860072680 con 0x561204cd8930 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.732+0000 7f1864ff9640 1 -- 172.21.15.69:0/243845356 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186009b2c0 con 0x561204cd8930 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- 172.21.15.69:0/243845356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 msgr2=0x561204cdd430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f18600cc960 tx=0x7f18600662c0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- 172.21.15.69:0/243845356 shutdown_connections 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cddaa0 0x561204cdff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cdd430 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- 172.21.15.69:0/243845356 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204cd8360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- 172.21.15.69:0/243845356 >> 172.21.15.69:0/243845356 conn(0x561204cd35b0 msgr2=0x561204cd39b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- 172.21.15.69:0/243845356 shutdown_connections 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- 172.21.15.69:0/243845356 wait complete. 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- start start 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 Processor -- start 2024-05-09T09:20:59.738 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cddaa0 0x561204ce9790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561204ce0990 con 0x561204cd7f40 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561204bf7c40 con 0x561204cd8930 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561204ce0b10 con 0x561204cddaa0 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.733+0000 7f18777f5740 10 client.0 _ll_register_callbacks cb 0x56120494fbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41368/0 (socket says 172.21.15.69:41368) 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 -- 172.21.15.69:0/2164912566 learned_addr learned my addr 172.21.15.69:0/2164912566 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1867fff640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186ed9d640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cddaa0 0x561204ce9790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 -- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cddaa0 msgr2=0x561204ce9790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cddaa0 0x561204ce9790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 -- 172.21.15.69:0/2164912566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 msgr2=0x561204ce2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce2750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 -- 172.21.15.69:0/2164912566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561204a1a7d0 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f186e59c640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f18600ccd90 tx=0x7f1860003f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69b6c5b21600f022 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186001d040 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f1860099a80 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x561204c4a370 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1860072280 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f18777f5740 10 client.16164 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561204cecb00 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x561204c4a370 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.734+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f1860007ca0 con 0x561204cd7f40 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 10 client.16164 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 10 client.16164 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:20:59.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 20 client.16164 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cede50 0x561204cf0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 4 client.16164 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561204c446c0 con 0x561204cede50 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f1867fff640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cede50 0x561204cf0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f1855ffb640 10 client.16164 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.735+0000 7f1867fff640 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cede50 0x561204cf0240 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=71b7677238f4f9d7 server_cookie=fc61f92716d510f0 in_seq=0 out_seq=0 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f1855ffb640 1 -- 172.21.15.69:0/2164912566 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561204c446c0 con 0x561204cede50 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f1855ffb640 10 client.16164 handle_command_reply: tid=0 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 client.16164 shutdown 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 msgr2=0x561204ce2210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7f18600ccd90 tx=0x7f1860003f80 comp rx=0 tx=0).stop 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 shutdown_connections 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cede50 0x561204cf0240 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cddaa0 0x561204ce9790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f1855ffb640 0 client.16164 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 --2- 172.21.15.69:0/2164912566 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd7f40 0x561204ce2210 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 >> 172.21.15.69:0/2164912566 conn(0x561204cd35b0 msgr2=0x561204cd27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 shutdown_connections 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 1 -- 172.21.15.69:0/2164912566 wait complete. 2024-05-09T09:20:59.742 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.736+0000 7f18777f5740 20 client.16164 trim_cache size 0 max 16384 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.749+0000 7f18777f5740 1 Processor -- start 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.749+0000 7f18777f5740 1 -- start start 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd35b0 0x561204ce2530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18701ace90 con 0x561204cd8930 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18701aa700 con 0x561204cd35b0 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f18777f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18701a8550 con 0x561204cd7f40 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.754 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41370/0 (socket says 172.21.15.69:41370) 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 -- 172.21.15.69:0/1384799661 learned_addr learned my addr 172.21.15.69:0/1384799661 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f1867fff640 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186e59c640 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd35b0 0x561204ce2530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 -- 172.21.15.69:0/1384799661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 msgr2=0x561204ce9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 -- 172.21.15.69:0/1384799661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd35b0 msgr2=0x561204ce2530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd35b0 0x561204ce2530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 -- 172.21.15.69:0/1384799661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x561204cd4000 con 0x561204cd8930 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f186ed9d640 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f18680117c0 tx=0x7f1868012000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5f0d459744a89e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f1857fff640 1 -- 172.21.15.69:0/1384799661 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186801c880 con 0x561204cd8930 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f1857fff640 1 -- 172.21.15.69:0/1384799661 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f186801a440 con 0x561204cd8930 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.750+0000 7f1857fff640 1 -- 172.21.15.69:0/1384799661 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1868025620 con 0x561204cd8930 2024-05-09T09:20:59.755 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- 172.21.15.69:0/1384799661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 msgr2=0x561204cd1f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7f18680117c0 tx=0x7f1868012000 comp rx=0 tx=0).stop 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- 172.21.15.69:0/1384799661 shutdown_connections 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd8930 0x561204cd1f20 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- 172.21.15.69:0/1384799661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd35b0 0x561204ce2530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- 172.21.15.69:0/1384799661 >> 172.21.15.69:0/1384799661 conn(0x561204cd9530 msgr2=0x561204ce1e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- 172.21.15.69:0/1384799661 shutdown_connections 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- 172.21.15.69:0/1384799661 wait complete. 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- start start 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 Processor -- start 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce86b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f18701a8bc0 con 0x561204cd35b0 2024-05-09T09:20:59.756 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f18701aa030 con 0x561204cd8930 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.751+0000 7f18777f5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f18701a9290 con 0x561204cd7f40 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f18777f5740 10 client.0 _ll_register_callbacks cb 0x7f1858009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41386/0 (socket says 172.21.15.69:41386) 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 -- 172.21.15.69:0/1104380290 learned_addr learned my addr 172.21.15.69:0/1104380290 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1867fff640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 -- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 msgr2=0x561204ce76a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce76a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 -- 172.21.15.69:0/1104380290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 msgr2=0x561204ce86b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce86b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 -- 172.21.15.69:0/1104380290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561204ce8ca0 con 0x561204cd35b0 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1867fff640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f186e59c640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f18600a4460 tx=0x7f186006ee70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7286ce8216a5f54c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.757 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f186001d040 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f18600d2a80 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f186801a440 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f1860072480 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f18777f5740 10 client.16170 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x561204d54570 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.752+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f186801a440 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f186006a0d0 con 0x561204cd35b0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 10 client.16170 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 10 client.16170 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.i 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 20 client.16170 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cddaa0 0x561204d56df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 4 client.16170 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 --> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] -- command(tid 0: {"prefix": "damage ls"}) -- 0x561204bc1820 con 0x561204cddaa0 2024-05-09T09:20:59.758 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f1867fff640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cddaa0 0x561204d56df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f1855ffb640 10 client.16170 ms_handle_connect on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.753+0000 7f1867fff640 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cddaa0 0x561204d56df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=49033d9a4376ee01 server_cookie=5516e9a3525dfd in_seq=0 out_seq=0 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1855ffb640 1 -- 172.21.15.69:0/1104380290 <== mds.1 v2:172.21.15.163:6838/282166335 1 ==== command_reply(tid 0: 0 ) ==== 8+0+4651 (crc 0 0 0) 0x561204bc1820 con 0x561204cddaa0 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1855ffb640 10 client.16170 handle_command_reply: tid=0 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 client.16170 shutdown 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 msgr2=0x561204ce7160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7f18600a4460 tx=0x7f186006ee70 comp rx=0 tx=0).stop 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 shutdown_connections 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:6838/282166335,v1:172.21.15.163:6839/282166335] conn(0x561204cddaa0 0x561204d56df0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561204cd8930 0x561204ce86b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561204cd7f40 0x561204ce76a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1855ffb640 0 client.16170 ms_handle_reset on v2:172.21.15.163:6838/282166335 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 --2- 172.21.15.69:0/1104380290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561204cd35b0 0x561204ce7160 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 >> 172.21.15.69:0/1104380290 conn(0x561204cd9530 msgr2=0x561204cd1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.759 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 shutdown_connections 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 1 -- 172.21.15.69:0/1104380290 wait complete. 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f18777f5740 20 client.16170 trim_cache size 0 max 16384 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout:[ 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 118752583, 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628343, 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX" 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.760 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 232227133, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628334, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB" 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 358625904, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628347, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV" 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 458482697, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628333, 2024-05-09T09:20:59.761 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04" 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 478646865, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628337, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0" 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 854412352, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628346, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL" 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 902962587, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628336, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0" 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 914121271, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628340, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL" 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 986703558, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628339, 2024-05-09T09:20:59.763 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1" 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1398200076, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628344, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03" 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 1511506649, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628345, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03" 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.764 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2140678848, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628330, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB" 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2542176692, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628338, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1" 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2653528617, 2024-05-09T09:20:59.765 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628335, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX" 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2747365424, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628332, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04" 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 2830062381, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628331, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX" 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.766 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3220339795, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628341, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB" 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3231290568, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628349, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL" 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 3520619889, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628348, 2024-05-09T09:20:59.767 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL" 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: "damage_type": "backtrace", 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: "id": 4223655767, 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: "ino": 1099511628342, 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: "path": "/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB" 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:20:59.768 INFO:teuthology.orchestra.run.smithi069.stdout:] 2024-05-09T09:20:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 msgr2=0x7f185c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 0x7f185c0a5050 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x561204cd9190 tx=0x7f1868004b50 comp rx=0 tx=0).stop 2024-05-09T09:20:59.769 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 msgr2=0x7f1870197c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f186000c970 tx=0x7f186000ce30 comp rx=0 tx=0).stop 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 shutdown_connections 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f185c0a2b90 0x7f185c0a5050 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1870102320 0x7f187019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1870101960 0x7f1870198170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 --2- 172.21.15.69:0/2567161783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1870100770 0x7f1870197c30 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 >> 172.21.15.69:0/2567161783 conn(0x7f18700fbce0 msgr2=0x7f1870069e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 shutdown_connections 2024-05-09T09:20:59.770 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.754+0000 7f1874ae1640 1 -- 172.21.15.69:0/2567161783 wait complete. 2024-05-09T09:20:59.770 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-05-09T09:20:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 -- 172.21.15.69:0/2600300096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc101960 msgr2=0x7fecbc101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 --2- 172.21.15.69:0/2600300096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc101960 0x7fecbc101de0 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fecac009900 tx=0x7fecac02a410 comp rx=0 tx=0).stop 2024-05-09T09:20:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 -- 172.21.15.69:0/2600300096 shutdown_connections 2024-05-09T09:20:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 --2- 172.21.15.69:0/2600300096 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc102320 0x7fecbc108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 --2- 172.21.15.69:0/2600300096 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc101960 0x7fecbc101de0 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 --2- 172.21.15.69:0/2600300096 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 -- 172.21.15.69:0/2600300096 >> 172.21.15.69:0/2600300096 conn(0x7fecbc0fbce0 msgr2=0x7fecbc0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 -- 172.21.15.69:0/2600300096 shutdown_connections 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.874+0000 7fecc45ce640 1 -- 172.21.15.69:0/2600300096 wait complete. 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 Processor -- start 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 -- start start 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc101960 0x7fecbc102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10c3f0 con 0x7fecbc102320 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10c270 con 0x7fecbc100770 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc45ce640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc10c570 con 0x7fecbc101960 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc2b6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc2b6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41402/0 (socket says 172.21.15.69:41402) 2024-05-09T09:20:59.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc2b6b640 1 -- 172.21.15.69:0/3286073662 learned_addr learned my addr 172.21.15.69:0/3286073662 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc236a640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.875+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc101960 0x7fecbc102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 -- 172.21.15.69:0/3286073662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc101960 msgr2=0x7fecbc102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc101960 0x7fecbc102d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 -- 172.21.15.69:0/3286073662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 msgr2=0x7fecbc104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc104700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 -- 172.21.15.69:0/3286073662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fecbc103b10 con 0x7fecbc102320 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc236a640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:59.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc2b6b640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fecb400cad0 tx=0x7fecb400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecb37fe640 1 -- 172.21.15.69:0/3286073662 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb400ae20 con 0x7fecbc102320 2024-05-09T09:20:59.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecb37fe640 1 -- 172.21.15.69:0/3286073662 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb4005020 con 0x7fecbc102320 2024-05-09T09:20:59.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fecbc1a1d60 con 0x7fecbc102320 2024-05-09T09:20:59.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecb37fe640 1 -- 172.21.15.69:0/3286073662 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb400f6b0 con 0x7fecbc102320 2024-05-09T09:20:59.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.876+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fecbc1a21d0 con 0x7fecbc102320 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecb37fe640 1 -- 172.21.15.69:0/3286073662 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fecb40051d0 con 0x7fecbc102320 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecc73b9740 1 Processor -- start 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecc73b9740 1 -- start start 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb502f40 0x560efb503360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 0x560efb508430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.879+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc73b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560efb467c40 con 0x560efb508aa0 2024-05-09T09:20:59.882 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc73b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560efb40f580 con 0x560efb502f40 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc73b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560efb463c00 con 0x560efb503930 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41404/0 (socket says 172.21.15.69:41404) 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc1b69640 1 -- 172.21.15.69:0/849914314 learned_addr learned my addr 172.21.15.69:0/849914314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 0x560efb508430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc1b69640 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb502f40 0x560efb503360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecb37fe640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 0x7feca80a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 -- 172.21.15.69:0/849914314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb502f40 msgr2=0x560efb503360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb502f40 0x560efb503360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 -- 172.21.15.69:0/849914314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 msgr2=0x560efb50af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.883 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 -- 172.21.15.69:0/849914314 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560efb17b3f0 con 0x560efb503930 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc1b69640 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecb37fe640 1 -- 172.21.15.69:0/3286073662 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fecb40cf680 con 0x7fecbc102320 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc2b6b640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 0x7feca80a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc236a640 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 0x560efb508430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecbc1017c0 tx=0x7fecb80108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c9040e6e2e91d47a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.880+0000 7fecc2b6b640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 0x7feca80a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fecb40985e0 tx=0x7fecb4000f40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecb0bfe640 1 -- 172.21.15.69:0/849914314 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8005220 con 0x560efb503930 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecb0bfe640 1 -- 172.21.15.69:0/849914314 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8004c80 con 0x560efb503930 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecb0bfe640 1 -- 172.21.15.69:0/849914314 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8011b80 con 0x560efb503930 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 -- 172.21.15.69:0/849914314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 msgr2=0x560efb508430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 0x560efb508430 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fecbc1017c0 tx=0x7fecb80108e0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.884 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 -- 172.21.15.69:0/849914314 shutdown_connections 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb50af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb503930 0x560efb508430 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 --2- 172.21.15.69:0/849914314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb502f40 0x560efb503360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 -- 172.21.15.69:0/849914314 >> 172.21.15.69:0/849914314 conn(0x560efb4fe5b0 msgr2=0x560efb4fe9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 -- 172.21.15.69:0/849914314 shutdown_connections 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.881+0000 7fecc73b9740 1 -- 172.21.15.69:0/849914314 wait complete. 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 -- start start 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 Processor -- start 2024-05-09T09:20:59.885 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50d330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb5148b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560efb50ba50 con 0x560efb503930 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x560efb422c40 con 0x560efb508aa0 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560efb50bbd0 con 0x560efb502f40 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc73b9740 10 client.0 _ll_register_callbacks cb 0x560efb17abe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41412/0 (socket says 172.21.15.69:41412) 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 -- 172.21.15.69:0/3723867240 learned_addr learned my addr 172.21.15.69:0/3723867240 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.894 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb5148b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50d330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 -- 172.21.15.69:0/3723867240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 msgr2=0x560efb50d330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50d330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.882+0000 7fecc236a640 1 -- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 msgr2=0x560efb5148b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc236a640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb5148b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc236a640 1 -- 172.21.15.69:0/3723867240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560efb2457d0 con 0x560efb503930 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb5148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc236a640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fecb8011660 tx=0x7fecb800bc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adddcbbe4161de74 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8027020 con 0x560efb503930 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fecb8004c80 con 0x560efb503930 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fecb8004020 con 0x560efb503930 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fecb8004240 con 0x560efb503930 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 10 client.16182 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.895 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560efb517c70 con 0x560efb503930 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb8004c80 con 0x560efb503930 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fecb8005410 con 0x560efb503930 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 10 client.16182 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 10 client.16182 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 20 client.16182 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb5191d0 0x560efb51b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 4 client.16182 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.883+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x560efb475370 con 0x560efb5191d0 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.884+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb5191d0 0x560efb51b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.884+0000 7fecc1b69640 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb5191d0 0x560efb51b5f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b73c52f8d60a8d1 server_cookie=7f8640e79007461e in_seq=0 out_seq=0 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.884+0000 7feca5ffb640 10 client.16182 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7feca5ffb640 1 -- 172.21.15.69:0/3723867240 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x560efb475370 con 0x560efb5191d0 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7feca5ffb640 10 client.16182 handle_command_reply: tid=0 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 client.16182 shutdown 2024-05-09T09:20:59.896 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 msgr2=0x560efb50d870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fecb8011660 tx=0x7fecb800bc90 comp rx=0 tx=0).stop 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 shutdown_connections 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb5148b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb50d870 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb5191d0 0x560efb51b5f0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 --2- 172.21.15.69:0/3723867240 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50d330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7feca5ffb640 0 client.16182 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 >> 172.21.15.69:0/3723867240 conn(0x560efb4fe5b0 msgr2=0x560efb4fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 shutdown_connections 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 1 -- 172.21.15.69:0/3723867240 wait complete. 2024-05-09T09:20:59.897 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.885+0000 7fecc73b9740 20 client.16182 trim_cache size 0 max 16384 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.898+0000 7fecc73b9740 1 Processor -- start 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.898+0000 7fecc73b9740 1 -- start start 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb50cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1aefe0 con 0x560efb503930 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1ac850 con 0x560efb508aa0 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc73b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1aa6a0 con 0x560efb502f40 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41414/0 (socket says 172.21.15.69:41414) 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 -- 172.21.15.69:0/2915529657 learned_addr learned my addr 172.21.15.69:0/2915529657 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc1b69640 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc1b69640 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb50cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 -- 172.21.15.69:0/2915529657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 msgr2=0x560efb50c600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50c600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 -- 172.21.15.69:0/2915529657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 msgr2=0x560efb50cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb50cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 -- 172.21.15.69:0/2915529657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560efb50bda0 con 0x560efb503930 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7fecc236a640 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bf70 tx=0x7fecb800bfa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bb04ecd394a4df5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.899+0000 7feca7fff640 1 -- 172.21.15.69:0/2915529657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb803a070 con 0x560efb503930 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7feca7fff640 1 -- 172.21.15.69:0/2915529657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fecb80057b0 con 0x560efb503930 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7feca7fff640 1 -- 172.21.15.69:0/2915529657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecb8027040 con 0x560efb503930 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 -- 172.21.15.69:0/2915529657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 msgr2=0x560efb514870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7fecb800bf70 tx=0x7fecb800bfa0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 -- 172.21.15.69:0/2915529657 shutdown_connections 2024-05-09T09:20:59.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb508aa0 0x560efb50cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb503930 0x560efb514870 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 --2- 172.21.15.69:0/2915529657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb50c600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 -- 172.21.15.69:0/2915529657 >> 172.21.15.69:0/2915529657 conn(0x560efb4fe5b0 msgr2=0x560efb504c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 -- 172.21.15.69:0/2915529657 shutdown_connections 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.900+0000 7fecc73b9740 1 -- 172.21.15.69:0/2915529657 wait complete. 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 -- start start 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 Processor -- start 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb511610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 0x560efb511b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1aad10 con 0x560efb508aa0 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1ac180 con 0x560efb503930 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fecbc1ab3e0 con 0x560efb502f40 2024-05-09T09:20:59.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc73b9740 10 client.0 _ll_register_callbacks cb 0x7fecac005dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41418/0 (socket says 172.21.15.69:41418) 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 -- 172.21.15.69:0/1245204963 learned_addr learned my addr 172.21.15.69:0/1245204963 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc236a640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 0x560efb511b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb511610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 -- 172.21.15.69:0/1245204963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 msgr2=0x560efb511610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb511610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 -- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 msgr2=0x560efb511b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 0x560efb511b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 -- 172.21.15.69:0/1245204963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x560efb512190 con 0x560efb508aa0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc236a640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 0x560efb511b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7fecc1b69640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fecac009230 tx=0x7fecb8004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83cef675021a288 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.901+0000 7feca5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fecac035040 con 0x560efb508aa0 2024-05-09T09:20:59.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7fecac004240 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x560efb46f6c0 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7fecac062050 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 10 client.16188 fetch_fsmap learned FSMap version 30 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x560efb57ee00 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x560efb46f6c0 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7fecac004810 con 0x560efb508aa0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 10 client.16188 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 10 client.16188 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 20 client.16188 populate_metadata read hostname 'smithi069' 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb57f7c0 0x560efb581be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 4 client.16188 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "damage ls"}) -- 0x560efb3ec820 con 0x560efb57f7c0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.902+0000 7fecc236a640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb57f7c0 0x560efb581be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:20:59.910 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7feca5ffb640 10 client.16188 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc236a640 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb57f7c0 0x560efb581be0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17932b2c9fa29a32 server_cookie=c246066c7d8ab8a1 in_seq=0 out_seq=0 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7feca5ffb640 1 -- 172.21.15.69:0/1245204963 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+3 (crc 0 0 0) 0x560efb3ec820 con 0x560efb57f7c0 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7feca5ffb640 10 client.16188 handle_command_reply: tid=0 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 client.16188 shutdown 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 msgr2=0x560efb57d720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7fecac009230 tx=0x7fecb8004e30 comp rx=0 tx=0).stop 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 shutdown_connections 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560efb508aa0 0x560efb57d720 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x560efb503930 0x560efb511b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560efb57f7c0 0x560efb581be0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 --2- 172.21.15.69:0/1245204963 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560efb502f40 0x560efb511610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 >> 172.21.15.69:0/1245204963 conn(0x560efb4fe5b0 msgr2=0x560efb4fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7feca5ffb640 0 client.16188 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 shutdown_connections 2024-05-09T09:20:59.911 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 1 -- 172.21.15.69:0/1245204963 wait complete. 2024-05-09T09:20:59.912 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.903+0000 7fecc73b9740 20 client.16188 trim_cache size 0 max 16384 2024-05-09T09:20:59.912 INFO:teuthology.orchestra.run.smithi069.stdout:[] 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 msgr2=0x7feca80a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 0x7feca80a5050 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fecb40985e0 tx=0x7fecb4000f40 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 msgr2=0x7fecbc103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fecb400cad0 tx=0x7fecb400cf90 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 shutdown_connections 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7feca80a2b90 0x7feca80a5050 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fecbc102320 0x7fecbc103280 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fecbc101960 0x7fecbc102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 --2- 172.21.15.69:0/3286073662 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fecbc100770 0x7fecbc104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 >> 172.21.15.69:0/3286073662 conn(0x7fecbc0fbce0 msgr2=0x7fecbc0fd810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 shutdown_connections 2024-05-09T09:20:59.913 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:20:59.904+0000 7fecc45ce640 1 -- 172.21.15.69:0/3286073662 wait complete. 2024-05-09T09:20:59.914 ERROR:tasks.fwd_scrub:rank 2 damaged: [{'damage_type': 'backtrace', 'id': 146283825, 'ino': 1099511628327, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/EVENTS.DOC'}, {'damage_type': 'backtrace', 'id': 414764831, 'ino': 1099511628308, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TRIDOTS.POT'}, {'damage_type': 'backtrace', 'id': 717784298, 'ino': 1099511628313, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC1.CDR'}, {'damage_type': 'backtrace', 'id': 1290319711, 'ino': 1099511628307, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/TIPS.PPT'}, {'damage_type': 'backtrace', 'id': 1399979270, 'ino': 1099511628317, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TRAIN.CDR'}, {'damage_type': 'backtrace', 'id': 1502547320, 'ino': 1099511628318, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/GRAPHIC2.CDR'}, {'damage_type': 'backtrace', 'id': 1772988225, 'ino': 1099511628311, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRBARS.CFG'}, {'damage_type': 'backtrace', 'id': 1914427505, 'ino': 1099511628320, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CORELDRW.CDT'}, {'damage_type': 'backtrace', 'id': 2047076189, 'ino': 1099511628305, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/ZD16.BMP'}, {'damage_type': 'backtrace', 'id': 2264206023, 'ino': 1099511628310, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/CDRROLS.CFG'}, {'damage_type': 'backtrace', 'id': 2347558041, 'ino': 1099511628328, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/MOVED.DOC'}, {'damage_type': 'backtrace', 'id': 2404976457, 'ino': 1099511628304, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PCBENCHM.PPT'}, {'damage_type': 'backtrace', 'id': 2544295677, 'ino': 1099511628312, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ARTISTIC.ACL'}, {'damage_type': 'backtrace', 'id': 3068605026, 'ino': 1099511628326, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PM/T1.XLS'}, {'damage_type': 'backtrace', 'id': 3219078762, 'ino': 1099511628315, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/TEAPOT2.CPT'}, {'damage_type': 'backtrace', 'id': 3641844568, 'ino': 1099511628319, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/ROSE.CPT'}, {'damage_type': 'backtrace', 'id': 3823753704, 'ino': 1099511628314, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/MAILBOX2.CPT'}, {'damage_type': 'backtrace', 'id': 4110299333, 'ino': 1099511628306, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PWRPNT/PPTOOLS1.PPA'}, {'damage_type': 'backtrace', 'id': 4290404541, 'ino': 1099511628316, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/COREL/SIGN2.CPT'}] 2024-05-09T09:20:59.915 ERROR:tasks.fwd_scrub:rank 1 damaged: [{'damage_type': 'backtrace', 'id': 118752583, 'ino': 1099511628343, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.PX'}, {'damage_type': 'backtrace', 'id': 232227133, 'ino': 1099511628334, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.DB'}, {'damage_type': 'backtrace', 'id': 358625904, 'ino': 1099511628347, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.TV'}, {'damage_type': 'backtrace', 'id': 458482697, 'ino': 1099511628333, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.Y04'}, {'damage_type': 'backtrace', 'id': 478646865, 'ino': 1099511628337, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG0'}, {'damage_type': 'backtrace', 'id': 854412352, 'ino': 1099511628346, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.VAL'}, {'damage_type': 'backtrace', 'id': 902962587, 'ino': 1099511628336, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG0'}, {'damage_type': 'backtrace', 'id': 914121271, 'ino': 1099511628340, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.VAL'}, {'damage_type': 'backtrace', 'id': 986703558, 'ino': 1099511628339, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.YG1'}, {'damage_type': 'backtrace', 'id': 1398200076, 'ino': 1099511628344, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.X03'}, {'damage_type': 'backtrace', 'id': 1511506649, 'ino': 1099511628345, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.Y03'}, {'damage_type': 'backtrace', 'id': 2140678848, 'ino': 1099511628330, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.DB'}, {'damage_type': 'backtrace', 'id': 2542176692, 'ino': 1099511628338, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.XG1'}, {'damage_type': 'backtrace', 'id': 2653528617, 'ino': 1099511628335, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.PX'}, {'damage_type': 'backtrace', 'id': 2747365424, 'ino': 1099511628332, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.X04'}, {'damage_type': 'backtrace', 'id': 2830062381, 'ino': 1099511628331, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.PX'}, {'damage_type': 'backtrace', 'id': 3220339795, 'ino': 1099511628341, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.MB'}, {'damage_type': 'backtrace', 'id': 3231290568, 'ino': 1099511628349, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/COURSES.FSL'}, {'damage_type': 'backtrace', 'id': 3520619889, 'ino': 1099511628348, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/STUDENTS.FSL'}, {'damage_type': 'backtrace', 'id': 4223655767, 'ino': 1099511628342, 'path': '/volumes/qa/sv_1/84be3150-10b2-4ab3-ab07-1a2632c1c47d/client.0/tmp/clients/client0/~dmtmp/PARADOX/ENROLL.DB'}] 2024-05-09T09:20:59.915 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 90, in _check_damage raise RuntimeError(f"rank damage found: {types}") RuntimeError: rank damage found: {'backtrace'} 2024-05-09T09:21:00.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 43982 4.40 MB/sec warmup 58 sec latency 4884.984 ms 2024-05-09T09:21:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:00 smithi163 ceph-mon[34354]: pgmap v429: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-05-09T09:21:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:00 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3998648188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:21:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:00 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:21:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:00 smithi103 ceph-mon[34421]: pgmap v429: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-05-09T09:21:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:00 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3998648188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:21:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:00 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:21:00.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:00 smithi069 ceph-mon[30123]: pgmap v429: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 463 KiB/s rd, 1.0 MiB/s wr, 92 op/s 2024-05-09T09:21:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:00 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3998648188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:21:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:00 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:21:01.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46532 4.57 MB/sec warmup 59 sec latency 11.509 ms 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:01 smithi069 ceph-mon[30123]: pgmap v430: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:01 smithi163 ceph-mon[34354]: pgmap v430: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-05-09T09:21:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:21:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:21:01.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:21:01.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:01 smithi103 ceph-mon[34421]: pgmap v430: 129 pgs: 129 active+clean; 28 MiB data, 453 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.0 MiB/s wr, 94 op/s 2024-05-09T09:21:02.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46814 4.49 MB/sec warmup 60 sec latency 708.757 ms 2024-05-09T09:21:03.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.063+0000 7f64c7309640 1 -- 172.21.15.69:0/304033864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c0101a40 msgr2=0x7f64c0108360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.063+0000 7f64c7309640 1 --2- 172.21.15.69:0/304033864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c0101a40 0x7f64c0108360 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f64b8009ca0 tx=0x7f64b802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- 172.21.15.69:0/304033864 shutdown_connections 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- 172.21.15.69:0/304033864 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c0101a40 0x7f64c0108360 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- 172.21.15.69:0/304033864 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 0x7f64c0101500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- 172.21.15.69:0/304033864 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c00fff30 0x7f64c0100350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- 172.21.15.69:0/304033864 >> 172.21.15.69:0/304033864 conn(0x7f64c00fb4a0 msgr2=0x7f64c00fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- 172.21.15.69:0/304033864 shutdown_connections 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- 172.21.15.69:0/304033864 wait complete. 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 Processor -- start 2024-05-09T09:21:03.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- start start 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 0x7f64c0198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 0x7f64c019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64c010bbe0 con 0x7f64c00fff30 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64c010ba60 con 0x7f64c0101a40 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.064+0000 7f64c7309640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64c010bd60 con 0x7f64c0101080 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41448/0 (socket says 172.21.15.69:41448) 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 -- 172.21.15.69:0/2099468298 learned_addr learned my addr 172.21.15.69:0/2099468298 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c48a4640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 0x7f64c0198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c58a6640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 0x7f64c019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 -- 172.21.15.69:0/2099468298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 msgr2=0x7f64c0198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 0x7f64c0198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 -- 172.21.15.69:0/2099468298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 msgr2=0x7f64c019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 0x7f64c019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 -- 172.21.15.69:0/2099468298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f64c019f6e0 con 0x7f64c00fff30 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c58a6640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 0x7f64c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c50a5640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c890 tx=0x7f64bc00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64b67fc640 1 -- 172.21.15.69:0/2099468298 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc004220 con 0x7f64c00fff30 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64b67fc640 1 -- 172.21.15.69:0/2099468298 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64bc0043c0 con 0x7f64c00fff30 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64b67fc640 1 -- 172.21.15.69:0/2099468298 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc0049e0 con 0x7f64c00fff30 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f64c019f9d0 con 0x7f64c00fff30 2024-05-09T09:21:03.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.065+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f64c019ff20 con 0x7f64c00fff30 2024-05-09T09:21:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64b67fc640 1 -- 172.21.15.69:0/2099468298 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f64bc0079b0 con 0x7f64c00fff30 2024-05-09T09:21:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 Processor -- start 2024-05-09T09:21:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 -- start start 2024-05-09T09:21:03.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921426b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921429650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562921381340 con 0x562921421660 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562921329290 con 0x5629214271c0 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64ca00e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56292137d300 con 0x562921422050 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.068+0000 7f64b67fc640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 0x7f64ac0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41460/0 (socket says 172.21.15.69:41460) 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 -- 172.21.15.69:0/126243628 learned_addr learned my addr 172.21.15.69:0/126243628 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:03.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 0x7f64ac0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64b67fc640 1 -- 172.21.15.69:0/2099468298 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f64bc0cf420 con 0x7f64c00fff30 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c50a5640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921426b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c48a4640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921429650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 -- 172.21.15.69:0/126243628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 msgr2=0x562921426b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921426b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 -- 172.21.15.69:0/126243628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 msgr2=0x562921429650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921429650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 -- 172.21.15.69:0/126243628 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56292134d8e0 con 0x562921421660 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c48a4640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921429650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f64b8005cc0 tx=0x7f64b80317e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8b9f2beccfb5702 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64c58a6640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 0x7f64ac0a50f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f64b8031c10 tx=0x7f64b8048000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64ab7fe640 1 -- 172.21.15.69:0/126243628 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b804e030 con 0x562921421660 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64ab7fe640 1 -- 172.21.15.69:0/126243628 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b8036040 con 0x562921421660 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.069+0000 7f64ab7fe640 1 -- 172.21.15.69:0/126243628 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8003970 con 0x562921421660 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- 172.21.15.69:0/126243628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 msgr2=0x562921421a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f64b8005cc0 tx=0x7f64b80317e0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- 172.21.15.69:0/126243628 shutdown_connections 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921429650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921426b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 --2- 172.21.15.69:0/126243628 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921421a80 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- 172.21.15.69:0/126243628 >> 172.21.15.69:0/126243628 conn(0x56292141ccd0 msgr2=0x56292141d0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- 172.21.15.69:0/126243628 shutdown_connections 2024-05-09T09:21:03.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- 172.21.15.69:0/126243628 wait complete. 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.070+0000 7f64ca00e740 1 -- start start 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 Processor -- start 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5629214271c0 0x562921432da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56292142a020 con 0x562921422050 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56292133c790 con 0x562921421660 2024-05-09T09:21:03.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56292142a1a0 con 0x5629214271c0 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64ca00e740 10 client.0 _ll_register_callbacks cb 0x562921377a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41468/0 (socket says 172.21.15.69:41468) 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 -- 172.21.15.69:0/4028529185 learned_addr learned my addr 172.21.15.69:0/4028529185 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c50a5640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 -- 172.21.15.69:0/4028529185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5629214271c0 msgr2=0x562921432da0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:21:03.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5629214271c0 0x562921432da0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 -- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 msgr2=0x56292142b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.071+0000 7f64c48a4640 1 -- 172.21.15.69:0/4028529185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562921095210 con 0x562921422050 2024-05-09T09:21:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64c48a4640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f64b000be20 tx=0x7f64b0012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f8d3b1d8414e9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b0005220 con 0x562921422050 2024-05-09T09:21:03.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f64b00053c0 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64b8036040 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64b0025050 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 10 client.16200 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562921161620 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b8036040 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f64b0019030 con 0x562921422050 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 10 client.16200 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 10 client.16200 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:03.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 20 client.16200 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562921437480 0x562921439920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 4 client.16200 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.072+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56292138ea70 con 0x562921437480 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.073+0000 7f64c50a5640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562921437480 0x562921439920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.073+0000 7f64a8ff9640 10 client.16200 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.073+0000 7f64c50a5640 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562921437480 0x562921439920 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b8079ab2965ed90e server_cookie=779d837cadd2bc13 in_seq=0 out_seq=0 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64a8ff9640 1 -- 172.21.15.69:0/4028529185 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56292138ea70 con 0x562921437480 2024-05-09T09:21:03.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64a8ff9640 10 client.16200 handle_command_reply: tid=0 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 client.16200 shutdown 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 msgr2=0x56292142bd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f64b000be20 tx=0x7f64b0012820 comp rx=0 tx=0).stop 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 shutdown_connections 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562921437480 0x562921439920 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5629214271c0 0x562921432da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921422050 0x56292142bd60 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 --2- 172.21.15.69:0/4028529185 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64a8ff9640 0 client.16200 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 >> 172.21.15.69:0/4028529185 conn(0x56292141ccd0 msgr2=0x562921417a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 shutdown_connections 2024-05-09T09:21:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 1 -- 172.21.15.69:0/4028529185 wait complete. 2024-05-09T09:21:03.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.074+0000 7f64ca00e740 20 client.16200 trim_cache size 0 max 16384 2024-05-09T09:21:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 Processor -- start 2024-05-09T09:21:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 -- start start 2024-05-09T09:21:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921432d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01aab20 con 0x5629214271c0 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01a8390 con 0x562921421660 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64ca00e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01a61e0 con 0x562921422050 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41484/0 (socket says 172.21.15.69:41484) 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 -- 172.21.15.69:0/3825953284 learned_addr learned my addr 172.21.15.69:0/3825953284 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:03.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 -- 172.21.15.69:0/3825953284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 msgr2=0x562921432d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921432d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 -- 172.21.15.69:0/3825953284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 msgr2=0x56292142ae40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 -- 172.21.15.69:0/3825953284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562921416f00 con 0x5629214271c0 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64c58a6640 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f64b80316b0 tx=0x7f64b8005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a25c68b6385234c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.089+0000 7f64aaffd640 1 -- 172.21.15.69:0/3825953284 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b8058070 con 0x5629214271c0 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64aaffd640 1 -- 172.21.15.69:0/3825953284 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b8036040 con 0x5629214271c0 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64aaffd640 1 -- 172.21.15.69:0/3825953284 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64b804e040 con 0x5629214271c0 2024-05-09T09:21:03.100 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 -- 172.21.15.69:0/3825953284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 msgr2=0x5629214295b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f64b80316b0 tx=0x7f64b8005f80 comp rx=0 tx=0).stop 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 -- 172.21.15.69:0/3825953284 shutdown_connections 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5629214271c0 0x5629214295b0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921432d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3825953284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562921421660 0x56292142ae40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 -- 172.21.15.69:0/3825953284 >> 172.21.15.69:0/3825953284 conn(0x56292141ccd0 msgr2=0x5629214268a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 -- 172.21.15.69:0/3825953284 shutdown_connections 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.090+0000 7f64ca00e740 1 -- 172.21.15.69:0/3825953284 wait complete. 2024-05-09T09:21:03.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 -- start start 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 Processor -- start 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921430640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921431650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01a6850 con 0x562921421660 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01a7cc0 con 0x5629214271c0 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f64c01a6f20 con 0x562921422050 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64ca00e740 10 client.0 _ll_register_callbacks cb 0x7f64b0005850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:03.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.091+0000 7f64c50a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41490/0 (socket says 172.21.15.69:41490) 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 -- 172.21.15.69:0/3984281894 learned_addr learned my addr 172.21.15.69:0/3984281894 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 -- 172.21.15.69:0/3984281894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 msgr2=0x562921430640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921430640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 -- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 msgr2=0x562921431650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c58a6640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921431650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921431650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 -- 172.21.15.69:0/3984281894 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562921431c40 con 0x562921421660 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64c50a5640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f64bc000c00 tx=0x7f64bc0054a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be42cc4e2dde0a8b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64a8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f64bc01d040 con 0x562921421660 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f64bc0ba2b0 con 0x562921421660 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f64b8036040 con 0x562921421660 2024-05-09T09:21:03.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f64bc081050 con 0x562921421660 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64ca00e740 10 client.16206 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x562921433020 con 0x562921421660 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.092+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f64b8036040 con 0x562921421660 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f64bc007ca0 con 0x562921421660 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 10 client.16206 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 10 client.16206 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 20 client.16206 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56292149e250 0x5629214a0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 4 client.16206 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562921388dc0 con 0x56292149e250 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64c48a4640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56292149e250 0x5629214a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64a8ff9640 10 client.16206 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:03.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.093+0000 7f64c48a4640 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56292149e250 0x5629214a0670 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=30a9a9cee2f0b80d server_cookie=601e8ee16bf41ae0 in_seq=0 out_seq=0 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64a8ff9640 1 -- 172.21.15.69:0/3984281894 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562921388dc0 con 0x56292149e250 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64a8ff9640 10 client.16206 handle_command_reply: tid=0 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 client.16206 shutdown 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 msgr2=0x562921430100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f64bc000c00 tx=0x7f64bc0054a0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 shutdown_connections 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5629214271c0 0x562921431650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56292149e250 0x5629214a0670 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562921422050 0x562921430640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 --2- 172.21.15.69:0/3984281894 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562921421660 0x562921430100 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64a8ff9640 0 client.16206 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:03.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 >> 172.21.15.69:0/3984281894 conn(0x56292141ccd0 msgr2=0x562921423270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 shutdown_connections 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 1 -- 172.21.15.69:0/3984281894 wait complete. 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.135+0000 7f64ca00e740 20 client.16206 trim_cache size 0 max 16384 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:21:03.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:21:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 msgr2=0x7f64ac0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 0x7f64ac0a50f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f64b8031c10 tx=0x7f64b8048000 comp rx=0 tx=0).stop 2024-05-09T09:21:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 msgr2=0x7f64c0197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f64bc00c890 tx=0x7f64bc00cd50 comp rx=0 tx=0).stop 2024-05-09T09:21:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 shutdown_connections 2024-05-09T09:21:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f64ac0a2c30 0x7f64ac0a50f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f64c0101a40 0x7f64c019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f64c0101080 0x7f64c0198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 --2- 172.21.15.69:0/2099468298 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f64c00fff30 0x7f64c0197b10 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 >> 172.21.15.69:0/2099468298 conn(0x7f64c00fb4a0 msgr2=0x7f64c00fcbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 shutdown_connections 2024-05-09T09:21:03.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:03.136+0000 7f64c7309640 1 -- 172.21.15.69:0/2099468298 wait complete. 2024-05-09T09:21:03.473 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46814 4.42 MB/sec warmup 61 sec latency 1708.820 ms 2024-05-09T09:21:04.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46814 4.35 MB/sec warmup 62 sec latency 2708.964 ms 2024-05-09T09:21:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:04 smithi069 ceph-mon[30123]: pgmap v431: 129 pgs: 129 active+clean; 28 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:21:04.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:04 smithi163 ceph-mon[34354]: pgmap v431: 129 pgs: 129 active+clean; 28 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:21:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:04 smithi103 ceph-mon[34421]: pgmap v431: 129 pgs: 129 active+clean; 28 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:21:05.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46870 4.28 MB/sec warmup 63 sec latency 3463.798 ms 2024-05-09T09:21:06.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46870 4.22 MB/sec warmup 64 sec latency 1195.755 ms 2024-05-09T09:21:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:06 smithi069 ceph-mon[30123]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 681 KiB/s wr, 95 op/s 2024-05-09T09:21:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:06 smithi163 ceph-mon[34354]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 681 KiB/s wr, 95 op/s 2024-05-09T09:21:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:06 smithi103 ceph-mon[34421]: pgmap v432: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 659 KiB/s rd, 681 KiB/s wr, 95 op/s 2024-05-09T09:21:07.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46870 4.15 MB/sec warmup 65 sec latency 2195.845 ms 2024-05-09T09:21:08.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46870 4.09 MB/sec warmup 66 sec latency 3195.892 ms 2024-05-09T09:21:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:08 smithi069 ceph-mon[30123]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 337 KiB/s wr, 80 op/s 2024-05-09T09:21:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:08 smithi163 ceph-mon[34354]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 337 KiB/s wr, 80 op/s 2024-05-09T09:21:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:08 smithi103 ceph-mon[34421]: pgmap v433: 129 pgs: 129 active+clean; 25 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 472 KiB/s rd, 337 KiB/s wr, 80 op/s 2024-05-09T09:21:09.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 46870 4.03 MB/sec warmup 67 sec latency 4196.002 ms 2024-05-09T09:21:10.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 47259 3.99 MB/sec warmup 68 sec latency 4950.567 ms 2024-05-09T09:21:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:10 smithi069 ceph-mon[30123]: pgmap v434: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 225 KiB/s wr, 69 op/s 2024-05-09T09:21:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:10 smithi163 ceph-mon[34354]: pgmap v434: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 225 KiB/s wr, 69 op/s 2024-05-09T09:21:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:10 smithi103 ceph-mon[34421]: pgmap v434: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 467 KiB/s rd, 225 KiB/s wr, 69 op/s 2024-05-09T09:21:11.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 49916 4.18 MB/sec warmup 69 sec latency 13.521 ms 2024-05-09T09:21:12.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 50411 4.13 MB/sec warmup 70 sec latency 462.197 ms 2024-05-09T09:21:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:12 smithi069 ceph-mon[30123]: pgmap v435: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 201 KiB/s wr, 62 op/s 2024-05-09T09:21:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:12 smithi163 ceph-mon[34354]: pgmap v435: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 201 KiB/s wr, 62 op/s 2024-05-09T09:21:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:12 smithi103 ceph-mon[34421]: pgmap v435: 129 pgs: 129 active+clean; 25 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 201 KiB/s wr, 62 op/s 2024-05-09T09:21:13.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 50411 4.07 MB/sec warmup 71 sec latency 1462.259 ms 2024-05-09T09:21:14.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 50411 4.01 MB/sec warmup 72 sec latency 2462.350 ms 2024-05-09T09:21:14.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:14 smithi069 ceph-mon[30123]: pgmap v436: 129 pgs: 129 active+clean; 26 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 245 KiB/s wr, 74 op/s 2024-05-09T09:21:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:14 smithi163 ceph-mon[34354]: pgmap v436: 129 pgs: 129 active+clean; 26 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 245 KiB/s wr, 74 op/s 2024-05-09T09:21:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:14 smithi103 ceph-mon[34421]: pgmap v436: 129 pgs: 129 active+clean; 26 MiB data, 452 MiB used, 1.0 TiB / 1.0 TiB avail; 747 KiB/s rd, 245 KiB/s wr, 74 op/s 2024-05-09T09:21:15.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 50844 3.98 MB/sec warmup 73 sec latency 3217.505 ms 2024-05-09T09:21:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.226+0000 7f4e316d4640 1 -- 172.21.15.69:0/4079587671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c0ffb60 msgr2=0x7f4e2c0fff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4079587671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c0fff80 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009930 tx=0x7f4e1c02a420 comp rx=0 tx=0).stop 2024-05-09T09:21:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 -- 172.21.15.69:0/4079587671 shutdown_connections 2024-05-09T09:21:16.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4079587671 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c109d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4079587671 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c102b30 0x7f4e2c102fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4079587671 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c0fff80 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 -- 172.21.15.69:0/4079587671 >> 172.21.15.69:0/4079587671 conn(0x7f4e2c0fb490 msgr2=0x7f4e2c0fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 -- 172.21.15.69:0/4079587671 shutdown_connections 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 -- 172.21.15.69:0/4079587671 wait complete. 2024-05-09T09:21:16.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.227+0000 7f4e316d4640 1 Processor -- start 2024-05-09T09:21:16.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 -- start start 2024-05-09T09:21:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c1024a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c102b30 0x7f4e2c100ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10d530 con 0x7f4e2c1034f0 2024-05-09T09:21:16.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10d3b0 con 0x7f4e2c102b30 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e316d4640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c10d6b0 con 0x7f4e2c0ffb60 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45650/0 (socket says 172.21.15.69:45650) 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c102b30 0x7f4e2c100ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 -- 172.21.15.69:0/4289120993 learned_addr learned my addr 172.21.15.69:0/4289120993 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2affd640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c1024a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 -- 172.21.15.69:0/4289120993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c0ffb60 msgr2=0x7f4e2c1024a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c1024a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 -- 172.21.15.69:0/4289120993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c102b30 msgr2=0x7f4e2c100ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c102b30 0x7f4e2c100ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 -- 172.21.15.69:0/4289120993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4e2c1018b0 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e2b7fe640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400eb10 tx=0x7f4e2400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.228+0000 7f4e13fff640 1 -- 172.21.15.69:0/4289120993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24004b20 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.229+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4e2c1a1c90 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.229+0000 7f4e13fff640 1 -- 172.21.15.69:0/4289120993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e24004cc0 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.229+0000 7f4e13fff640 1 -- 172.21.15.69:0/4289120993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e24010760 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.229+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4e2c1a2190 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e13fff640 1 -- 172.21.15.69:0/4289120993 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4e24005020 con 0x7f4e2c1034f0 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 Processor -- start 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 -- start start 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c528cdb0 0x5627c528f240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5627c51eb340 con 0x5627c528cdb0 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5627c5193290 con 0x5627c5287250 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e343e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627c51e7300 con 0x5627c5287c40 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c528cdb0 0x5627c528f240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:42422/0 (socket says 172.21.15.69:42422) 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:41910/0 (socket says 172.21.15.69:41910) 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.231+0000 7f4e2affd640 1 -- 172.21.15.69:0/3637592101 learned_addr learned my addr 172.21.15.69:0/3637592101 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:16.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 -- 172.21.15.69:0/3637592101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 msgr2=0x5627c5287670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e13fff640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 0x7f4e140a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 -- 172.21.15.69:0/3637592101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c528cdb0 msgr2=0x5627c528f240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c528cdb0 0x5627c528f240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 -- 172.21.15.69:0/3637592101 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c51b78e0 con 0x5627c5287c40 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2b7fe640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 0x7f4e140a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009900 tx=0x7f4e1c032bf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf7fbf1db9bf7e2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e13fff640 1 -- 172.21.15.69:0/4289120993 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f4e240cf9b0 con 0x7f4e2c1034f0 2024-05-09T09:21:16.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e117fa640 1 -- 172.21.15.69:0/3637592101 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c040070 con 0x5627c5287c40 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e117fa640 1 -- 172.21.15.69:0/3637592101 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e1c004030 con 0x5627c5287c40 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.232+0000 7f4e117fa640 1 -- 172.21.15.69:0/3637592101 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c035040 con 0x5627c5287c40 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- 172.21.15.69:0/3637592101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 msgr2=0x5627c528c740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e2b7fe640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 0x7f4e140a50f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e2409bb90 tx=0x7f4e2406bd40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c009900 tx=0x7f4e1c032bf0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- 172.21.15.69:0/3637592101 shutdown_connections 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c528cdb0 0x5627c528f240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c528c740 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3637592101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5287670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- 172.21.15.69:0/3637592101 >> 172.21.15.69:0/3637592101 conn(0x5627c52828c0 msgr2=0x5627c5282cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- 172.21.15.69:0/3637592101 shutdown_connections 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- 172.21.15.69:0/3637592101 wait complete. 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 -- start start 2024-05-09T09:21:16.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 Processor -- start 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5291460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.233+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 0x5627c52989e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5627c528fd00 con 0x5627c5287c40 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5627c51a6790 con 0x5627c5287250 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5627c528fe80 con 0x5627c528cdb0 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 10 client.0 _ll_register_callbacks cb 0x5627c51e1a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45662/0 (socket says 172.21.15.69:45662) 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 -- 172.21.15.69:0/3152058388 learned_addr learned my addr 172.21.15.69:0/3152058388 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5291460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 0x5627c52989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 -- 172.21.15.69:0/3152058388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 msgr2=0x5627c52989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 0x5627c52989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 -- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 msgr2=0x5627c5291460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5291460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 -- 172.21.15.69:0/3152058388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c4eff210 con 0x5627c5287c40 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 0x5627c52989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c02a940 tx=0x7f4e1c009520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36f408f843227d00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e1c04c070 con 0x5627c5287c40 2024-05-09T09:21:16.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f4e1c0316b0 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e1c004030 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4e1c058050 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 10 client.16218 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.234+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5627c4fcb620 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e1c004030 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4e1c004780 con 0x5627c5287c40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 10 client.16218 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 10 client.16218 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 20 client.16218 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c529cb40 0x5627c529ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 4 client.16218 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5627c51f8a70 con 0x5627c529cb40 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c529cb40 0x5627c529ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e0eff5640 10 client.16218 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.235+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c529cb40 0x5627c529ef70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=167826677a62691e server_cookie=bf29517966a58ae5 in_seq=0 out_seq=0 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e0eff5640 1 -- 172.21.15.69:0/3152058388 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5627c51f8a70 con 0x5627c529cb40 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e0eff5640 10 client.16218 handle_command_reply: tid=0 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 client.16218 shutdown 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 msgr2=0x5627c52919a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f4e1c02a940 tx=0x7f4e1c009520 comp rx=0 tx=0).stop 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 shutdown_connections 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c529cb40 0x5627c529ef70 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c528cdb0 0x5627c52989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287c40 0x5627c52919a0 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3152058388 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c5287250 0x5627c5291460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e0eff5640 0 client.16218 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:16.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 >> 172.21.15.69:0/3152058388 conn(0x5627c52828c0 msgr2=0x5627c5281a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 shutdown_connections 2024-05-09T09:21:16.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 1 -- 172.21.15.69:0/3152058388 wait complete. 2024-05-09T09:21:16.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.236+0000 7f4e343e4740 20 client.16218 trim_cache size 0 max 16384 2024-05-09T09:21:16.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 Processor -- start 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 -- start start 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c52989a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c52a3830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1aef70 con 0x5627c5287250 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1ac7e0 con 0x5627c528cdb0 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e343e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1aa630 con 0x5627c5287c40 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45678/0 (socket says 172.21.15.69:45678) 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/3763705407 learned_addr learned my addr 172.21.15.69:0/3763705407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:16.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2affd640 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c52989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c52a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/3763705407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 msgr2=0x5627c52989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c52989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/3763705407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 msgr2=0x5627c52a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c52a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/3763705407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5627c5283630 con 0x5627c5287250 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.255+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000bdf0 tx=0x7f4e200046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ed8b1b2cd925809 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e10ff9640 1 -- 172.21.15.69:0/3763705407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20012d30 con 0x5627c5287250 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e10ff9640 1 -- 172.21.15.69:0/3763705407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e20004c10 con 0x5627c5287250 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e10ff9640 1 -- 172.21.15.69:0/3763705407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e200246a0 con 0x5627c5287250 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 -- 172.21.15.69:0/3763705407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 msgr2=0x5627c5285f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7f4e2000bdf0 tx=0x7f4e200046c0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 -- 172.21.15.69:0/3763705407 shutdown_connections 2024-05-09T09:21:16.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c52a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c52989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 --2- 172.21.15.69:0/3763705407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5285f50 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 -- 172.21.15.69:0/3763705407 >> 172.21.15.69:0/3763705407 conn(0x5627c52828c0 msgr2=0x5627c5285840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 -- 172.21.15.69:0/3763705407 shutdown_connections 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.256+0000 7f4e343e4740 1 -- 172.21.15.69:0/3763705407 wait complete. 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 -- start start 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 Processor -- start 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c5296750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c5297710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1aaca0 con 0x5627c5287250 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1ac110 con 0x5627c528cdb0 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4e2c1ab370 con 0x5627c5287c40 2024-05-09T09:21:16.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e343e4740 10 client.0 _ll_register_callbacks cb 0x7f4e2000be20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:45688/0 (socket says 172.21.15.69:45688) 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/841309206 learned_addr learned my addr 172.21.15.69:0/841309206 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2affd640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c5296750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c5297710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/841309206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 msgr2=0x5627c5296750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c5296750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 msgr2=0x5627c5297710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c5297710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 -- 172.21.15.69:0/841309206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5627c5297d00 con 0x5627c5287250 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2affd640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c5296750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:21:16.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e2a7fc640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4e20026010 tx=0x7f4e20012710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7393bcf9826a0b6b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e0eff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.257+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4e20036030 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f4e20004730 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4e20004c10 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f4e20004930 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 10 client.16224 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5627c53031a0 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4e20004c10 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f4e2002f880 con 0x5627c5287250 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 10 client.16224 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 10 client.16224 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 20 client.16224 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c5303b20 0x5627c5305f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:16.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 4 client.16224 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:21:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5627c51f2dc0 con 0x5627c5303b20 2024-05-09T09:21:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e2affd640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c5303b20 0x5627c5305f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e0eff5640 10 client.16224 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:16.268 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.258+0000 7f4e2affd640 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c5303b20 0x5627c5305f40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17ef77f6759c2af5 server_cookie=adee31b1aa4af168 in_seq=0 out_seq=0 2024-05-09T09:21:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e0eff5640 1 -- 172.21.15.69:0/841309206 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5627c51f2dc0 con 0x5627c5303b20 2024-05-09T09:21:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e0eff5640 10 client.16224 handle_command_reply: tid=0 2024-05-09T09:21:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 client.16224 shutdown 2024-05-09T09:21:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 msgr2=0x5627c5296210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7f4e20026010 tx=0x7f4e20012710 comp rx=0 tx=0).stop 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 shutdown_connections 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5627c528cdb0 0x5627c5297710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5627c5303b20 0x5627c5305f40 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5627c5287c40 0x5627c5296750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e0eff5640 0 client.16224 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 --2- 172.21.15.69:0/841309206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5627c5287250 0x5627c5296210 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.302+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 >> 172.21.15.69:0/841309206 conn(0x5627c52828c0 msgr2=0x5627c5280f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 shutdown_connections 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e343e4740 1 -- 172.21.15.69:0/841309206 wait complete. 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e343e4740 20 client.16224 trim_cache size 0 max 16384 2024-05-09T09:21:16.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:21:16.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:21:16.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:21:16.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:21:16.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 msgr2=0x7f4e140a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 0x7f4e140a50f0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f4e2409bb90 tx=0x7f4e2406bd40 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 msgr2=0x7f4e2c101020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f4e2400eb10 tx=0x7f4e2400c2d0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 shutdown_connections 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4e140a2c30 0x7f4e140a50f0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4e2c1034f0 0x7f4e2c101020 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4e2c102b30 0x7f4e2c100ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 --2- 172.21.15.69:0/4289120993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4e2c0ffb60 0x7f4e2c1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 >> 172.21.15.69:0/4289120993 conn(0x7f4e2c0fb490 msgr2=0x7f4e2c105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 shutdown_connections 2024-05-09T09:21:16.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:16.303+0000 7f4e316d4640 1 -- 172.21.15.69:0/4289120993 wait complete. 2024-05-09T09:21:16.474 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53514 4.16 MB/sec warmup 74 sec latency 37.377 ms 2024-05-09T09:21:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:16 smithi069 ceph-mon[30123]: pgmap v437: 129 pgs: 129 active+clean; 26 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 215 KiB/s wr, 64 op/s 2024-05-09T09:21:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:16 smithi163 ceph-mon[34354]: pgmap v437: 129 pgs: 129 active+clean; 26 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 215 KiB/s wr, 64 op/s 2024-05-09T09:21:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:16 smithi103 ceph-mon[34421]: pgmap v437: 129 pgs: 129 active+clean; 26 MiB data, 466 MiB used, 1.0 TiB / 1.0 TiB avail; 712 KiB/s rd, 215 KiB/s wr, 64 op/s 2024-05-09T09:21:17.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53903 4.11 MB/sec warmup 75 sec latency 588.151 ms 2024-05-09T09:21:18.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53903 4.05 MB/sec warmup 76 sec latency 1588.225 ms 2024-05-09T09:21:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:18 smithi069 ceph-mon[30123]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 285 KiB/s wr, 78 op/s 2024-05-09T09:21:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:18 smithi163 ceph-mon[34354]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 285 KiB/s wr, 78 op/s 2024-05-09T09:21:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:18 smithi103 ceph-mon[34421]: pgmap v438: 129 pgs: 129 active+clean; 27 MiB data, 471 MiB used, 1.0 TiB / 1.0 TiB avail; 709 KiB/s rd, 285 KiB/s wr, 78 op/s 2024-05-09T09:21:19.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53903 4.00 MB/sec warmup 77 sec latency 2588.262 ms 2024-05-09T09:21:20.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53959 3.95 MB/sec warmup 78 sec latency 3343.177 ms 2024-05-09T09:21:20.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:20 smithi069 ceph-mon[30123]: pgmap v439: 129 pgs: 129 active+clean; 28 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 298 KiB/s wr, 78 op/s 2024-05-09T09:21:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:20 smithi163 ceph-mon[34354]: pgmap v439: 129 pgs: 129 active+clean; 28 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 298 KiB/s wr, 78 op/s 2024-05-09T09:21:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:20 smithi103 ceph-mon[34421]: pgmap v439: 129 pgs: 129 active+clean; 28 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 298 KiB/s wr, 78 op/s 2024-05-09T09:21:21.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 53959 3.90 MB/sec warmup 79 sec latency 1192.409 ms 2024-05-09T09:21:22.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 54027 3.85 MB/sec warmup 80 sec latency 1783.981 ms 2024-05-09T09:21:22.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:22 smithi069 ceph-mon[30123]: pgmap v440: 129 pgs: 129 active+clean; 29 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 362 KiB/s wr, 100 op/s 2024-05-09T09:21:22.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:22 smithi163 ceph-mon[34354]: pgmap v440: 129 pgs: 129 active+clean; 29 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 362 KiB/s wr, 100 op/s 2024-05-09T09:21:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:22 smithi103 ceph-mon[34421]: pgmap v440: 129 pgs: 129 active+clean; 29 MiB data, 478 MiB used, 1.0 TiB / 1.0 TiB avail; 1011 KiB/s rd, 362 KiB/s wr, 100 op/s 2024-05-09T09:21:23.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 54027 3.80 MB/sec warmup 81 sec latency 1335.781 ms 2024-05-09T09:21:24.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 54027 3.76 MB/sec warmup 82 sec latency 2335.884 ms 2024-05-09T09:21:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:24 smithi069 ceph-mon[30123]: pgmap v441: 129 pgs: 129 active+clean; 31 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 623 KiB/s wr, 110 op/s 2024-05-09T09:21:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:24 smithi163 ceph-mon[34354]: pgmap v441: 129 pgs: 129 active+clean; 31 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 623 KiB/s wr, 110 op/s 2024-05-09T09:21:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:24 smithi103 ceph-mon[34421]: pgmap v441: 129 pgs: 129 active+clean; 31 MiB data, 482 MiB used, 1.0 TiB / 1.0 TiB avail; 999 KiB/s rd, 623 KiB/s wr, 110 op/s 2024-05-09T09:21:25.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 54499 3.73 MB/sec warmup 83 sec latency 3093.951 ms 2024-05-09T09:21:26.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57147 3.89 MB/sec warmup 84 sec latency 11.621 ms 2024-05-09T09:21:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:26 smithi069 ceph-mon[30123]: pgmap v442: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T09:21:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:26 smithi163 ceph-mon[34354]: pgmap v442: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T09:21:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:26 smithi103 ceph-mon[34421]: pgmap v442: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 717 KiB/s rd, 1.2 MiB/s wr, 121 op/s 2024-05-09T09:21:27.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57447 3.85 MB/sec warmup 85 sec latency 668.604 ms 2024-05-09T09:21:28.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57447 3.80 MB/sec warmup 86 sec latency 1668.724 ms 2024-05-09T09:21:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:28 smithi163 ceph-mon[34354]: pgmap v443: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-05-09T09:21:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:28 smithi103 ceph-mon[34421]: pgmap v443: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-05-09T09:21:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:28 smithi069 ceph-mon[30123]: pgmap v443: 129 pgs: 129 active+clean; 30 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.3 MiB/s wr, 137 op/s 2024-05-09T09:21:29.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 -- 172.21.15.69:0/1577819015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c0ffb70 msgr2=0x7f926c0fff90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 --2- 172.21.15.69:0/1577819015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c0ffb70 0x7f926c0fff90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7f9260009930 tx=0x7f926002a420 comp rx=0 tx=0).stop 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 -- 172.21.15.69:0/1577819015 shutdown_connections 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 --2- 172.21.15.69:0/1577819015 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c103500 0x7f926c109d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 --2- 172.21.15.69:0/1577819015 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c102b40 0x7f926c102fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 --2- 172.21.15.69:0/1577819015 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c0ffb70 0x7f926c0fff90 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 -- 172.21.15.69:0/1577819015 >> 172.21.15.69:0/1577819015 conn(0x7f926c0fb4a0 msgr2=0x7f926c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 -- 172.21.15.69:0/1577819015 shutdown_connections 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.399+0000 7f9272c9e640 1 -- 172.21.15.69:0/1577819015 wait complete. 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 Processor -- start 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 -- start start 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c0ffb70 0x7f926c197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c102b40 0x7f926c1980c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926c10d5a0 con 0x7f926c103500 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f926c10d420 con 0x7f926c0ffb70 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9272c9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f926c10d720 con 0x7f926c102b40 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41838/0 (socket says 172.21.15.69:41838) 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 -- 172.21.15.69:0/2993757701 learned_addr learned my addr 172.21.15.69:0/2993757701 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f9270a3a640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c0ffb70 0x7f926c197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f926bfff640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c102b40 0x7f926c1980c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 -- 172.21.15.69:0/2993757701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c102b40 msgr2=0x7f926c1980c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c102b40 0x7f926c1980c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 -- 172.21.15.69:0/2993757701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c0ffb70 msgr2=0x7f926c197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c0ffb70 0x7f926c197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.400+0000 7f927123b640 1 -- 172.21.15.69:0/2993757701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f926c19f7a0 con 0x7f926c103500 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f927123b640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f926400b970 tx=0x7f926400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f9269ffb640 1 -- 172.21.15.69:0/2993757701 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264019070 con 0x7f926c103500 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f9269ffb640 1 -- 172.21.15.69:0/2993757701 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f926400cb40 con 0x7f926c103500 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f9269ffb640 1 -- 172.21.15.69:0/2993757701 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9264007650 con 0x7f926c103500 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f926c19fa90 con 0x7f926c103500 2024-05-09T09:21:29.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.401+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f926c19ffe0 con 0x7f926c103500 2024-05-09T09:21:29.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.403+0000 7f9269ffb640 1 -- 172.21.15.69:0/2993757701 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f926401c070 con 0x7f926c103500 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 Processor -- start 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 -- start start 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f9269ffb640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 0x7f925c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 0x55f71d359a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d361650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f71d2b9340 con 0x55f71d35a050 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f71d261290 con 0x55f71d359660 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f92759ad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f71d2b5300 con 0x55f71d35f1c0 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 0x7f925c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f9269ffb640 1 -- 172.21.15.69:0/2993757701 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f92640ce920 con 0x7f926c103500 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41848/0 (socket says 172.21.15.69:41848) 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 -- 172.21.15.69:0/674289054 learned_addr learned my addr 172.21.15.69:0/674289054 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f9270a3a640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 0x55f71d359a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f927123b640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d361650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 -- 172.21.15.69:0/674289054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 msgr2=0x55f71d361650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d361650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 -- 172.21.15.69:0/674289054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 msgr2=0x55f71d359a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 0x55f71d359a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f926bfff640 1 -- 172.21.15.69:0/674289054 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f71d2858e0 con 0x55f71d35a050 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.404+0000 7f9270a3a640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 0x55f71d359a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f926bfff640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 0x7f925c0a50f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9258012940 tx=0x7f9258009ae0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f926bfff640 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9258012fd0 tx=0x7f925801c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12a1dc4cb6856a06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f9256ffd640 1 -- 172.21.15.69:0/674289054 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9258024070 con 0x55f71d35a050 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f9256ffd640 1 -- 172.21.15.69:0/674289054 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258012580 con 0x55f71d35a050 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f9256ffd640 1 -- 172.21.15.69:0/674289054 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92580209c0 con 0x55f71d35a050 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f92759ad740 1 -- 172.21.15.69:0/674289054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 msgr2=0x55f71d35eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.405+0000 7f92759ad740 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7f9258012fd0 tx=0x7f925801c000 comp rx=0 tx=0).stop 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- 172.21.15.69:0/674289054 shutdown_connections 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d361650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d35a050 0x55f71d35eb50 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- 172.21.15.69:0/674289054 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d359660 0x55f71d359a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- 172.21.15.69:0/674289054 >> 172.21.15.69:0/674289054 conn(0x55f71d3508c0 msgr2=0x55f71d350cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- 172.21.15.69:0/674289054 shutdown_connections 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- 172.21.15.69:0/674289054 wait complete. 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- start start 2024-05-09T09:21:29.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 Processor -- start 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d363d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d36adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.406+0000 7f92759ad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f71d362020 con 0x55f71d359660 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f71d274790 con 0x55f71d35a050 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f71d3621a0 con 0x55f71d35f1c0 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 10 client.0 _ll_register_callbacks cb 0x55f71d2afa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41860/0 (socket says 172.21.15.69:41860) 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 -- 172.21.15.69:0/3411483114 learned_addr learned my addr 172.21.15.69:0/3411483114 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f927123b640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d36adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f926bfff640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d363d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 -- 172.21.15.69:0/3411483114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 msgr2=0x55f71d36adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d36adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 -- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 msgr2=0x55f71d363d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d363d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 -- 172.21.15.69:0/3411483114 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71cfcd210 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f9270a3a640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9260009900 tx=0x7f9260032f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=476622a3e7413f3e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f924bfff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260035040 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f9260004040 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9258012580 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9260004240 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 10 client.16236 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.407+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f71d099620 con 0x55f71d359660 2024-05-09T09:21:29.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9258012580 con 0x55f71d359660 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9260004b80 con 0x55f71d359660 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 10 client.16236 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 10 client.16236 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 20 client.16236 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d36f320 0x55f71d3717e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 4 client.16236 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f71d2c6a70 con 0x55f71d36f320 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f926bfff640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d36f320 0x55f71d3717e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f926bfff640 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d36f320 0x55f71d3717e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2028c3d427febb91 server_cookie=d478cf85bd20f92b in_seq=0 out_seq=0 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.408+0000 7f924bfff640 10 client.16236 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f924bfff640 1 -- 172.21.15.69:0/3411483114 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f71d2c6a70 con 0x55f71d36f320 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f924bfff640 10 client.16236 handle_command_reply: tid=0 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 client.16236 shutdown 2024-05-09T09:21:29.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 msgr2=0x55f71d363830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7f9260009900 tx=0x7f9260032f90 comp rx=0 tx=0).stop 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 shutdown_connections 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d36f320 0x55f71d3717e0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d36adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d363d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f924bfff640 0 client.16236 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 --2- 172.21.15.69:0/3411483114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d363830 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.409+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 >> 172.21.15.69:0/3411483114 conn(0x55f71d3508c0 msgr2=0x55f71d34fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.410+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 shutdown_connections 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.410+0000 7f92759ad740 1 -- 172.21.15.69:0/3411483114 wait complete. 2024-05-09T09:21:29.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.410+0000 7f92759ad740 20 client.16236 trim_cache size 0 max 16384 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 Processor -- start 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 -- start start 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d36ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d363390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1aaac0 con 0x55f71d359660 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1a8330 con 0x55f71d35a050 2024-05-09T09:21:29.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f92759ad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1a6180 con 0x55f71d35f1c0 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.424+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41870/0 (socket says 172.21.15.69:41870) 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 -- 172.21.15.69:0/440721038 learned_addr learned my addr 172.21.15.69:0/440721038 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 -- 172.21.15.69:0/440721038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 msgr2=0x55f71d363390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d363390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 -- 172.21.15.69:0/440721038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 msgr2=0x55f71d36ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d36ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 -- 172.21.15.69:0/440721038 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f71d362320 con 0x55f71d359660 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f9270a3a640 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f92600314e0 tx=0x7f92600095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31577566045bd765 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f92567fc640 1 -- 172.21.15.69:0/440721038 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260035030 con 0x55f71d359660 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f92567fc640 1 -- 172.21.15.69:0/440721038 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9260041070 con 0x55f71d359660 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.425+0000 7f92567fc640 1 -- 172.21.15.69:0/440721038 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9260004a70 con 0x55f71d359660 2024-05-09T09:21:29.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- 172.21.15.69:0/440721038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 msgr2=0x55f71d362e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7f92600314e0 tx=0x7f92600095d0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- 172.21.15.69:0/440721038 shutdown_connections 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d363390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d36ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- 172.21.15.69:0/440721038 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d362e50 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- 172.21.15.69:0/440721038 >> 172.21.15.69:0/440721038 conn(0x55f71d3508c0 msgr2=0x55f71d353ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- 172.21.15.69:0/440721038 shutdown_connections 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- 172.21.15.69:0/440721038 wait complete. 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- start start 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 Processor -- start 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d368950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d3698f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1a67f0 con 0x55f71d359660 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1a7c60 con 0x55f71d35a050 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f926c1a6ec0 con 0x55f71d35f1c0 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.426+0000 7f92759ad740 10 client.0 _ll_register_callbacks cb 0x7f9260009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:41878/0 (socket says 172.21.15.69:41878) 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 -- 172.21.15.69:0/869535117 learned_addr learned my addr 172.21.15.69:0/869535117 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f927123b640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d3698f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 -- 172.21.15.69:0/869535117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 msgr2=0x55f71d3698f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d3698f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 -- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 msgr2=0x55f71d368950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d368950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 -- 172.21.15.69:0/869535117 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f71d369ee0 con 0x55f71d359660 2024-05-09T09:21:29.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f9270a3a640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9260006010 tx=0x7f92600041a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15853495626a0b71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f924bfff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f92600047f0 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f9260004990 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9260041070 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f9260069050 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f92759ad740 10 client.16242 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.427+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x55f71d3d5960 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9260041070 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f9260046460 con 0x55f71d359660 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 10 client.16242 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 10 client.16242 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 20 client.16242 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:29.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d3d6220 0x55f71d3d8640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 4 client.16242 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:21:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f71d2c0dc0 con 0x55f71d3d6220 2024-05-09T09:21:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f926bfff640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d3d6220 0x55f71d3d8640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f926bfff640 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d3d6220 0x55f71d3d8640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1bb0d4217daad43d server_cookie=e2c1d018b9866b50 in_seq=0 out_seq=0 2024-05-09T09:21:29.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.428+0000 7f924bfff640 10 client.16242 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f924bfff640 1 -- 172.21.15.69:0/869535117 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f71d2c0dc0 con 0x55f71d3d6220 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f924bfff640 10 client.16242 handle_command_reply: tid=0 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 client.16242 shutdown 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 msgr2=0x55f71d368410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7f9260006010 tx=0x7f92600041a0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 shutdown_connections 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f71d3d6220 0x55f71d3d8640 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f71d35f1c0 0x55f71d3698f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f924bfff640 0 client.16242 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f71d35a050 0x55f71d368950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 --2- 172.21.15.69:0/869535117 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f71d359660 0x55f71d368410 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 >> 172.21.15.69:0/869535117 conn(0x55f71d3508c0 msgr2=0x55f71d34f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 shutdown_connections 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 1 -- 172.21.15.69:0/869535117 wait complete. 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.468+0000 7f92759ad740 20 client.16242 trim_cache size 0 max 16384 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:21:29.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:21:29.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 msgr2=0x7f925c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 0x7f925c0a50f0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9258012940 tx=0x7f9258009ae0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 msgr2=0x7f926c19f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7f926400b970 tx=0x7f926400be30 comp rx=0 tx=0).stop 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 shutdown_connections 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f925c0a2c30 0x7f925c0a50f0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f926c103500 0x7f926c19f130 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f926c102b40 0x7f926c1980c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 --2- 172.21.15.69:0/2993757701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f926c0ffb70 0x7f926c197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 >> 172.21.15.69:0/2993757701 conn(0x7f926c0fb4a0 msgr2=0x7f926c105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 shutdown_connections 2024-05-09T09:21:29.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:29.469+0000 7f9272c9e640 1 -- 172.21.15.69:0/2993757701 wait complete. 2024-05-09T09:21:29.475 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57447 3.76 MB/sec warmup 87 sec latency 2668.802 ms 2024-05-09T09:21:30.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57503 3.72 MB/sec warmup 88 sec latency 3421.356 ms 2024-05-09T09:21:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:30 smithi163 ceph-mon[34354]: pgmap v444: 129 pgs: 129 active+clean; 28 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-05-09T09:21:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:30 smithi103 ceph-mon[34421]: pgmap v444: 129 pgs: 129 active+clean; 28 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-05-09T09:21:30.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:30 smithi069 ceph-mon[30123]: pgmap v444: 129 pgs: 129 active+clean; 28 MiB data, 492 MiB used, 1.0 TiB / 1.0 TiB avail; 735 KiB/s rd, 1.3 MiB/s wr, 119 op/s 2024-05-09T09:21:31.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57503 3.68 MB/sec warmup 89 sec latency 1199.293 ms 2024-05-09T09:21:32.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57503 3.64 MB/sec warmup 90 sec latency 2199.417 ms 2024-05-09T09:21:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:32 smithi163 ceph-mon[34354]: pgmap v445: 129 pgs: 129 active+clean; 29 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-05-09T09:21:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:32 smithi103 ceph-mon[34421]: pgmap v445: 129 pgs: 129 active+clean; 29 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-05-09T09:21:32.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:32 smithi069 ceph-mon[30123]: pgmap v445: 129 pgs: 129 active+clean; 29 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 548 KiB/s rd, 1.3 MiB/s wr, 121 op/s 2024-05-09T09:21:33.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57503 3.60 MB/sec warmup 91 sec latency 3199.502 ms 2024-05-09T09:21:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:33 smithi103 ceph-mon[34421]: pgmap v446: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-05-09T09:21:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:33 smithi069 ceph-mon[30123]: pgmap v446: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-05-09T09:21:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:33 smithi163 ceph-mon[34354]: pgmap v446: 129 pgs: 129 active+clean; 29 MiB data, 481 MiB used, 1.0 TiB / 1.0 TiB avail; 478 KiB/s rd, 1.2 MiB/s wr, 108 op/s 2024-05-09T09:21:34.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57503 3.56 MB/sec warmup 92 sec latency 4199.612 ms 2024-05-09T09:21:35.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57571 3.52 MB/sec warmup 93 sec latency 4951.983 ms 2024-05-09T09:21:36.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57571 3.48 MB/sec warmup 94 sec latency 1186.019 ms 2024-05-09T09:21:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:36 smithi069 ceph-mon[30123]: pgmap v447: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1019 KiB/s wr, 107 op/s 2024-05-09T09:21:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:36 smithi163 ceph-mon[34354]: pgmap v447: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1019 KiB/s wr, 107 op/s 2024-05-09T09:21:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:36 smithi103 ceph-mon[34421]: pgmap v447: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 470 KiB/s rd, 1019 KiB/s wr, 107 op/s 2024-05-09T09:21:37.476 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57571 3.45 MB/sec warmup 95 sec latency 2186.148 ms 2024-05-09T09:21:38.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57571 3.41 MB/sec warmup 96 sec latency 3186.251 ms 2024-05-09T09:21:38.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:38 smithi069 ceph-mon[30123]: pgmap v448: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 398 KiB/s wr, 82 op/s 2024-05-09T09:21:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:38 smithi163 ceph-mon[34354]: pgmap v448: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 398 KiB/s wr, 82 op/s 2024-05-09T09:21:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:38 smithi103 ceph-mon[34421]: pgmap v448: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 398 KiB/s wr, 82 op/s 2024-05-09T09:21:39.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 57571 3.37 MB/sec warmup 97 sec latency 4186.366 ms 2024-05-09T09:21:40.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 58064 3.36 MB/sec warmup 98 sec latency 4937.368 ms 2024-05-09T09:21:40.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:40 smithi069 ceph-mon[30123]: pgmap v449: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 176 KiB/s wr, 63 op/s 2024-05-09T09:21:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:40 smithi163 ceph-mon[34354]: pgmap v449: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 176 KiB/s wr, 63 op/s 2024-05-09T09:21:40.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:40 smithi103 ceph-mon[34421]: pgmap v449: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 363 KiB/s rd, 176 KiB/s wr, 63 op/s 2024-05-09T09:21:41.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 60712 3.50 MB/sec warmup 99 sec latency 12.124 ms 2024-05-09T09:21:42.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 61119 3.46 MB/sec warmup 100 sec latency 611.248 ms 2024-05-09T09:21:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:42 smithi069 ceph-mon[30123]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 147 KiB/s wr, 54 op/s 2024-05-09T09:21:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:42 smithi163 ceph-mon[34354]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 147 KiB/s wr, 54 op/s 2024-05-09T09:21:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 -- 172.21.15.69:0/1062921990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928101980 msgr2=0x7f2928101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 --2- 172.21.15.69:0/1062921990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928101980 0x7f2928101e00 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7f2910009900 tx=0x7f291002a410 comp rx=0 tx=0).stop 2024-05-09T09:21:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 -- 172.21.15.69:0/1062921990 shutdown_connections 2024-05-09T09:21:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 --2- 172.21.15.69:0/1062921990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928102340 0x7f2928108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 --2- 172.21.15.69:0/1062921990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928101980 0x7f2928101e00 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 --2- 172.21.15.69:0/1062921990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928100790 0x7f2928100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 -- 172.21.15.69:0/1062921990 >> 172.21.15.69:0/1062921990 conn(0x7f29280fbd20 msgr2=0x7f29280fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 -- 172.21.15.69:0/1062921990 shutdown_connections 2024-05-09T09:21:42.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.561+0000 7f292c984640 1 -- 172.21.15.69:0/1062921990 wait complete. 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 Processor -- start 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 -- start start 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928100790 0x7f29281969c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928101980 0x7f2928196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f292810c410 con 0x7f2928102340 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f292810c290 con 0x7f2928100790 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292c984640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f292810c590 con 0x7f2928101980 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50730/0 (socket says 172.21.15.69:50730) 2024-05-09T09:21:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 -- 172.21.15.69:0/404809158 learned_addr learned my addr 172.21.15.69:0/404809158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:42.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f291ffff640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928101980 0x7f2928196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f292659c640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928100790 0x7f29281969c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 -- 172.21.15.69:0/404809158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928101980 msgr2=0x7f2928196f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928101980 0x7f2928196f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 -- 172.21.15.69:0/404809158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928100790 msgr2=0x7f29281969c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928100790 0x7f29281969c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.562+0000 7f2926d9d640 1 -- 172.21.15.69:0/404809158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f292819e5e0 con 0x7f2928102340 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f2926d9d640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f292000eaa0 tx=0x7f292000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f291f7fe640 1 -- 172.21.15.69:0/404809158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f292000cdb0 con 0x7f2928102340 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f291f7fe640 1 -- 172.21.15.69:0/404809158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2920005020 con 0x7f2928102340 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f292819e8d0 con 0x7f2928102340 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f291f7fe640 1 -- 172.21.15.69:0/404809158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29200116b0 con 0x7f2928102340 2024-05-09T09:21:42.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.563+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f292819eda0 con 0x7f2928102340 2024-05-09T09:21:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f291f7fe640 1 -- 172.21.15.69:0/404809158 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2920005560 con 0x7f2928102340 2024-05-09T09:21:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 Processor -- start 2024-05-09T09:21:42.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 -- start start 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e5841670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5849240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5561e57a5340 con 0x5561e5841c40 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5561e574d290 con 0x5561e5841250 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292f768740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5561e57a1300 con 0x5561e5846db0 2024-05-09T09:21:42.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50732/0 (socket says 172.21.15.69:50732) 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.566+0000 7f292659c640 1 -- 172.21.15.69:0/1894448310 learned_addr learned my addr 172.21.15.69:0/1894448310 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291ffff640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e5841670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291f7fe640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 0x7f29140a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291ffff640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5849240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f2926d9d640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 0x7f29140a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 -- 172.21.15.69:0/1894448310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 msgr2=0x5561e5849240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5849240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 -- 172.21.15.69:0/1894448310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 msgr2=0x5561e5841670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e5841670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 -- 172.21.15.69:0/1894448310 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561e57718e0 con 0x5561e5841c40 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291ffff640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5849240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291f7fe640 1 -- 172.21.15.69:0/404809158 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f29200cf590 con 0x7f2928102340 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f292659c640 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2918009ca0 tx=0x7f2918011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b323303b49dd28a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291cff9640 1 -- 172.21.15.69:0/1894448310 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918004220 con 0x5561e5841c40 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f2926d9d640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 0x7f29140a50f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2920097f80 tx=0x7f292006ebe0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291cff9640 1 -- 172.21.15.69:0/1894448310 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29180043c0 con 0x5561e5841c40 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.567+0000 7f291cff9640 1 -- 172.21.15.69:0/1894448310 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918019900 con 0x5561e5841c40 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- 172.21.15.69:0/1894448310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 msgr2=0x5561e5846740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7f2918009ca0 tx=0x7f2918011810 comp rx=0 tx=0).stop 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- 172.21.15.69:0/1894448310 shutdown_connections 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5849240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5846740 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 --2- 172.21.15.69:0/1894448310 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e5841670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- 172.21.15.69:0/1894448310 >> 172.21.15.69:0/1894448310 conn(0x5561e583c8c0 msgr2=0x5561e583ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- 172.21.15.69:0/1894448310 shutdown_connections 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- 172.21.15.69:0/1894448310 wait complete. 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 -- start start 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.568+0000 7f292f768740 1 Processor -- start 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841c40 0x5561e584b950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5852990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5561e5849d60 con 0x5561e5841250 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5561e5760790 con 0x5561e5841c40 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5561e5849ee0 con 0x5561e5846db0 2024-05-09T09:21:42.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 10 client.0 _ll_register_callbacks cb 0x5561e579ba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50746/0 (socket says 172.21.15.69:50746) 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 -- 172.21.15.69:0/2417538855 learned_addr learned my addr 172.21.15.69:0/2417538855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292659c640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841c40 0x5561e584b950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 -- 172.21.15.69:0/2417538855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 msgr2=0x5561e5852990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5852990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 -- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841c40 msgr2=0x5561e584b950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841c40 0x5561e584b950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 -- 172.21.15.69:0/2417538855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561e54b9210 con 0x5561e5841250 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f291ffff640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f291002a930 tx=0x7f2910032ef0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d513d30f3cbf5c09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f290dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2910035070 con 0x5561e5841250 2024-05-09T09:21:42.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f2910004250 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.569+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29180043c0 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2910004450 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 10 client.16254 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5561e5585620 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29180043c0 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f29100035f0 con 0x5561e5841250 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 10 client.16254 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 10 client.16254 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 20 client.16254 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e5859370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 4 client.16254 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561e57b2a70 con 0x5561e5856ee0 2024-05-09T09:21:42.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.570+0000 7f292659c640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e5859370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.571+0000 7f290dffb640 10 client.16254 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.571+0000 7f292659c640 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e5859370 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=11fe82397035653b server_cookie=5ee650cf49eef26c in_seq=0 out_seq=0 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f290dffb640 1 -- 172.21.15.69:0/2417538855 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561e57b2a70 con 0x5561e5856ee0 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f290dffb640 10 client.16254 handle_command_reply: tid=0 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 client.16254 shutdown 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 msgr2=0x5561e584b410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7f291002a930 tx=0x7f2910032ef0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 shutdown_connections 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e5859370 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5852990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841c40 0x5561e584b950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f290dffb640 0 client.16254 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 --2- 172.21.15.69:0/2417538855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841250 0x5561e584b410 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 >> 172.21.15.69:0/2417538855 conn(0x5561e583c8c0 msgr2=0x5561e583ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 shutdown_connections 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 1 -- 172.21.15.69:0/2417538855 wait complete. 2024-05-09T09:21:42.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.572+0000 7f292f768740 20 client.16254 trim_cache size 0 max 16384 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 Processor -- start 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 -- start start 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e584b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5840060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29281aba40 con 0x5561e5841c40 2024-05-09T09:21:42.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29281a92b0 con 0x5561e5841250 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292f768740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29281a7100 con 0x5561e5846db0 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50754/0 (socket says 172.21.15.69:50754) 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 -- 172.21.15.69:0/396147925 learned_addr learned my addr 172.21.15.69:0/396147925 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 -- 172.21.15.69:0/396147925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 msgr2=0x5561e5840060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5840060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 -- 172.21.15.69:0/396147925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 msgr2=0x5561e584b8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e584b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 -- 172.21.15.69:0/396147925 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561e5837bc0 con 0x5561e5841c40 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f292659c640 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f291801b040 tx=0x7f2918005220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f307c474b9ca743f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f290ffff640 1 -- 172.21.15.69:0/396147925 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918005310 con 0x5561e5841c40 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f290ffff640 1 -- 172.21.15.69:0/396147925 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29180054b0 con 0x5561e5841c40 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.587+0000 7f290ffff640 1 -- 172.21.15.69:0/396147925 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f29180195d0 con 0x5561e5841c40 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- 172.21.15.69:0/396147925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 msgr2=0x5561e5852950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7f291801b040 tx=0x7f2918005220 comp rx=0 tx=0).stop 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- 172.21.15.69:0/396147925 shutdown_connections 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5846db0 0x5561e5840060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e5852950 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 --2- 172.21.15.69:0/396147925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5841250 0x5561e584b8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- 172.21.15.69:0/396147925 >> 172.21.15.69:0/396147925 conn(0x5561e583c8c0 msgr2=0x5561e583f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- 172.21.15.69:0/396147925 shutdown_connections 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- 172.21.15.69:0/396147925 wait complete. 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 -- start start 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.588+0000 7f292f768740 1 Processor -- start 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5841250 0x5561e58529c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5846db0 0x5561e58bdab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f29281a7770 con 0x5561e5841c40 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f29281a8be0 con 0x5561e5846db0 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f29281a7e40 con 0x5561e5841250 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 10 client.0 _ll_register_callbacks cb 0x7f2910032c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50760/0 (socket says 172.21.15.69:50760) 2024-05-09T09:21:42.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 -- 172.21.15.69:0/2751228993 learned_addr learned my addr 172.21.15.69:0/2751228993 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f291ffff640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5841250 0x5561e58529c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f291ffff640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5846db0 0x5561e58bdab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 -- 172.21.15.69:0/2751228993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5841250 msgr2=0x5561e58529c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5841250 0x5561e58529c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 -- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5846db0 msgr2=0x5561e58bdab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5846db0 0x5561e58bdab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 -- 172.21.15.69:0/2751228993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561e58be2e0 con 0x5561e5841c40 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292659c640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2918009380 tx=0x7f29180097f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b177876bf6dc2c72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f290dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2918037030 con 0x5561e5841c40 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f2918003d10 con 0x5561e5841c40 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f29180054b0 con 0x5561e5841c40 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.589+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f2918005040 con 0x5561e5841c40 2024-05-09T09:21:42.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 10 client.16260 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5561e58bf5f0 con 0x5561e5841c40 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f29180054b0 con 0x5561e5841c40 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f2918019bc0 con 0x5561e5841c40 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 10 client.16260 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 10 client.16260 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 20 client.16260 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e58c1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 4 client.16260 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561e57acdc0 con 0x5561e5856ee0 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f291ffff640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e58c1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f290dffb640 10 client.16260 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:42.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.590+0000 7f291ffff640 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e58c1de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e447d2bdbd9d9f92 server_cookie=a56384063b55f292 in_seq=0 out_seq=0 2024-05-09T09:21:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:42 smithi103 ceph-mon[34421]: pgmap v450: 129 pgs: 129 active+clean; 29 MiB data, 485 MiB used, 1.0 TiB / 1.0 TiB avail; 100 KiB/s rd, 147 KiB/s wr, 54 op/s 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f290dffb640 1 -- 172.21.15.69:0/2751228993 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561e57acdc0 con 0x5561e5856ee0 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f290dffb640 10 client.16260 handle_command_reply: tid=0 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 client.16260 shutdown 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 msgr2=0x5561e58bb4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7f2918009380 tx=0x7f29180097f0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 shutdown_connections 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561e5846db0 0x5561e58bdab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561e5841c40 0x5561e58bb4e0 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561e5856ee0 0x5561e58c1de0 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 --2- 172.21.15.69:0/2751228993 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561e5841250 0x5561e58529c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f290dffb640 0 client.16260 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:42.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 >> 172.21.15.69:0/2751228993 conn(0x5561e583c8c0 msgr2=0x5561e583f950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 shutdown_connections 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 1 -- 172.21.15.69:0/2751228993 wait complete. 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.631+0000 7f292f768740 20 client.16260 trim_cache size 0 max 16384 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:21:42.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 msgr2=0x7f29140a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 0x7f29140a50f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f2920097f80 tx=0x7f292006ebe0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 msgr2=0x7f292819df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7f292000eaa0 tx=0x7f292000ef60 comp rx=0 tx=0).stop 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 shutdown_connections 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f29140a2c30 0x7f29140a50f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2928102340 0x7f292819df70 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2928101980 0x7f2928196f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 --2- 172.21.15.69:0/404809158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2928100790 0x7f29281969c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 >> 172.21.15.69:0/404809158 conn(0x7f29280fbd20 msgr2=0x7f2928104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 shutdown_connections 2024-05-09T09:21:42.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:42.632+0000 7f292c984640 1 -- 172.21.15.69:0/404809158 wait complete. 2024-05-09T09:21:43.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 61119 3.43 MB/sec warmup 101 sec latency 1611.359 ms 2024-05-09T09:21:44.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 61119 3.40 MB/sec warmup 102 sec latency 2611.434 ms 2024-05-09T09:21:44.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:44 smithi069 ceph-mon[30123]: pgmap v451: 129 pgs: 129 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 115 KiB/s wr, 46 op/s 2024-05-09T09:21:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:44 smithi163 ceph-mon[34354]: pgmap v451: 129 pgs: 129 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 115 KiB/s wr, 46 op/s 2024-05-09T09:21:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:44 smithi103 ceph-mon[34421]: pgmap v451: 129 pgs: 129 active+clean; 30 MiB data, 486 MiB used, 1.0 TiB / 1.0 TiB avail; 227 KiB/s rd, 115 KiB/s wr, 46 op/s 2024-05-09T09:21:45.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 61587 3.38 MB/sec warmup 103 sec latency 3363.039 ms 2024-05-09T09:21:46.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64194 3.51 MB/sec warmup 104 sec latency 18.079 ms 2024-05-09T09:21:46.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:46 smithi069 ceph-mon[30123]: pgmap v452: 129 pgs: 129 active+clean; 31 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 215 KiB/s wr, 61 op/s 2024-05-09T09:21:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:46 smithi163 ceph-mon[34354]: pgmap v452: 129 pgs: 129 active+clean; 31 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 215 KiB/s wr, 61 op/s 2024-05-09T09:21:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:46 smithi103 ceph-mon[34421]: pgmap v452: 129 pgs: 129 active+clean; 31 MiB data, 488 MiB used, 1.0 TiB / 1.0 TiB avail; 421 KiB/s rd, 215 KiB/s wr, 61 op/s 2024-05-09T09:21:47.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64536 3.48 MB/sec warmup 105 sec latency 653.168 ms 2024-05-09T09:21:48.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64536 3.45 MB/sec warmup 106 sec latency 1653.281 ms 2024-05-09T09:21:48.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:48 smithi069 ceph-mon[30123]: pgmap v453: 129 pgs: 129 active+clean; 31 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 237 KiB/s wr, 64 op/s 2024-05-09T09:21:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:48 smithi163 ceph-mon[34354]: pgmap v453: 129 pgs: 129 active+clean; 31 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 237 KiB/s wr, 64 op/s 2024-05-09T09:21:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:48 smithi103 ceph-mon[34421]: pgmap v453: 129 pgs: 129 active+clean; 31 MiB data, 489 MiB used, 1.0 TiB / 1.0 TiB avail; 514 KiB/s rd, 237 KiB/s wr, 64 op/s 2024-05-09T09:21:49.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64536 3.41 MB/sec warmup 107 sec latency 2653.379 ms 2024-05-09T09:21:50.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64592 3.38 MB/sec warmup 108 sec latency 3402.618 ms 2024-05-09T09:21:50.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:50 smithi069 ceph-mon[30123]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 279 KiB/s wr, 76 op/s 2024-05-09T09:21:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:50 smithi163 ceph-mon[34354]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 279 KiB/s wr, 76 op/s 2024-05-09T09:21:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:50 smithi103 ceph-mon[34421]: pgmap v454: 129 pgs: 129 active+clean; 31 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 792 KiB/s rd, 279 KiB/s wr, 76 op/s 2024-05-09T09:21:51.477 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64592 3.35 MB/sec warmup 109 sec latency 1194.065 ms 2024-05-09T09:21:52.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64592 3.32 MB/sec warmup 110 sec latency 2194.207 ms 2024-05-09T09:21:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:52 smithi163 ceph-mon[34354]: pgmap v455: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 346 KiB/s wr, 91 op/s 2024-05-09T09:21:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:52 smithi103 ceph-mon[34421]: pgmap v455: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 346 KiB/s wr, 91 op/s 2024-05-09T09:21:52.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:52 smithi069 ceph-mon[30123]: pgmap v455: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 838 KiB/s rd, 346 KiB/s wr, 91 op/s 2024-05-09T09:21:53.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64660 3.29 MB/sec warmup 111 sec latency 2994.387 ms 2024-05-09T09:21:54.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 64660 3.26 MB/sec warmup 112 sec latency 1144.544 ms 2024-05-09T09:21:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:54 smithi163 ceph-mon[34354]: pgmap v456: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 341 KiB/s wr, 91 op/s 2024-05-09T09:21:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:54 smithi103 ceph-mon[34421]: pgmap v456: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 341 KiB/s wr, 91 op/s 2024-05-09T09:21:54.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:54 smithi069 ceph-mon[30123]: pgmap v456: 129 pgs: 129 active+clean; 32 MiB data, 513 MiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 341 KiB/s wr, 91 op/s 2024-05-09T09:21:55.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 65128 3.25 MB/sec warmup 113 sec latency 1897.987 ms 2024-05-09T09:21:55.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 -- 172.21.15.69:0/2527616120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460100790 msgr2=0x7f0460100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 --2- 172.21.15.69:0/2527616120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460100790 0x7f0460100bb0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f0450009900 tx=0x7f045002a410 comp rx=0 tx=0).stop 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 -- 172.21.15.69:0/2527616120 shutdown_connections 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 --2- 172.21.15.69:0/2527616120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460102340 0x7f0460108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 --2- 172.21.15.69:0/2527616120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 0x7f0460101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 --2- 172.21.15.69:0/2527616120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460100790 0x7f0460100bb0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 -- 172.21.15.69:0/2527616120 >> 172.21.15.69:0/2527616120 conn(0x7f04600fbd00 msgr2=0x7f04600fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 -- 172.21.15.69:0/2527616120 shutdown_connections 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.721+0000 7f0464a6b640 1 -- 172.21.15.69:0/2527616120 wait complete. 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 Processor -- start 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 -- start start 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460100790 0x7f0460197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 0x7f0460198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f046010c410 con 0x7f0460102340 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f046010c290 con 0x7f0460101980 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0464a6b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f046010c590 con 0x7f0460100790 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f045ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f045ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50614/0 (socket says 172.21.15.69:50614) 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f045ed9d640 1 -- 172.21.15.69:0/342491994 learned_addr learned my addr 172.21.15.69:0/342491994 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f045e59c640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460100790 0x7f0460197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.722+0000 7f0457fff640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 0x7f0460198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 -- 172.21.15.69:0/342491994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460100790 msgr2=0x7f0460197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460100790 0x7f0460197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 -- 172.21.15.69:0/342491994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 msgr2=0x7f0460198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 0x7f0460198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 -- 172.21.15.69:0/342491994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f046019f740 con 0x7f0460102340 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f045ed9d640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f045800eb10 tx=0x7f045800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f04577fe640 1 -- 172.21.15.69:0/342491994 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458004b20 con 0x7f0460102340 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f04577fe640 1 -- 172.21.15.69:0/342491994 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0458004cc0 con 0x7f0460102340 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f046019fa30 con 0x7f0460102340 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f04577fe640 1 -- 172.21.15.69:0/342491994 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0458010760 con 0x7f0460102340 2024-05-09T09:21:55.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.723+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f046019ff80 con 0x7f0460102340 2024-05-09T09:21:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.725+0000 7f04577fe640 1 -- 172.21.15.69:0/342491994 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0458005020 con 0x7f0460102340 2024-05-09T09:21:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 Processor -- start 2024-05-09T09:21:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 -- start start 2024-05-09T09:21:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a90670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a95740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6a98240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5635a69f4340 con 0x5635a6a90250 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5635a699c290 con 0x5635a6a95db0 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f046777b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5635a69f0300 con 0x5635a6a90c40 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f0457fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a90670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a95740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f0457fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a90670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50628/0 (socket says 172.21.15.69:50628) 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f0457fff640 1 -- 172.21.15.69:0/1774040769 learned_addr learned my addr 172.21.15.69:0/1774040769 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f0457fff640 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6a98240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f04577fe640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 0x7f044c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 -- 172.21.15.69:0/1774040769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 msgr2=0x5635a6a98240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6a98240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 -- 172.21.15.69:0/1774040769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 msgr2=0x5635a6a90670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a90670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045e59c640 1 -- 172.21.15.69:0/1774040769 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635a69c08e0 con 0x5635a6a90c40 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.726+0000 7f045ed9d640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 0x7f044c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f04577fe640 1 -- 172.21.15.69:0/342491994 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f04580cfac0 con 0x7f0460102340 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f045e59c640 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a95740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0450009230 tx=0x7f0450005e20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f6a4550ce30a0be3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f045ed9d640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 0x7f044c0a50f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0458000f80 tx=0x7f0458067060 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f0454ff9640 1 -- 172.21.15.69:0/1774040769 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450040070 con 0x5635a6a90c40 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f0454ff9640 1 -- 172.21.15.69:0/1774040769 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0450004030 con 0x5635a6a90c40 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.727+0000 7f0454ff9640 1 -- 172.21.15.69:0/1774040769 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450035040 con 0x5635a6a90c40 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- 172.21.15.69:0/1774040769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 msgr2=0x5635a6a95740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a95740 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0450009230 tx=0x7f0450005e20 comp rx=0 tx=0).stop 2024-05-09T09:21:55.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- 172.21.15.69:0/1774040769 shutdown_connections 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6a98240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a95740 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- 172.21.15.69:0/1774040769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a90670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- 172.21.15.69:0/1774040769 >> 172.21.15.69:0/1774040769 conn(0x5635a6a8b8c0 msgr2=0x5635a6a8bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- 172.21.15.69:0/1774040769 shutdown_connections 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- 172.21.15.69:0/1774040769 wait complete. 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- start start 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 Processor -- start 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a9a470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5635a6a98d00 con 0x5635a6a90c40 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5635a69af790 con 0x5635a6a95db0 2024-05-09T09:21:55.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.728+0000 7f046777b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5635a6a98e80 con 0x5635a6a90250 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f046777b740 10 client.0 _ll_register_callbacks cb 0x5635a69eaa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0457fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a9a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50638/0 (socket says 172.21.15.69:50638) 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 -- 172.21.15.69:0/2639295814 learned_addr learned my addr 172.21.15.69:0/2639295814 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0457fff640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 -- 172.21.15.69:0/2639295814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 msgr2=0x5635a6a9a470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a9a470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 -- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 msgr2=0x5635a6aa19f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa19f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 -- 172.21.15.69:0/2639295814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635a6708210 con 0x5635a6a90c40 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0457fff640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a9a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f045e59c640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0450005ab0 tx=0x7f0450009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=db445f4ede8c6b3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0445ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045004c070 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f04500043e0 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0450004030 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0450058050 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f046777b740 10 client.16272 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635a67d4620 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.729+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0450004030 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0450004e40 con 0x5635a6a90c40 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 10 client.16272 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 10 client.16272 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 20 client.16272 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6aa5b50 0x5635a6aa7f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 4 client.16272 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5635a6a01a70 con 0x5635a6aa5b50 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f0457fff640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6aa5b50 0x5635a6aa7f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f0457fff640 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6aa5b50 0x5635a6aa7f40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e57fb90db2a123f2 server_cookie=6e2f9b1c3623e35c in_seq=0 out_seq=0 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.730+0000 7f0445ffb640 10 client.16272 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f0445ffb640 1 -- 172.21.15.69:0/2639295814 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5635a6a01a70 con 0x5635a6aa5b50 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f0445ffb640 10 client.16272 handle_command_reply: tid=0 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 client.16272 shutdown 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 msgr2=0x5635a6a9a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f0450005ab0 tx=0x7f0450009260 comp rx=0 tx=0).stop 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 shutdown_connections 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa19f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90c40 0x5635a6a9a9b0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6aa5b50 0x5635a6aa7f40 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 --2- 172.21.15.69:0/2639295814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a9a470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 >> 172.21.15.69:0/2639295814 conn(0x5635a6a8b8c0 msgr2=0x5635a6a8aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f0445ffb640 0 client.16272 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 shutdown_connections 2024-05-09T09:21:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 1 -- 172.21.15.69:0/2639295814 wait complete. 2024-05-09T09:21:55.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.731+0000 7f046777b740 20 client.16272 trim_cache size 0 max 16384 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 Processor -- start 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 -- start start 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a91e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 0x5635a6a923c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a95db0 0x5635a6aa19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04601acc60 con 0x5635a6a95db0 2024-05-09T09:21:55.753 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04601aa4d0 con 0x5635a6a90c40 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.746+0000 7f046777b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04601a8320 con 0x5635a6a90250 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0457fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a95db0 0x5635a6aa19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0457fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a91e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 0x5635a6a923c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0457fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a91e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:41802/0 (socket says 172.21.15.69:41802) 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0457fff640 1 -- 172.21.15.69:0/3374540037 learned_addr learned my addr 172.21.15.69:0/3374540037 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 -- 172.21.15.69:0/3374540037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 msgr2=0x5635a6a91e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a91e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 -- 172.21.15.69:0/3374540037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a95db0 msgr2=0x5635a6aa19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a95db0 0x5635a6aa19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 -- 172.21.15.69:0/3374540037 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635a6a89f00 con 0x5635a6a90c40 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f045e59c640 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 0x5635a6a923c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0450003670 tx=0x7f0450005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=482ed2d3d36c7c46 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0447fff640 1 -- 172.21.15.69:0/3374540037 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f045004c050 con 0x5635a6a90c40 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0447fff640 1 -- 172.21.15.69:0/3374540037 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0450003c10 con 0x5635a6a90c40 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.747+0000 7f0447fff640 1 -- 172.21.15.69:0/3374540037 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0450040050 con 0x5635a6a90c40 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- 172.21.15.69:0/3374540037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 msgr2=0x5635a6a923c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 0x5635a6a923c0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f0450003670 tx=0x7f0450005de0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- 172.21.15.69:0/3374540037 shutdown_connections 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a95db0 0x5635a6aa19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a90c40 0x5635a6a923c0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 --2- 172.21.15.69:0/3374540037 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90250 0x5635a6a91e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- 172.21.15.69:0/3374540037 >> 172.21.15.69:0/3374540037 conn(0x5635a6a8b8c0 msgr2=0x5635a6a954d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- 172.21.15.69:0/3374540037 shutdown_connections 2024-05-09T09:21:55.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- 172.21.15.69:0/3374540037 wait complete. 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 -- start start 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.748+0000 7f046777b740 1 Processor -- start 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a9f1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f04601a8990 con 0x5635a6a90250 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f04601a9e00 con 0x5635a6a95db0 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f04601a9060 con 0x5635a6a90c40 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 10 client.0 _ll_register_callbacks cb 0x7f0448009a90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50650/0 (socket says 172.21.15.69:50650) 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 -- 172.21.15.69:0/3496505158 learned_addr learned my addr 172.21.15.69:0/3496505158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f045e59c640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a9f1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 -- 172.21.15.69:0/3496505158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 msgr2=0x5635a6a9f1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a9f1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 -- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 msgr2=0x5635a6aa01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa01f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 -- 172.21.15.69:0/3496505158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5635a6aa07e0 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0457fff640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x5635a6aa8730 tx=0x7f04480046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4607cf8e3d78f77a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0445ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f044802e020 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0450003c10 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(94..94 src has 1..94) ==== 9129+0+0 (secure 0 0 0) 0x7f0448014c80 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=30) ==== 24+0+0 (secure 0 0 0) 0x7f0448014e80 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 10 client.16278 fetch_fsmap learned FSMap version 30 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.749+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=30,monmap=4+}) -- 0x5635a6aa1bb0 con 0x5635a6a90250 2024-05-09T09:21:55.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0450003c10 con 0x5635a6a90250 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 30) ==== 8354+0+0 (secure 0 0 0) 0x7f0448027790 con 0x5635a6a90250 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 10 client.16278 fetch_fsmap finished waiting for FSMap version 30 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 10 client.16278 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 20 client.16278 populate_metadata read hostname 'smithi069' 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6b0c750 0x5635a6b0eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 4 client.16278 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635a69fbdc0 con 0x5635a6b0c750 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f045e59c640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6b0c750 0x5635a6b0eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f0445ffb640 10 client.16278 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:55.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.750+0000 7f045e59c640 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6b0c750 0x5635a6b0eb70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5eeda17a0a53c4e2 server_cookie=88a24c080bd46a02 in_seq=0 out_seq=0 2024-05-09T09:21:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f0445ffb640 1 -- 172.21.15.69:0/3496505158 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635a69fbdc0 con 0x5635a6b0c750 2024-05-09T09:21:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f0445ffb640 10 client.16278 handle_command_reply: tid=0 2024-05-09T09:21:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 client.16278 shutdown 2024-05-09T09:21:55.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 msgr2=0x5635a6a9eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x5635a6aa8730 tx=0x7f04480046c0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 shutdown_connections 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5635a6a95db0 0x5635a6aa01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5635a6b0c750 0x5635a6b0eb70 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5635a6a90c40 0x5635a6a9f1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 --2- 172.21.15.69:0/3496505158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5635a6a90250 0x5635a6a9eca0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 >> 172.21.15.69:0/3496505158 conn(0x5635a6a8b8c0 msgr2=0x5635a6a9a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f0445ffb640 0 client.16278 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 shutdown_connections 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.774+0000 7f046777b740 1 -- 172.21.15.69:0/3496505158 wait complete. 2024-05-09T09:21:55.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f046777b740 20 client.16278 trim_cache size 0 max 16384 2024-05-09T09:21:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:21:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:21:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:21:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:21:55.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 msgr2=0x7f044c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 0x7f044c0a50f0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f0458000f80 tx=0x7f0458067060 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 msgr2=0x7f046019f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f045800eb10 tx=0x7f045800c2d0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 shutdown_connections 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f044c0a2c30 0x7f044c0a50f0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0460102340 0x7f046019f0d0 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0460101980 0x7f0460198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 --2- 172.21.15.69:0/342491994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0460100790 0x7f0460197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 >> 172.21.15.69:0/342491994 conn(0x7f04600fbd00 msgr2=0x7f04600fd730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.775+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 shutdown_connections 2024-05-09T09:21:55.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:21:55.776+0000 7f0464a6b640 1 -- 172.21.15.69:0/342491994 wait complete. 2024-05-09T09:21:56.272 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:21:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:21:56.272 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:21:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:21:56.272 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:21:55 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/2660737796 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x556fcc127a40 con 0x556fcc0fa800 2024-05-09T09:21:56.272 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:21:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:21:56.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 67792 3.37 MB/sec warmup 114 sec latency 15.814 ms 2024-05-09T09:21:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:56 smithi163 ceph-mon[34354]: pgmap v457: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 91 op/s 2024-05-09T09:21:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:56 smithi163 ceph-mon[34354]: daemon mds.c restarted 2024-05-09T09:21:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:56 smithi103 ceph-mon[34421]: pgmap v457: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 91 op/s 2024-05-09T09:21:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:56 smithi103 ceph-mon[34421]: daemon mds.c restarted 2024-05-09T09:21:56.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:56 smithi069 ceph-mon[30123]: pgmap v457: 129 pgs: 129 active+clean; 32 MiB data, 497 MiB used, 1.0 TiB / 1.0 TiB avail; 656 KiB/s rd, 587 KiB/s wr, 91 op/s 2024-05-09T09:21:56.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:56 smithi069 ceph-mon[30123]: daemon mds.c restarted 2024-05-09T09:21:57.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 68211 3.35 MB/sec warmup 115 sec latency 570.832 ms 2024-05-09T09:21:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T09:21:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/2291428128,v1:172.21.15.163:6837/2291428128] up:boot 2024-05-09T09:21:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:21:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:21:57.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:21:57.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:21:57 smithi163 ceph-mon[34354]: pgmap v459: 129 pgs: 129 active+clean; 33 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 950 KiB/s wr, 108 op/s 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/2291428128,v1:172.21.15.163:6837/2291428128] up:boot 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:21:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:21:57 smithi103 ceph-mon[34421]: pgmap v459: 129 pgs: 129 active+clean; 33 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 950 KiB/s wr, 108 op/s 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: osdmap e95: 12 total, 12 up, 12 in 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/2291428128,v1:172.21.15.163:6837/2291428128] up:boot 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:21:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:21:57 smithi069 ceph-mon[30123]: pgmap v459: 129 pgs: 129 active+clean; 33 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 950 KiB/s wr, 108 op/s 2024-05-09T09:21:58.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 68211 3.32 MB/sec warmup 116 sec latency 1570.946 ms 2024-05-09T09:21:59.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 68211 3.29 MB/sec warmup 117 sec latency 2570.981 ms 2024-05-09T09:22:00.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 68805 3.30 MB/sec warmup 118 sec latency 3322.301 ms 2024-05-09T09:22:00.509 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:00 smithi069 ceph-mon[30123]: pgmap v460: 129 pgs: 129 active+clean; 32 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-05-09T09:22:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:00 smithi163 ceph-mon[34354]: pgmap v460: 129 pgs: 129 active+clean; 32 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-05-09T09:22:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:00 smithi103 ceph-mon[34421]: pgmap v460: 129 pgs: 129 active+clean; 32 MiB data, 531 MiB used, 1.0 TiB / 1.0 TiB avail; 577 KiB/s rd, 1.2 MiB/s wr, 115 op/s 2024-05-09T09:22:01.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71344 3.39 MB/sec warmup 119 sec latency 9.115 ms 2024-05-09T09:22:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:02 smithi069 ceph-mon[30123]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-05-09T09:22:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:22:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:22:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:22:02.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:22:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:02 smithi163 ceph-mon[34354]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-05-09T09:22:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:22:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:22:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:22:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:22:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:02 smithi103 ceph-mon[34421]: pgmap v461: 129 pgs: 129 active+clean; 29 MiB data, 532 MiB used, 1.0 TiB / 1.0 TiB avail; 552 KiB/s rd, 1.2 MiB/s wr, 110 op/s 2024-05-09T09:22:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:22:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:22:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:22:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:22:03.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71685 0.00 MB/sec execute 1 sec latency 1626.449 ms 2024-05-09T09:22:04.478 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71685 0.00 MB/sec execute 2 sec latency 2626.527 ms 2024-05-09T09:22:04.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:04 smithi069 ceph-mon[30123]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T09:22:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:04 smithi163 ceph-mon[34354]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T09:22:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:04 smithi103 ceph-mon[34421]: pgmap v462: 129 pgs: 129 active+clean; 30 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 1.3 MiB/s wr, 134 op/s 2024-05-09T09:22:05.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71753 0.05 MB/sec execute 3 sec latency 3377.922 ms 2024-05-09T09:22:06.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71753 0.04 MB/sec execute 4 sec latency 1188.985 ms 2024-05-09T09:22:06.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:06 smithi069 ceph-mon[30123]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-09T09:22:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:06 smithi163 ceph-mon[34354]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-09T09:22:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:06 smithi103 ceph-mon[34421]: pgmap v463: 129 pgs: 129 active+clean; 30 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 980 KiB/s rd, 1.0 MiB/s wr, 122 op/s 2024-05-09T09:22:07.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71753 0.03 MB/sec execute 5 sec latency 2189.122 ms 2024-05-09T09:22:08.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71753 0.03 MB/sec execute 6 sec latency 3189.253 ms 2024-05-09T09:22:08.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:08 smithi069 ceph-mon[30123]: pgmap v464: 129 pgs: 129 active+clean; 31 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 911 KiB/s wr, 117 op/s 2024-05-09T09:22:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:08 smithi163 ceph-mon[34354]: pgmap v464: 129 pgs: 129 active+clean; 31 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 911 KiB/s wr, 117 op/s 2024-05-09T09:22:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:08 smithi103 ceph-mon[34421]: pgmap v464: 129 pgs: 129 active+clean; 31 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 942 KiB/s rd, 911 KiB/s wr, 117 op/s 2024-05-09T09:22:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2407278308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 msgr2=0x7fe4e4107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2407278308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4107030 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8009900 tx=0x7fe4d802a410 comp rx=0 tx=0).stop 2024-05-09T09:22:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2407278308 shutdown_connections 2024-05-09T09:22:08.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2407278308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4e4107570 0x7fe4e41099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2407278308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4107030 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2407278308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2407278308 >> 172.21.15.69:0/2407278308 conn(0x7fe4e40fbd00 msgr2=0x7fe4e40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2407278308 shutdown_connections 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2407278308 wait complete. 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 Processor -- start 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- start start 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4e4107570 0x7fe4e419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e410d510 con 0x7fe4e4104c40 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e410d390 con 0x7fe4e4102310 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.874+0000 7fe4eb3a9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e410d690 con 0x7fe4e4107570 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39432/0 (socket says 172.21.15.69:39432) 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 -- 172.21.15.69:0/2567047416 learned_addr learned my addr 172.21.15.69:0/2567047416 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 -- 172.21.15.69:0/2567047416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4e4107570 msgr2=0x7fe4e419f070 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4e4107570 0x7fe4e419f070 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 -- 172.21.15.69:0/2567047416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 msgr2=0x7fe4e4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 -- 172.21.15.69:0/2567047416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe4e419f6e0 con 0x7fe4e4104c40 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8031610 tx=0x7fe4d80043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e27fc640 1 -- 172.21.15.69:0/2567047416 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d803f030 con 0x7fe4e4104c40 2024-05-09T09:22:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e27fc640 1 -- 172.21.15.69:0/2567047416 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d8003ce0 con 0x7fe4e4104c40 2024-05-09T09:22:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe4e419f970 con 0x7fe4e4104c40 2024-05-09T09:22:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4e27fc640 1 -- 172.21.15.69:0/2567047416 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d8035070 con 0x7fe4e4104c40 2024-05-09T09:22:08.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.875+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe4e419fe60 con 0x7fe4e4104c40 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4e27fc640 1 -- 172.21.15.69:0/2567047416 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe4d8042030 con 0x7fe4e4104c40 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 Processor -- start 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 -- start start 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0250 0x5645793a0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793a8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564579304340 con 0x5645793a0c40 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645792ac290 con 0x5645793a5db0 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4ee197740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564579300300 con 0x5645793a0250 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39444/0 (socket says 172.21.15.69:39444) 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.878+0000 7fe4e9145640 1 -- 172.21.15.69:0/2419607426 learned_addr learned my addr 172.21.15.69:0/2419607426 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9946640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0250 0x5645793a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793a8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e27fc640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 0x7fe4cc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9145640 1 -- 172.21.15.69:0/2419607426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0250 msgr2=0x5645793a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0250 0x5645793a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9145640 1 -- 172.21.15.69:0/2419607426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 msgr2=0x5645793a8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793a8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9145640 1 -- 172.21.15.69:0/2419607426 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645792d08e0 con 0x5645793a0c40 2024-05-09T09:22:08.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9946640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 0x7fe4cc0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e27fc640 1 -- 172.21.15.69:0/2567047416 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe4d80efa10 con 0x7fe4e4104c40 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.879+0000 7fe4e9946640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 0x7fe4cc0a50f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5645793a14a0 tx=0x7fe4dc004730 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400e320 tx=0x7fe4d400e7e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60f5e1bbd997de66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4cb7fe640 1 -- 172.21.15.69:0/2419607426 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4016070 con 0x5645793a0c40 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4cb7fe640 1 -- 172.21.15.69:0/2419607426 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4010b70 con 0x5645793a0c40 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4cb7fe640 1 -- 172.21.15.69:0/2419607426 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4011cf0 con 0x5645793a0c40 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 -- 172.21.15.69:0/2419607426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 msgr2=0x5645793a5740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400e320 tx=0x7fe4d400e7e0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 -- 172.21.15.69:0/2419607426 shutdown_connections 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793a8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0c40 0x5645793a5740 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2419607426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0250 0x5645793a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.880+0000 7fe4ee197740 1 -- 172.21.15.69:0/2419607426 >> 172.21.15.69:0/2419607426 conn(0x56457939b8c0 msgr2=0x56457939bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- 172.21.15.69:0/2419607426 shutdown_connections 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- 172.21.15.69:0/2419607426 wait complete. 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- start start 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 Processor -- start 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x5645793aaa10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x5645793b19e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5645793a8d60 con 0x5645793a0250 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5645792bf790 con 0x5645793a0c40 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5645793a8ee0 con 0x5645793a5db0 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4ee197740 10 client.0 _ll_register_callbacks cb 0x5645792faa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:08.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.881+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39450/0 (socket says 172.21.15.69:39450) 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 -- 172.21.15.69:0/2570568801 learned_addr learned my addr 172.21.15.69:0/2570568801 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x5645793aaa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 -- 172.21.15.69:0/2570568801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 msgr2=0x5645793b19e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x5645793b19e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 -- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 msgr2=0x5645793aaa10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x5645793aaa10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 -- 172.21.15.69:0/2570568801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564579018210 con 0x5645793a0250 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x5645793aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4e9145640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006010 tx=0x7fe4d4009d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a03230dffdaa528 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4028060 con 0x5645793a0250 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe4d4004a40 con 0x5645793a0250 2024-05-09T09:22:08.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d4010b70 con 0x5645793a0250 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe4d4016070 con 0x5645793a0250 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4ee197740 10 client.16290 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645790e4620 con 0x5645793a0250 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.882+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4010b70 con 0x5645793a0250 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe4d400f040 con 0x5645793a0250 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 10 client.16290 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 10 client.16290 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 20 client.16290 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5645793b5dc0 0x5645793b81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 4 client.16290 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564579311a70 con 0x5645793b5dc0 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5645793b5dc0 0x5645793b81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4c8ff9640 10 client.16290 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:08.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.883+0000 7fe4e8944640 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5645793b5dc0 0x5645793b81f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec2fcaccba9b0904 server_cookie=302079520778bc5e in_seq=0 out_seq=0 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/2570568801 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564579311a70 con 0x5645793b5dc0 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4c8ff9640 10 client.16290 handle_command_reply: tid=0 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 client.16290 shutdown 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 msgr2=0x5645793aa4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4006010 tx=0x7fe4d4009d70 comp rx=0 tx=0).stop 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 shutdown_connections 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5645793b5dc0 0x5645793b81f0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x5645793b19e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x5645793aaa10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 --2- 172.21.15.69:0/2570568801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793aa4d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4c8ff9640 0 client.16290 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.884+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 >> 172.21.15.69:0/2570568801 conn(0x56457939b8c0 msgr2=0x56457939aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.885+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 shutdown_connections 2024-05-09T09:22:08.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.885+0000 7fe4ee197740 1 -- 172.21.15.69:0/2570568801 wait complete. 2024-05-09T09:22:08.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.885+0000 7fe4ee197740 20 client.16290 trim_cache size 0 max 16384 2024-05-09T09:22:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 Processor -- start 2024-05-09T09:22:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 -- start start 2024-05-09T09:22:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x56457939a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x56457939abf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41acd10 con 0x5645793a0250 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41aa580 con 0x5645793a0c40 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4ee197740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41a83d0 con 0x5645793a5db0 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39454/0 (socket says 172.21.15.69:39454) 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4e9145640 1 -- 172.21.15.69:0/3849249998 learned_addr learned my addr 172.21.15.69:0/3849249998 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4e9946640 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x56457939abf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.902+0000 7fe4e8944640 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x56457939a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 -- 172.21.15.69:0/3849249998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 msgr2=0x56457939abf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x56457939abf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 -- 172.21.15.69:0/3849249998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 msgr2=0x56457939a6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x56457939a6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 -- 172.21.15.69:0/3849249998 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5645793a5150 con 0x5645793a0250 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4e9145640 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ece0 tx=0x7fe4d4011a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68b0f8cd527a85e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4caffd640 1 -- 172.21.15.69:0/3849249998 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4038070 con 0x5645793a0250 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4caffd640 1 -- 172.21.15.69:0/3849249998 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4004320 con 0x5645793a0250 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.903+0000 7fe4caffd640 1 -- 172.21.15.69:0/3849249998 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4028030 con 0x5645793a0250 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- 172.21.15.69:0/3849249998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 msgr2=0x56457939a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7fe4d400ece0 tx=0x7fe4d4011a00 comp rx=0 tx=0).stop 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- 172.21.15.69:0/3849249998 shutdown_connections 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a5db0 0x56457939abf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a0c40 0x56457939a6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 --2- 172.21.15.69:0/3849249998 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x56457939a170 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- 172.21.15.69:0/3849249998 >> 172.21.15.69:0/3849249998 conn(0x56457939b8c0 msgr2=0x5645793a54d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- 172.21.15.69:0/3849249998 shutdown_connections 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- 172.21.15.69:0/3849249998 wait complete. 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 -- start start 2024-05-09T09:22:08.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.904+0000 7fe4ee197740 1 Processor -- start 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0c40 0x5645793ae840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793af850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41a8a40 con 0x5645793a0250 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41a9eb0 con 0x5645793a5db0 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe4e41a9110 con 0x5645793a0c40 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 10 client.0 _ll_register_callbacks cb 0x7fe4dc005e70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:08.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39458/0 (socket says 172.21.15.69:39458) 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 -- 172.21.15.69:0/1414727344 learned_addr learned my addr 172.21.15.69:0/1414727344 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e8944640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0c40 0x5645793ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9946640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 -- 172.21.15.69:0/1414727344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0c40 msgr2=0x5645793ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0c40 0x5645793ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 -- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 msgr2=0x5645793af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 -- 172.21.15.69:0/1414727344 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5645793afe40 con 0x5645793a0250 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9946640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4e9145640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009860 tx=0x7fe4d40117e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bec66c18aed23ea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:08.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4c8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe4d4038070 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe4d4010b20 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe4d4004320 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.905+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fe4d4045050 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 10 client.16296 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x5645793b1150 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe4d4004320 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fe4d4005040 con 0x5645793a0250 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 10 client.16296 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 10 client.16296 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 20 client.16296 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56457941c9c0 0x56457941ede0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 4 client.16296 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:22:08.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56457930bdc0 con 0x56457941c9c0 2024-05-09T09:22:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4e8944640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56457941c9c0 0x56457941ede0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4e8944640 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56457941c9c0 0x56457941ede0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f3717a4b73d88 server_cookie=1bbd05c33e97131e in_seq=0 out_seq=0 2024-05-09T09:22:08.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.906+0000 7fe4c8ff9640 10 client.16296 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.942+0000 7fe4c8ff9640 1 -- 172.21.15.69:0/1414727344 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56457930bdc0 con 0x56457941c9c0 2024-05-09T09:22:08.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.942+0000 7fe4c8ff9640 10 client.16296 handle_command_reply: tid=0 2024-05-09T09:22:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.942+0000 7fe4ee197740 1 client.16296 shutdown 2024-05-09T09:22:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 msgr2=0x5645793ae300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7fe4d4009860 tx=0x7fe4d40117e0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 shutdown_connections 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5645793a5db0 0x5645793af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56457941c9c0 0x56457941ede0 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5645793a0c40 0x5645793ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 --2- 172.21.15.69:0/1414727344 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5645793a0250 0x5645793ae300 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4c8ff9640 0 client.16296 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 >> 172.21.15.69:0/1414727344 conn(0x56457939b8c0 msgr2=0x56457939e640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 shutdown_connections 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 1 -- 172.21.15.69:0/1414727344 wait complete. 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4ee197740 20 client.16296 trim_cache size 0 max 16384 2024-05-09T09:22:08.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:22:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:22:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:22:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:22:08.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 msgr2=0x7fe4cc0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 0x7fe4cc0a50f0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x5645793a14a0 tx=0x7fe4dc004730 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 msgr2=0x7fe4e4198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x7fe4d8031610 tx=0x7fe4d80043b0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 shutdown_connections 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe4cc0a2c30 0x7fe4cc0a50f0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe4e4107570 0x7fe4e419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe4e4104c40 0x7fe4e4198000 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.943+0000 7fe4eb3a9640 1 --2- 172.21.15.69:0/2567047416 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe4e4102310 0x7fe4e4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.944+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 >> 172.21.15.69:0/2567047416 conn(0x7fe4e40fbd00 msgr2=0x7fe4e4102c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.944+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 shutdown_connections 2024-05-09T09:22:08.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:08.944+0000 7fe4eb3a9640 1 -- 172.21.15.69:0/2567047416 wait complete. 2024-05-09T09:22:09.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 71753 0.02 MB/sec execute 7 sec latency 4189.408 ms 2024-05-09T09:22:10.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 72194 0.23 MB/sec execute 8 sec latency 4940.763 ms 2024-05-09T09:22:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:10 smithi163 ceph-mon[34354]: pgmap v465: 129 pgs: 129 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 561 KiB/s wr, 76 op/s 2024-05-09T09:22:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:10 smithi103 ceph-mon[34421]: pgmap v465: 129 pgs: 129 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 561 KiB/s wr, 76 op/s 2024-05-09T09:22:10.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:10 smithi069 ceph-mon[30123]: pgmap v465: 129 pgs: 129 active+clean; 30 MiB data, 514 MiB used, 1.0 TiB / 1.0 TiB avail; 727 KiB/s rd, 561 KiB/s wr, 76 op/s 2024-05-09T09:22:11.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 74824 2.07 MB/sec execute 9 sec latency 11.936 ms 2024-05-09T09:22:12.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75210 1.90 MB/sec execute 10 sec latency 616.633 ms 2024-05-09T09:22:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:12 smithi163 ceph-mon[34354]: pgmap v466: 129 pgs: 129 active+clean; 30 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 281 KiB/s wr, 63 op/s 2024-05-09T09:22:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:12 smithi103 ceph-mon[34421]: pgmap v466: 129 pgs: 129 active+clean; 30 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 281 KiB/s wr, 63 op/s 2024-05-09T09:22:12.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:12 smithi069 ceph-mon[30123]: pgmap v466: 129 pgs: 129 active+clean; 30 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 484 KiB/s rd, 281 KiB/s wr, 63 op/s 2024-05-09T09:22:13.479 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75210 1.73 MB/sec execute 11 sec latency 1616.730 ms 2024-05-09T09:22:14.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75210 1.59 MB/sec execute 12 sec latency 2616.842 ms 2024-05-09T09:22:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:14 smithi163 ceph-mon[34354]: pgmap v467: 129 pgs: 129 active+clean; 31 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 287 KiB/s wr, 78 op/s 2024-05-09T09:22:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:14 smithi103 ceph-mon[34421]: pgmap v467: 129 pgs: 129 active+clean; 31 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 287 KiB/s wr, 78 op/s 2024-05-09T09:22:14.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:14 smithi069 ceph-mon[30123]: pgmap v467: 129 pgs: 129 active+clean; 31 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 287 KiB/s wr, 78 op/s 2024-05-09T09:22:15.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75334 1.49 MB/sec execute 13 sec latency 3368.514 ms 2024-05-09T09:22:16.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75334 1.38 MB/sec execute 14 sec latency 1143.875 ms 2024-05-09T09:22:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:16 smithi163 ceph-mon[34354]: pgmap v468: 129 pgs: 129 active+clean; 32 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 224 KiB/s wr, 63 op/s 2024-05-09T09:22:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:16 smithi103 ceph-mon[34421]: pgmap v468: 129 pgs: 129 active+clean; 32 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 224 KiB/s wr, 63 op/s 2024-05-09T09:22:16.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:16 smithi069 ceph-mon[30123]: pgmap v468: 129 pgs: 129 active+clean; 32 MiB data, 525 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 224 KiB/s wr, 63 op/s 2024-05-09T09:22:17.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75334 1.29 MB/sec execute 15 sec latency 2143.973 ms 2024-05-09T09:22:18.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 75334 1.21 MB/sec execute 16 sec latency 3144.074 ms 2024-05-09T09:22:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:18 smithi163 ceph-mon[34354]: pgmap v469: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 231 KiB/s wr, 70 op/s 2024-05-09T09:22:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:18 smithi103 ceph-mon[34421]: pgmap v469: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 231 KiB/s wr, 70 op/s 2024-05-09T09:22:18.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:18 smithi069 ceph-mon[30123]: pgmap v469: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 231 KiB/s wr, 70 op/s 2024-05-09T09:22:19.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 76913 1.52 MB/sec execute 17 sec latency 3553.355 ms 2024-05-09T09:22:20.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 78754 2.12 MB/sec execute 18 sec latency 11.792 ms 2024-05-09T09:22:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:20 smithi163 ceph-mon[34354]: pgmap v470: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 294 KiB/s wr, 60 op/s 2024-05-09T09:22:20.561 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:20 smithi069 ceph-mon[30123]: pgmap v470: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 294 KiB/s wr, 60 op/s 2024-05-09T09:22:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:20 smithi103 ceph-mon[34421]: pgmap v470: 129 pgs: 129 active+clean; 32 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 602 KiB/s rd, 294 KiB/s wr, 60 op/s 2024-05-09T09:22:21.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 78809 2.02 MB/sec execute 19 sec latency 961.177 ms 2024-05-09T09:22:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:21 smithi163 ceph-mon[34354]: pgmap v471: 129 pgs: 129 active+clean; 32 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 465 KiB/s wr, 75 op/s 2024-05-09T09:22:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:21 smithi103 ceph-mon[34421]: pgmap v471: 129 pgs: 129 active+clean; 32 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 465 KiB/s wr, 75 op/s 2024-05-09T09:22:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:21 smithi069 ceph-mon[30123]: pgmap v471: 129 pgs: 129 active+clean; 32 MiB data, 528 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 465 KiB/s wr, 75 op/s 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 -- 172.21.15.69:0/40431161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b24100840 msgr2=0x7f8b24100c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/40431161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b24100840 0x7f8b24100c60 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f8b14009900 tx=0x7f8b1402a410 comp rx=0 tx=0).stop 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 -- 172.21.15.69:0/40431161 shutdown_connections 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/40431161 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 0x7f8b24108c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/40431161 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 0x7f8b24101eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/40431161 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b24100840 0x7f8b24100c60 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 -- 172.21.15.69:0/40431161 >> 172.21.15.69:0/40431161 conn(0x7f8b240fbdb0 msgr2=0x7f8b240fe1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 -- 172.21.15.69:0/40431161 shutdown_connections 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.039+0000 7f8b29f3b640 1 -- 172.21.15.69:0/40431161 wait complete. 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 Processor -- start 2024-05-09T09:22:22.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 -- start start 2024-05-09T09:22:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 0x7f8b2406abd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 0x7f8b2419c9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2410c420 con 0x7f8b2419cf00 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2410c2a0 con 0x7f8b241023f0 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b29f3b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8b2410c5a0 con 0x7f8b24101a30 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b23fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b23fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47484/0 (socket says 172.21.15.69:47484) 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b23fff640 1 -- 172.21.15.69:0/1902245663 learned_addr learned my addr 172.21.15.69:0/1902245663 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b237fe640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 0x7f8b2406abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.040+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 0x7f8b2419c9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 -- 172.21.15.69:0/1902245663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 msgr2=0x7f8b2406abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 0x7f8b2406abd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 -- 172.21.15.69:0/1902245663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 msgr2=0x7f8b2419c9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 0x7f8b2419c9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 -- 172.21.15.69:0/1902245663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8b2419fcd0 con 0x7f8b2419cf00 2024-05-09T09:22:22.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b23fff640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00eaa0 tx=0x7f8b1c00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b20ff9640 1 -- 172.21.15.69:0/1902245663 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c00cdb0 con 0x7f8b2419cf00 2024-05-09T09:22:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b20ff9640 1 -- 172.21.15.69:0/1902245663 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b1c005020 con 0x7f8b2419cf00 2024-05-09T09:22:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8b2419ff60 con 0x7f8b2419cf00 2024-05-09T09:22:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b20ff9640 1 -- 172.21.15.69:0/1902245663 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b1c0116b0 con 0x7f8b2419cf00 2024-05-09T09:22:22.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.041+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8b241a03d0 con 0x7f8b2419cf00 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.043+0000 7f8b20ff9640 1 -- 172.21.15.69:0/1902245663 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8b1c005560 con 0x7f8b2419cf00 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 Processor -- start 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 -- start start 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963f5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a696351340 con 0x55a6963ed250 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a6962f9290 con 0x55a6963edc40 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a69634d300 con 0x55a6963f2db0 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b20ff9640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 0x7f8b0c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47500/0 (socket says 172.21.15.69:47500) 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2815720889 learned_addr learned my addr 172.21.15.69:0/2815720889 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b23fff640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 0x7f8b0c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2815720889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 msgr2=0x55a6963f5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963f5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b20ff9640 1 -- 172.21.15.69:0/1902245663 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7f8b1c0cf610 con 0x7f8b2419cf00 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2815720889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 msgr2=0x55a6963f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.044+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2815720889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a69631d8e0 con 0x55a6963ed250 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800cee0 tx=0x7f8b1800c620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d03b5cee11ba0ce1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b09ffb640 1 -- 172.21.15.69:0/2815720889 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18005220 con 0x55a6963ed250 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b23fff640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 0x7f8b0c0a50f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c09b7f0 tx=0x7f8b1c090fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b09ffb640 1 -- 172.21.15.69:0/2815720889 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b18016030 con 0x55a6963ed250 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b09ffb640 1 -- 172.21.15.69:0/2815720889 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18012930 con 0x55a6963ed250 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2815720889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 msgr2=0x55a6963ed670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800cee0 tx=0x7f8b1800c620 comp rx=0 tx=0).stop 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2815720889 shutdown_connections 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963f5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2815720889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963ed670 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.045+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2815720889 >> 172.21.15.69:0/2815720889 conn(0x55a6963e88c0 msgr2=0x55a6963e8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2815720889 shutdown_connections 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2815720889 wait complete. 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- start start 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 Processor -- start 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f7970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963fe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a6963f5d60 con 0x55a6963ed250 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a69630c790 con 0x55a6963edc40 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a6963f5ee0 con 0x55a6963f2db0 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b2cc46740 10 client.0 _ll_register_callbacks cb 0x55a696347a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47512/0 (socket says 172.21.15.69:47512) 2024-05-09T09:22:22.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.046+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2743011846 learned_addr learned my addr 172.21.15.69:0/2743011846 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2743011846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 msgr2=0x55a6963fe9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963fe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 msgr2=0x55a6963f7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f7970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 -- 172.21.15.69:0/2743011846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a696065210 con 0x55a6963ed250 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800c5f0 tx=0x7f8b1800ca90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c5e640fd87d3dd43 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18004050 con 0x55a6963ed250 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7f8b180041f0 con 0x55a6963ed250 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b18016030 con 0x55a6963ed250 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8b18030050 con 0x55a6963ed250 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b2cc46740 10 client.16308 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:22.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a696131620 con 0x55a6963ed250 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.047+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b18016030 con 0x55a6963ed250 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8b18004960 con 0x55a6963ed250 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 10 client.16308 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 10 client.16308 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 20 client.16308 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a696402cf0 0x55a6964050e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 4 client.16308 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a69635ea70 con 0x55a696402cf0 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a696402cf0 0x55a6964050e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b02ffd640 10 client.16308 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.048+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a696402cf0 0x55a6964050e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4091d23dc8bd871b server_cookie=372b2529067d74bd in_seq=0 out_seq=0 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.049+0000 7f8b02ffd640 1 -- 172.21.15.69:0/2743011846 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a69635ea70 con 0x55a696402cf0 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.049+0000 7f8b02ffd640 10 client.16308 handle_command_reply: tid=0 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.049+0000 7f8b2cc46740 1 client.16308 shutdown 2024-05-09T09:22:22.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.049+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 msgr2=0x55a6963f7430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.049+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f8b1800c5f0 tx=0x7f8b1800ca90 comp rx=0 tx=0).stop 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 shutdown_connections 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a696402cf0 0x55a6964050e0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963f2db0 0x55a6963fe9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b02ffd640 0 client.16308 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963edc40 0x55a6963f7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2743011846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963f7430 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 >> 172.21.15.69:0/2743011846 conn(0x55a6963e88c0 msgr2=0x55a6963e7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 shutdown_connections 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2743011846 wait complete. 2024-05-09T09:22:22.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.050+0000 7f8b2cc46740 20 client.16308 trim_cache size 0 max 16384 2024-05-09T09:22:22.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 Processor -- start 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 -- start start 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963ed250 0x55a6963e8180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963eb930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241ad260 con 0x55a6963edc40 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241aaad0 con 0x55a6963f2db0 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241a8920 con 0x55a6963ed250 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47518/0 (socket says 172.21.15.69:47518) 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 -- 172.21.15.69:0/2417142413 learned_addr learned my addr 172.21.15.69:0/2417142413 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963eb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 -- 172.21.15.69:0/2417142413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963ed250 msgr2=0x55a6963e8180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963ed250 0x55a6963e8180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 -- 172.21.15.69:0/2417142413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 msgr2=0x55a6963eb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963eb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 -- 172.21.15.69:0/2417142413 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a6963e6f00 con 0x55a6963edc40 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b237fe640 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8b14031610 tx=0x7f8b14031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c96afccbf84c52d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b097fa640 1 -- 172.21.15.69:0/2417142413 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14035040 con 0x55a6963edc40 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b097fa640 1 -- 172.21.15.69:0/2417142413 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14049070 con 0x55a6963edc40 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.065+0000 7f8b097fa640 1 -- 172.21.15.69:0/2417142413 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b14044310 con 0x55a6963edc40 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2417142413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 msgr2=0x55a6963fe970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f8b14031610 tx=0x7f8b14031400 comp rx=0 tx=0).stop 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2417142413 shutdown_connections 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963eb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963edc40 0x55a6963fe970 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/2417142413 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963ed250 0x55a6963e8180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2417142413 >> 172.21.15.69:0/2417142413 conn(0x55a6963e88c0 msgr2=0x55a6963e8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2417142413 shutdown_connections 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- 172.21.15.69:0/2417142413 wait complete. 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.066+0000 7f8b2cc46740 1 -- start start 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 Processor -- start 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 0x55a6963fbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963fcf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241a8f90 con 0x55a6963ed250 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241aa400 con 0x55a6963f2db0 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8b241a9660 con 0x55a6963edc40 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b2cc46740 10 client.0 _ll_register_callbacks cb 0x7f8b14009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47534/0 (socket says 172.21.15.69:47534) 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 -- 172.21.15.69:0/1500835968 learned_addr learned my addr 172.21.15.69:0/1500835968 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b237fe640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 0x55a6963fbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963fcf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 -- 172.21.15.69:0/1500835968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 msgr2=0x55a6963fbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 0x55a6963fbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 -- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 msgr2=0x55a6963fcf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963fcf90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 -- 172.21.15.69:0/1500835968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a6963fd580 con 0x55a6963ed250 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b237fe640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 0x55a6963fbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b22ffd640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8b18005e10 tx=0x7f8b18005d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a2e23404d8b726 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.067+0000 7f8b02ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8b18012660 con 0x55a6963ed250 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8b14049070 con 0x55a6963ed250 2024-05-09T09:22:22.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7f8b18012800 con 0x55a6963ed250 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7f8b18012a00 con 0x55a6963ed250 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 10 client.16314 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x55a6963fe800 con 0x55a6963ed250 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8b14049070 con 0x55a6963ed250 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7f8b1801b6d0 con 0x55a6963ed250 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 10 client.16314 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 10 client.16314 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 20 client.16314 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a6964695d0 0x55a69646b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 4 client.16314 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a696358dc0 con 0x55a6964695d0 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.068+0000 7f8b237fe640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a6964695d0 0x55a69646b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.069+0000 7f8b02ffd640 10 client.16314 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:22.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.069+0000 7f8b237fe640 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a6964695d0 0x55a69646b9f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3ee6d50c4dbfbeef server_cookie=fb403ea100a25165 in_seq=0 out_seq=0 2024-05-09T09:22:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b02ffd640 1 -- 172.21.15.69:0/1500835968 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a696358dc0 con 0x55a6964695d0 2024-05-09T09:22:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b02ffd640 10 client.16314 handle_command_reply: tid=0 2024-05-09T09:22:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 client.16314 shutdown 2024-05-09T09:22:22.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 msgr2=0x55a6963fba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f8b18005e10 tx=0x7f8b18005d60 comp rx=0 tx=0).stop 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 shutdown_connections 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a6963f2db0 0x55a6963fcf90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a6964695d0 0x55a69646b9f0 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a6963edc40 0x55a6963fbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 --2- 172.21.15.69:0/1500835968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a6963ed250 0x55a6963fba40 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b02ffd640 0 client.16314 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 >> 172.21.15.69:0/1500835968 conn(0x55a6963e88c0 msgr2=0x55a6963eb930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 shutdown_connections 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 1 -- 172.21.15.69:0/1500835968 wait complete. 2024-05-09T09:22:22.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.112+0000 7f8b2cc46740 20 client.16314 trim_cache size 0 max 16384 2024-05-09T09:22:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:22:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:22:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:22:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:22:22.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 msgr2=0x7f8b0c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 0x7f8b0c0a50f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c09b7f0 tx=0x7f8b1c090fd0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 msgr2=0x7f8b2419f3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f8b1c00eaa0 tx=0x7f8b1c00ef60 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 shutdown_connections 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8b0c0a2c30 0x7f8b0c0a50f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8b2419cf00 0x7f8b2419f3c0 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8b241023f0 0x7f8b2419c9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 --2- 172.21.15.69:0/1902245663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8b24101a30 0x7f8b2406abd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 >> 172.21.15.69:0/1902245663 conn(0x7f8b240fbdb0 msgr2=0x7f8b24104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 shutdown_connections 2024-05-09T09:22:22.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:22.113+0000 7f8b29f3b640 1 -- 172.21.15.69:0/1902245663 wait complete. 2024-05-09T09:22:22.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 78809 1.91 MB/sec execute 20 sec latency 1961.206 ms 2024-05-09T09:22:23.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 78809 1.82 MB/sec execute 21 sec latency 2961.244 ms 2024-05-09T09:22:24.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 78809 1.74 MB/sec execute 22 sec latency 3961.335 ms 2024-05-09T09:22:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:24 smithi069 ceph-mon[30123]: pgmap v472: 129 pgs: 129 active+clean; 33 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 926 KiB/s wr, 110 op/s 2024-05-09T09:22:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:24 smithi163 ceph-mon[34354]: pgmap v472: 129 pgs: 129 active+clean; 33 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 926 KiB/s wr, 110 op/s 2024-05-09T09:22:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:24 smithi103 ceph-mon[34421]: pgmap v472: 129 pgs: 129 active+clean; 33 MiB data, 533 MiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 926 KiB/s wr, 110 op/s 2024-05-09T09:22:25.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 79250 1.74 MB/sec execute 23 sec latency 4709.336 ms 2024-05-09T09:22:26.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 81971 2.38 MB/sec execute 24 sec latency 12.230 ms 2024-05-09T09:22:26.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:26 smithi069 ceph-mon[30123]: pgmap v473: 129 pgs: 129 active+clean; 32 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:22:26.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:26 smithi163 ceph-mon[34354]: pgmap v473: 129 pgs: 129 active+clean; 32 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:22:26.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:26 smithi103 ceph-mon[34421]: pgmap v473: 129 pgs: 129 active+clean; 32 MiB data, 534 MiB used, 1.0 TiB / 1.0 TiB avail; 675 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:22:27.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 82426 2.30 MB/sec execute 25 sec latency 569.172 ms 2024-05-09T09:22:28.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 82426 2.21 MB/sec execute 26 sec latency 1569.260 ms 2024-05-09T09:22:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:28 smithi163 ceph-mon[34354]: pgmap v474: 129 pgs: 129 active+clean; 33 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:28 smithi103 ceph-mon[34421]: pgmap v474: 129 pgs: 129 active+clean; 33 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:28 smithi069 ceph-mon[30123]: pgmap v474: 129 pgs: 129 active+clean; 33 MiB data, 538 MiB used, 1.0 TiB / 1.0 TiB avail; 720 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:29.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 82426 2.13 MB/sec execute 27 sec latency 2569.313 ms 2024-05-09T09:22:30.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 83038 2.22 MB/sec execute 28 sec latency 3316.044 ms 2024-05-09T09:22:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:30 smithi163 ceph-mon[34354]: pgmap v475: 129 pgs: 129 active+clean; 34 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:22:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:30 smithi103 ceph-mon[34421]: pgmap v475: 129 pgs: 129 active+clean; 34 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:22:30.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:30 smithi069 ceph-mon[30123]: pgmap v475: 129 pgs: 129 active+clean; 34 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 823 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:22:31.480 INFO:tasks.workunit.client.0.smithi069.stdout: 1 85548 2.63 MB/sec execute 29 sec latency 21.673 ms 2024-05-09T09:22:32.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 87578 2.78 MB/sec execute 30 sec latency 15.808 ms 2024-05-09T09:22:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:32 smithi163 ceph-mon[34354]: pgmap v476: 129 pgs: 129 active+clean; 34 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:32 smithi103 ceph-mon[34421]: pgmap v476: 129 pgs: 129 active+clean; 34 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:32.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:32 smithi069 ceph-mon[30123]: pgmap v476: 129 pgs: 129 active+clean; 34 MiB data, 553 MiB used, 1.0 TiB / 1.0 TiB avail; 853 KiB/s rd, 1.1 MiB/s wr, 130 op/s 2024-05-09T09:22:33.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 89318 3.08 MB/sec execute 31 sec latency 14.701 ms 2024-05-09T09:22:34.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 89448 2.99 MB/sec execute 32 sec latency 903.361 ms 2024-05-09T09:22:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:34 smithi163 ceph-mon[34354]: pgmap v477: 129 pgs: 129 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T09:22:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:34 smithi103 ceph-mon[34421]: pgmap v477: 129 pgs: 129 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T09:22:34.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:34 smithi069 ceph-mon[30123]: pgmap v477: 129 pgs: 129 active+clean; 35 MiB data, 554 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 1.0 MiB/s wr, 137 op/s 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 -- 172.21.15.69:0/1896938097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 msgr2=0x7fa100109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 --2- 172.21.15.69:0/1896938097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa100109a00 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8009ca0 tx=0x7fa0f802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 -- 172.21.15.69:0/1896938097 shutdown_connections 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 --2- 172.21.15.69:0/1896938097 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa100109a00 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 --2- 172.21.15.69:0/1896938097 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 0x7fa1001070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 --2- 172.21.15.69:0/1896938097 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 0x7fa1001047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 -- 172.21.15.69:0/1896938097 >> 172.21.15.69:0/1896938097 conn(0x7fa1000fbd20 msgr2=0x7fa1000fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 -- 172.21.15.69:0/1896938097 shutdown_connections 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.209+0000 7fa104a15640 1 -- 172.21.15.69:0/1896938097 wait complete. 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 Processor -- start 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 -- start start 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 0x7fa100197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 0x7fa100197f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010d510 con 0x7fa100107610 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010d390 con 0x7fa100104ce0 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.210+0000 7fa104a15640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa10010d690 con 0x7fa1001023b0 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52156/0 (socket says 172.21.15.69:52156) 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 -- 172.21.15.69:0/1360728029 learned_addr learned my addr 172.21.15.69:0/1360728029 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 0x7fa100197f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 0x7fa100197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 -- 172.21.15.69:0/1360728029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 msgr2=0x7fa100197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 0x7fa100197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 -- 172.21.15.69:0/1360728029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 msgr2=0x7fa100197f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 0x7fa100197f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 -- 172.21.15.69:0/1360728029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa10019f660 con 0x7fa100107610 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0fed9d640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8005e30 tx=0x7fa0f8031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0f77fe640 1 -- 172.21.15.69:0/1360728029 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f8003c30 con 0x7fa100107610 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0f77fe640 1 -- 172.21.15.69:0/1360728029 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0f8003dd0 con 0x7fa100107610 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa10019f8f0 con 0x7fa100107610 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa0f77fe640 1 -- 172.21.15.69:0/1360728029 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0f802fde0 con 0x7fa100107610 2024-05-09T09:22:35.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.211+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa10019fe40 con 0x7fa100107610 2024-05-09T09:22:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa0f77fe640 1 -- 172.21.15.69:0/1360728029 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa0f8004050 con 0x7fa100107610 2024-05-09T09:22:35.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 Processor -- start 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 -- start start 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb41a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb46b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56470baa1340 con 0x56470bb471c0 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56470ba49290 con 0x56470bb42050 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa107800740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56470ba9d300 con 0x56470bb41660 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.214+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52164/0 (socket says 172.21.15.69:52164) 2024-05-09T09:22:35.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3282119771 learned_addr learned my addr 172.21.15.69:0/3282119771 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb46b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f77fe640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 0x7fa0ec0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb41a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0fed9d640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 0x7fa0ec0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3282119771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 msgr2=0x56470bb41a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb41a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3282119771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 msgr2=0x56470bb46b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb46b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3282119771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470ba6d8e0 con 0x56470bb471c0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb46b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f77fe640 1 -- 172.21.15.69:0/1360728029 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fa0f80f0840 con 0x7fa100107610 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8012530 tx=0x7fa0e800b9b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfa3118d7721e0e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0fed9d640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 0x7fa0ec0a50f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80ee010 tx=0x7fa0f808fde0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f4ff9640 1 -- 172.21.15.69:0/3282119771 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e801d5f0 con 0x56470bb471c0 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f4ff9640 1 -- 172.21.15.69:0/3282119771 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e8010910 con 0x56470bb471c0 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.215+0000 7fa0f4ff9640 1 -- 172.21.15.69:0/3282119771 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e801c9c0 con 0x56470bb471c0 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- 172.21.15.69:0/3282119771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 msgr2=0x56470bb49650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8012530 tx=0x7fa0e800b9b0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- 172.21.15.69:0/3282119771 shutdown_connections 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb49650 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb46b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 --2- 172.21.15.69:0/3282119771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb41a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- 172.21.15.69:0/3282119771 >> 172.21.15.69:0/3282119771 conn(0x56470bb388c0 msgr2=0x56470bb38cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- 172.21.15.69:0/3282119771 shutdown_connections 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- 172.21.15.69:0/3282119771 wait complete. 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.216+0000 7fa107800740 1 -- start start 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 Processor -- start 2024-05-09T09:22:35.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb41660 0x56470bb4b8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb42050 0x56470bb4be10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56470bb4a0e0 con 0x56470bb471c0 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56470ba5c790 con 0x56470bb41660 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56470bb4a260 con 0x56470bb42050 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa107800740 10 client.0 _ll_register_callbacks cb 0x56470ba97a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52180/0 (socket says 172.21.15.69:52180) 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 -- 172.21.15.69:0/2156720190 learned_addr learned my addr 172.21.15.69:0/2156720190 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb41660 0x56470bb4b8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb42050 0x56470bb4be10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 -- 172.21.15.69:0/2156720190 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb42050 msgr2=0x56470bb4be10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb42050 0x56470bb4be10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 -- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb41660 msgr2=0x56470bb4b8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb41660 0x56470bb4b8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 -- 172.21.15.69:0/2156720190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470b7b5210 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005780 tx=0x7fa0e800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f198fcdd1fbc62a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.217+0000 7fa0e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8004020 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0e8010910 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fa0e80041c0 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa0e80043c0 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 10 client.16326 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56470b881620 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e8010910 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fa0e8028070 con 0x56470bb471c0 2024-05-09T09:22:35.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 10 client.16326 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 10 client.16326 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 20 client.16326 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bb57140 0x56470bb59580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 4 client.16326 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.218+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56470baaea70 con 0x56470bb57140 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.219+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bb57140 0x56470bb59580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.219+0000 7fa0e5ffb640 10 client.16326 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.219+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bb57140 0x56470bb59580 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82835113ec74bb8d server_cookie=8d07a34d2cc7f776 in_seq=0 out_seq=0 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/2156720190 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56470baaea70 con 0x56470bb57140 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa0e5ffb640 10 client.16326 handle_command_reply: tid=0 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 client.16326 shutdown 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 msgr2=0x56470bb52de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005780 tx=0x7fa0e800b390 comp rx=0 tx=0).stop 2024-05-09T09:22:35.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 shutdown_connections 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb52de0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bb57140 0x56470bb59580 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb42050 0x56470bb4be10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 --2- 172.21.15.69:0/2156720190 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb41660 0x56470bb4b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa0e5ffb640 0 client.16326 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 >> 172.21.15.69:0/2156720190 conn(0x56470bb388c0 msgr2=0x56470bb37a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 shutdown_connections 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 1 -- 172.21.15.69:0/2156720190 wait complete. 2024-05-09T09:22:35.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.220+0000 7fa107800740 20 client.16326 trim_cache size 0 max 16384 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 Processor -- start 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 -- start start 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb5ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.236+0000 7fa107800740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001aa9a0 con 0x56470bb471c0 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa107800740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a8210 con 0x56470bb42050 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa107800740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a6060 con 0x56470bb41660 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb5ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0fe59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:46724/0 (socket says 172.21.15.69:46724) 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0fe59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:41064/0 (socket says 172.21.15.69:41064) 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3731950203 learned_addr learned my addr 172.21.15.69:0/3731950203 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3731950203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 msgr2=0x56470bb52d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3731950203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 msgr2=0x56470bb5ee50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb5ee50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 -- 172.21.15.69:0/3731950203 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56470bb33bc0 con 0x56470bb41660 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005c40 tx=0x7fa0e8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99ba68bf537eaf16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0e7fff640 1 -- 172.21.15.69:0/3731950203 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8010890 con 0x56470bb41660 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.237+0000 7fa0e7fff640 1 -- 172.21.15.69:0/3731950203 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e8010d50 con 0x56470bb41660 2024-05-09T09:22:35.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa0e7fff640 1 -- 172.21.15.69:0/3731950203 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8005260 con 0x56470bb41660 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 -- 172.21.15.69:0/3731950203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 msgr2=0x56470bb38ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa0e8005c40 tx=0x7fa0e8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 -- 172.21.15.69:0/3731950203 shutdown_connections 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb471c0 0x56470bb5ee50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb52d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 --2- 172.21.15.69:0/3731950203 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb41660 0x56470bb38ca0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 -- 172.21.15.69:0/3731950203 >> 172.21.15.69:0/3731950203 conn(0x56470bb388c0 msgr2=0x56470bb4a730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 -- 172.21.15.69:0/3731950203 shutdown_connections 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.238+0000 7fa107800740 1 -- 172.21.15.69:0/3731950203 wait complete. 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 -- start start 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 Processor -- start 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb50c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb471c0 0x56470bb51c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a66d0 con 0x56470bb41660 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a7b40 con 0x56470bb42050 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa1001a6da0 con 0x56470bb471c0 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa107800740 10 client.0 _ll_register_callbacks cb 0x7fa0f000b3a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52190/0 (socket says 172.21.15.69:52190) 2024-05-09T09:22:35.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 -- 172.21.15.69:0/1140772026 learned_addr learned my addr 172.21.15.69:0/1140772026 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb50c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb471c0 0x56470bb51c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 -- 172.21.15.69:0/1140772026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb471c0 msgr2=0x56470bb51c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb471c0 0x56470bb51c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 -- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 msgr2=0x56470bb50c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb50c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 -- 172.21.15.69:0/1140772026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56470bb52220 con 0x56470bb41660 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0f7fff640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800b9f0 tx=0x7fa0e800b840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=356e1e5bcb5e94b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.239+0000 7fa0e5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa0e8005030 con 0x56470bb41660 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fa0e80051d0 con 0x56470bb41660 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa0e8010d50 con 0x56470bb41660 2024-05-09T09:22:35.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=32) ==== 24+0+0 (secure 0 0 0) 0x7fa0e8057050 con 0x56470bb41660 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 10 client.16332 fetch_fsmap learned FSMap version 32 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=32,monmap=4+}) -- 0x56470bbbd840 con 0x56470bb41660 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa0e8010d50 con 0x56470bb41660 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 32) ==== 8354+0+0 (secure 0 0 0) 0x7fa0e8028070 con 0x56470bb41660 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 10 client.16332 fetch_fsmap finished waiting for FSMap version 32 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 10 client.16332 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 20 client.16332 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bbbe1e0 0x56470bbc0600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 4 client.16332 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:22:35.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56470baa8dc0 con 0x56470bbbe1e0 2024-05-09T09:22:35.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.240+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bbbe1e0 0x56470bbc0600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:35.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.241+0000 7fa0e5ffb640 10 client.16332 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:35.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.241+0000 7fa0fe59c640 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bbbe1e0 0x56470bbc0600 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7251cd58e7ef2fa server_cookie=864f1f124fc78683 in_seq=0 out_seq=0 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa0e5ffb640 1 -- 172.21.15.69:0/1140772026 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56470baa8dc0 con 0x56470bbbe1e0 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa0e5ffb640 10 client.16332 handle_command_reply: tid=0 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 client.16332 shutdown 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 msgr2=0x56470bb506e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7fa0e800b9f0 tx=0x7fa0e800b840 comp rx=0 tx=0).stop 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 shutdown_connections 2024-05-09T09:22:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56470bbbe1e0 0x56470bbc0600 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56470bb471c0 0x56470bb51c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56470bb42050 0x56470bb50c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 --2- 172.21.15.69:0/1140772026 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56470bb41660 0x56470bb506e0 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 >> 172.21.15.69:0/1140772026 conn(0x56470bb388c0 msgr2=0x56470bb38ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.312+0000 7fa0e5ffb640 0 client.16332 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 shutdown_connections 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa107800740 1 -- 172.21.15.69:0/1140772026 wait complete. 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa107800740 20 client.16332 trim_cache size 0 max 16384 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:22:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:22:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:22:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:22:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 msgr2=0x7fa0ec0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 0x7fa0ec0a50f0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fa0f80ee010 tx=0x7fa0f808fde0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 msgr2=0x7fa10019eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7fa0f8005e30 tx=0x7fa0f8031ee0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 shutdown_connections 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa0ec0a2c30 0x7fa0ec0a50f0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa100107610 0x7fa10019eff0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa100104ce0 0x7fa100197f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 --2- 172.21.15.69:0/1360728029 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa1001023b0 0x7fa100197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 >> 172.21.15.69:0/1360728029 conn(0x7fa1000fbd20 msgr2=0x7fa100102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.313+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 shutdown_connections 2024-05-09T09:22:35.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:35.314+0000 7fa104a15640 1 -- 172.21.15.69:0/1360728029 wait complete. 2024-05-09T09:22:35.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 89798 2.95 MB/sec execute 33 sec latency 1651.216 ms 2024-05-09T09:22:36.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 92508 3.36 MB/sec execute 34 sec latency 14.643 ms 2024-05-09T09:22:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:36 smithi163 ceph-mon[34354]: pgmap v478: 129 pgs: 129 active+clean; 36 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 637 KiB/s wr, 102 op/s 2024-05-09T09:22:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:36 smithi103 ceph-mon[34421]: pgmap v478: 129 pgs: 129 active+clean; 36 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 637 KiB/s wr, 102 op/s 2024-05-09T09:22:36.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:36 smithi069 ceph-mon[30123]: pgmap v478: 129 pgs: 129 active+clean; 36 MiB data, 561 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 637 KiB/s wr, 102 op/s 2024-05-09T09:22:37.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 92992 3.28 MB/sec execute 35 sec latency 502.151 ms 2024-05-09T09:22:38.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 92992 3.19 MB/sec execute 36 sec latency 1502.206 ms 2024-05-09T09:22:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:38 smithi163 ceph-mon[34354]: pgmap v479: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 634 KiB/s wr, 155 op/s 2024-05-09T09:22:38.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:38 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:22:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:38 smithi103 ceph-mon[34421]: pgmap v479: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 634 KiB/s wr, 155 op/s 2024-05-09T09:22:38.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:38 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:22:38.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:38 smithi069 ceph-mon[30123]: pgmap v479: 129 pgs: 129 active+clean; 34 MiB data, 545 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 634 KiB/s wr, 155 op/s 2024-05-09T09:22:38.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:38 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:22:39.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 92992 3.10 MB/sec execute 37 sec latency 2502.254 ms 2024-05-09T09:22:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:39 smithi103 ceph-mon[34421]: pgmap v480: 129 pgs: 129 active+clean; 35 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 577 KiB/s wr, 140 op/s 2024-05-09T09:22:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:39 smithi069 ceph-mon[30123]: pgmap v480: 129 pgs: 129 active+clean; 35 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 577 KiB/s wr, 140 op/s 2024-05-09T09:22:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:39 smithi163 ceph-mon[34354]: pgmap v480: 129 pgs: 129 active+clean; 35 MiB data, 530 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 577 KiB/s wr, 140 op/s 2024-05-09T09:22:40.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 93452 3.07 MB/sec execute 38 sec latency 3247.496 ms 2024-05-09T09:22:41.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 96102 3.42 MB/sec execute 39 sec latency 18.653 ms 2024-05-09T09:22:42.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 96539 3.35 MB/sec execute 40 sec latency 558.224 ms 2024-05-09T09:22:42.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:42 smithi069 ceph-mon[30123]: pgmap v481: 129 pgs: 129 active+clean; 35 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 623 KiB/s wr, 165 op/s 2024-05-09T09:22:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:42 smithi163 ceph-mon[34354]: pgmap v481: 129 pgs: 129 active+clean; 35 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 623 KiB/s wr, 165 op/s 2024-05-09T09:22:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:42 smithi103 ceph-mon[34421]: pgmap v481: 129 pgs: 129 active+clean; 35 MiB data, 520 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 623 KiB/s wr, 165 op/s 2024-05-09T09:22:43.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 96539 3.27 MB/sec execute 41 sec latency 1558.345 ms 2024-05-09T09:22:44.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 96539 3.19 MB/sec execute 42 sec latency 2558.448 ms 2024-05-09T09:22:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:44 smithi163 ceph-mon[34354]: pgmap v482: 129 pgs: 129 active+clean; 36 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 KiB/s wr, 165 op/s 2024-05-09T09:22:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:44 smithi103 ceph-mon[34421]: pgmap v482: 129 pgs: 129 active+clean; 36 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 KiB/s wr, 165 op/s 2024-05-09T09:22:44.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:44 smithi069 ceph-mon[30123]: pgmap v482: 129 pgs: 129 active+clean; 36 MiB data, 509 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 652 KiB/s wr, 165 op/s 2024-05-09T09:22:45.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 96980 3.16 MB/sec execute 43 sec latency 3308.550 ms 2024-05-09T09:22:46.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 99640 3.47 MB/sec execute 44 sec latency 12.016 ms 2024-05-09T09:22:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:46 smithi163 ceph-mon[34354]: pgmap v483: 129 pgs: 129 active+clean; 36 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 560 KiB/s wr, 145 op/s 2024-05-09T09:22:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:46 smithi103 ceph-mon[34421]: pgmap v483: 129 pgs: 129 active+clean; 36 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 560 KiB/s wr, 145 op/s 2024-05-09T09:22:46.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:46 smithi069 ceph-mon[30123]: pgmap v483: 129 pgs: 129 active+clean; 36 MiB data, 510 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 560 KiB/s wr, 145 op/s 2024-05-09T09:22:47.481 INFO:tasks.workunit.client.0.smithi069.stdout: 1 100084 3.40 MB/sec execute 45 sec latency 544.647 ms 2024-05-09T09:22:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 -- 172.21.15.69:0/1653443012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c107660 msgr2=0x7fe07c109a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 --2- 172.21.15.69:0/1653443012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c107660 0x7fe07c109a50 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7fe074009ca0 tx=0x7fe07402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 -- 172.21.15.69:0/1653443012 shutdown_connections 2024-05-09T09:22:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 --2- 172.21.15.69:0/1653443012 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c107660 0x7fe07c109a50 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 --2- 172.21.15.69:0/1653443012 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c104d30 0x7fe07c107120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 --2- 172.21.15.69:0/1653443012 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c102400 0x7fe07c1047f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 -- 172.21.15.69:0/1653443012 >> 172.21.15.69:0/1653443012 conn(0x7fe07c0fbd30 msgr2=0x7fe07c0fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 -- 172.21.15.69:0/1653443012 shutdown_connections 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.405+0000 7fe081b41640 1 -- 172.21.15.69:0/1653443012 wait complete. 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 Processor -- start 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 -- start start 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c102400 0x7fe07c197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 0x7fe07c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c10d570 con 0x7fe07c104d30 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c10d3f0 con 0x7fe07c107660 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe081b41640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c10d6f0 con 0x7fe07c102400 2024-05-09T09:22:48.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe07affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe07affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59354/0 (socket says 172.21.15.69:59354) 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.406+0000 7fe07affd640 1 -- 172.21.15.69:0/846159641 learned_addr learned my addr 172.21.15.69:0/846159641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07bfff640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 0x7fe07c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c102400 0x7fe07c197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 -- 172.21.15.69:0/846159641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c102400 msgr2=0x7fe07c197a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c102400 0x7fe07c197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 -- 172.21.15.69:0/846159641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 msgr2=0x7fe07c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 0x7fe07c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 -- 172.21.15.69:0/846159641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe07c19f680 con 0x7fe07c104d30 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07bfff640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 0x7fe07c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe07affd640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe07000d740 tx=0x7fe07000dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe078ff9640 1 -- 172.21.15.69:0/846159641 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070017070 con 0x7fe07c104d30 2024-05-09T09:22:48.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe07c19f970 con 0x7fe07c104d30 2024-05-09T09:22:48.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe078ff9640 1 -- 172.21.15.69:0/846159641 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe07000de50 con 0x7fe07c104d30 2024-05-09T09:22:48.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe078ff9640 1 -- 172.21.15.69:0/846159641 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe070004e00 con 0x7fe07c104d30 2024-05-09T09:22:48.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.407+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe07c19fec0 con 0x7fe07c104d30 2024-05-09T09:22:48.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.409+0000 7fe078ff9640 1 -- 172.21.15.69:0/846159641 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe070004fa0 con 0x7fe07c104d30 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.409+0000 7fe084845740 1 Processor -- start 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 -- start start 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 0x556f14819670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 0x556f14821240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556f1477d340 con 0x556f14819c40 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556f14725290 con 0x556f1481edb0 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe084845740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556f14779300 con 0x556f14819250 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59368/0 (socket says 172.21.15.69:59368) 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 -- 172.21.15.69:0/4210393462 learned_addr learned my addr 172.21.15.69:0/4210393462 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe078ff9640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 0x7fe0640a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07bfff640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 0x556f14819670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 -- 172.21.15.69:0/4210393462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 msgr2=0x556f14819670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 0x556f14819670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 -- 172.21.15.69:0/4210393462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 msgr2=0x556f14821240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07affd640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 0x556f14821240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 0x556f14821240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 -- 172.21.15.69:0/4210393462 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f147498e0 con 0x556f14819c40 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07bfff640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 0x556f14819670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe078ff9640 1 -- 172.21.15.69:0/846159641 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe0700cee40 con 0x7fe07c104d30 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07affd640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 0x556f14821240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07bfff640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 0x7fe0640a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.410+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00bb10 tx=0x7fe06c00d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed194dd1fc26ba0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe061ffb640 1 -- 172.21.15.69:0/4210393462 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c017420 con 0x556f14819c40 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe061ffb640 1 -- 172.21.15.69:0/4210393462 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06c017aa0 con 0x556f14819c40 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe061ffb640 1 -- 172.21.15.69:0/4210393462 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c00fb20 con 0x556f14819c40 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 -- 172.21.15.69:0/4210393462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 msgr2=0x556f1481e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe07bfff640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 0x7fe0640a50f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe074005cc0 tx=0x7fe074005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00bb10 tx=0x7fe06c00d340 comp rx=0 tx=0).stop 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 -- 172.21.15.69:0/4210393462 shutdown_connections 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f1481edb0 0x556f14821240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819c40 0x556f1481e740 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 --2- 172.21.15.69:0/4210393462 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819250 0x556f14819670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 -- 172.21.15.69:0/4210393462 >> 172.21.15.69:0/4210393462 conn(0x556f148148c0 msgr2=0x556f14814cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 -- 172.21.15.69:0/4210393462 shutdown_connections 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.411+0000 7fe084845740 1 -- 172.21.15.69:0/4210393462 wait complete. 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 -- start start 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 Processor -- start 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f14823430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f14823970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556f14821d60 con 0x556f1481edb0 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556f14738790 con 0x556f14819250 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556f14821ee0 con 0x556f14819c40 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe084845740 10 client.0 _ll_register_callbacks cb 0x556f14773a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59376/0 (socket says 172.21.15.69:59376) 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe07bfff640 1 -- 172.21.15.69:0/334052731 learned_addr learned my addr 172.21.15.69:0/334052731 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f14823430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.412+0000 7fe07affd640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f14823970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 -- 172.21.15.69:0/334052731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 msgr2=0x556f14823970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f14823970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 -- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 msgr2=0x556f14823430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f14823430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 -- 172.21.15.69:0/334052731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f14491210 con 0x556f1481edb0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f14823430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe07bfff640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe074035000 tx=0x7fe0740095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc4fd08dc6cec7fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07404d070 con 0x556f1481edb0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe074004450 con 0x556f1481edb0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe06c017aa0 con 0x556f1481edb0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe074059050 con 0x556f1481edb0 2024-05-09T09:22:48.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 10 client.16344 fetch_fsmap learned FSMap version 33 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556f1455d620 con 0x556f1481edb0 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe06c017aa0 con 0x556f1481edb0 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe0740305c0 con 0x556f1481edb0 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 10 client.16344 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 10 client.16344 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 20 client.16344 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.413+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f1482ef50 0x556f14831410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.414+0000 7fe084845740 4 client.16344 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.414+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556f1478aa70 con 0x556f1482ef50 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.414+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f1482ef50 0x556f14831410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.414+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f1482ef50 0x556f14831410 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2f37a62b2047c4f server_cookie=2fce5d056b41686d in_seq=0 out_seq=0 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.414+0000 7fe05affd640 10 client.16344 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe05affd640 1 -- 172.21.15.69:0/334052731 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556f1478aa70 con 0x556f1482ef50 2024-05-09T09:22:48.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe05affd640 10 client.16344 handle_command_reply: tid=0 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 client.16344 shutdown 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 msgr2=0x556f1482a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7fe074035000 tx=0x7fe0740095c0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 shutdown_connections 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1482a9b0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f1482ef50 0x556f14831410 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f14823970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 --2- 172.21.15.69:0/334052731 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f14823430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe05affd640 0 client.16344 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 >> 172.21.15.69:0/334052731 conn(0x556f148148c0 msgr2=0x556f14813880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 shutdown_connections 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 1 -- 172.21.15.69:0/334052731 wait complete. 2024-05-09T09:22:48.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.415+0000 7fe084845740 20 client.16344 trim_cache size 0 max 16384 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 Processor -- start 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 -- start start 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f148211a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f1482a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1acb70 con 0x556f1481edb0 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1aa440 con 0x556f14819250 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe084845740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1a8290 con 0x556f14819c40 2024-05-09T09:22:48.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59390/0 (socket says 172.21.15.69:59390) 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 -- 172.21.15.69:0/1940796583 learned_addr learned my addr 172.21.15.69:0/1940796583 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 -- 172.21.15.69:0/1940796583 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 msgr2=0x556f1482a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f1482a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 -- 172.21.15.69:0/1940796583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 msgr2=0x556f148211a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f148211a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 -- 172.21.15.69:0/1940796583 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556f1482d840 con 0x556f1481edb0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe07bfff640 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe074009bf0 tx=0x7fe074005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=52ac86da2ee1e46a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.430+0000 7fe0617fa640 1 -- 172.21.15.69:0/1940796583 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07404d070 con 0x556f1481edb0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe0617fa640 1 -- 172.21.15.69:0/1940796583 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe074062070 con 0x556f1481edb0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe0617fa640 1 -- 172.21.15.69:0/1940796583 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe07402f870 con 0x556f1481edb0 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- 172.21.15.69:0/1940796583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 msgr2=0x556f1481e3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7fe074009bf0 tx=0x7fe074005c50 comp rx=0 tx=0).stop 2024-05-09T09:22:48.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- 172.21.15.69:0/1940796583 shutdown_connections 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f1481edb0 0x556f1481e3f0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f14819c40 0x556f1482a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 --2- 172.21.15.69:0/1940796583 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819250 0x556f148211a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- 172.21.15.69:0/1940796583 >> 172.21.15.69:0/1940796583 conn(0x556f148148c0 msgr2=0x556f14814ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- 172.21.15.69:0/1940796583 shutdown_connections 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- 172.21.15.69:0/1940796583 wait complete. 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.431+0000 7fe084845740 1 -- start start 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 Processor -- start 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819c40 0x556f14827f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f1481edb0 0x556f14828f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1a8900 con 0x556f14819250 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1a9d70 con 0x556f14819c40 2024-05-09T09:22:48.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe07c1a8fd0 con 0x556f1481edb0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe084845740 10 client.0 _ll_register_callbacks cb 0x7fe074009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59392/0 (socket says 172.21.15.69:59392) 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 -- 172.21.15.69:0/1839709139 learned_addr learned my addr 172.21.15.69:0/1839709139 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07affd640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819c40 0x556f14827f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 -- 172.21.15.69:0/1839709139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f1481edb0 msgr2=0x556f14828f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f1481edb0 0x556f14828f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 -- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819c40 msgr2=0x556f14827f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819c40 0x556f14827f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 -- 172.21.15.69:0/1839709139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556f14829550 con 0x556f14819250 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe07b7fe640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00c010 tx=0x7fe06c00f4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c87e5c8c9dea605 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe05affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.432+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe06c027b50 con 0x556f14819250 2024-05-09T09:22:48.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe074062070 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7fe06c026070 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7fe06c034050 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 10 client.16350 fetch_fsmap learned FSMap version 33 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x556f1482a8d0 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe074062070 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7fe06c00f7e0 con 0x556f14819250 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 10 client.16350 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 10 client.16350 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 20 client.16350 populate_metadata read hostname 'smithi069' 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f148957c0 0x556f14897bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 4 client.16350 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556f14784dc0 con 0x556f148957c0 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.433+0000 7fe07affd640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f148957c0 0x556f14897bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:22:48.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.434+0000 7fe07affd640 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f148957c0 0x556f14897bb0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a2ea754111d08d9 server_cookie=6d0195be52177ca0 in_seq=0 out_seq=0 2024-05-09T09:22:48.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.434+0000 7fe05affd640 10 client.16350 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:48.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 100084 3.33 MB/sec execute 46 sec latency 1544.695 ms 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe05affd640 1 -- 172.21.15.69:0/1839709139 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556f14784dc0 con 0x556f148957c0 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe05affd640 10 client.16350 handle_command_reply: tid=0 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 client.16350 shutdown 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 msgr2=0x556f14827a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7fe06c00c010 tx=0x7fe06c00f4d0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 shutdown_connections 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556f148957c0 0x556f14897bb0 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556f1481edb0 0x556f14828f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556f14819c40 0x556f14827f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 --2- 172.21.15.69:0/1839709139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556f14819250 0x556f14827a10 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 >> 172.21.15.69:0/1839709139 conn(0x556f148148c0 msgr2=0x556f1481dfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.484+0000 7fe05affd640 0 client.16350 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 shutdown_connections 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe084845740 1 -- 172.21.15.69:0/1839709139 wait complete. 2024-05-09T09:22:48.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe084845740 20 client.16350 trim_cache size 0 max 16384 2024-05-09T09:22:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:22:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:22:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:22:48.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 msgr2=0x7fe0640a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 0x7fe0640a50f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fe074005cc0 tx=0x7fe074005d80 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 msgr2=0x7fe07c197fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7fe07000d740 tx=0x7fe07000dc00 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 shutdown_connections 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0640a2c30 0x7fe0640a50f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe07c107660 0x7fe07c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe07c104d30 0x7fe07c197fa0 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 --2- 172.21.15.69:0/846159641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe07c102400 0x7fe07c197a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:22:48.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 >> 172.21.15.69:0/846159641 conn(0x7fe07c0fbd30 msgr2=0x7fe07c102d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:22:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 shutdown_connections 2024-05-09T09:22:48.489 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:22:48.485+0000 7fe081b41640 1 -- 172.21.15.69:0/846159641 wait complete. 2024-05-09T09:22:48.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:48 smithi163 ceph-mon[34354]: pgmap v484: 129 pgs: 129 active+clean; 38 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 638 KiB/s wr, 169 op/s 2024-05-09T09:22:48.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:48 smithi103 ceph-mon[34421]: pgmap v484: 129 pgs: 129 active+clean; 38 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 638 KiB/s wr, 169 op/s 2024-05-09T09:22:48.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:48 smithi069 ceph-mon[30123]: pgmap v484: 129 pgs: 129 active+clean; 38 MiB data, 512 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 638 KiB/s wr, 169 op/s 2024-05-09T09:22:49.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 100084 3.26 MB/sec execute 47 sec latency 2544.823 ms 2024-05-09T09:22:50.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 100525 3.23 MB/sec execute 48 sec latency 3296.507 ms 2024-05-09T09:22:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:50 smithi163 ceph-mon[34354]: pgmap v485: 129 pgs: 129 active+clean; 38 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 404 KiB/s wr, 103 op/s 2024-05-09T09:22:50.562 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:50 smithi069 ceph-mon[30123]: pgmap v485: 129 pgs: 129 active+clean; 38 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 404 KiB/s wr, 103 op/s 2024-05-09T09:22:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:50 smithi103 ceph-mon[34421]: pgmap v485: 129 pgs: 129 active+clean; 38 MiB data, 516 MiB used, 1.0 TiB / 1.0 TiB avail; 978 KiB/s rd, 404 KiB/s wr, 103 op/s 2024-05-09T09:22:51.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 103184 3.50 MB/sec execute 49 sec latency 10.109 ms 2024-05-09T09:22:52.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 103700 3.45 MB/sec execute 50 sec latency 438.237 ms 2024-05-09T09:22:52.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:52 smithi163 ceph-mon[34354]: pgmap v486: 129 pgs: 129 active+clean; 39 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 477 KiB/s wr, 115 op/s 2024-05-09T09:22:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:52 smithi103 ceph-mon[34421]: pgmap v486: 129 pgs: 129 active+clean; 39 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 477 KiB/s wr, 115 op/s 2024-05-09T09:22:52.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:52 smithi069 ceph-mon[30123]: pgmap v486: 129 pgs: 129 active+clean; 39 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 920 KiB/s rd, 477 KiB/s wr, 115 op/s 2024-05-09T09:22:53.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 103700 3.38 MB/sec execute 51 sec latency 1438.284 ms 2024-05-09T09:22:54.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 103700 3.31 MB/sec execute 52 sec latency 2438.371 ms 2024-05-09T09:22:54.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:54 smithi163 ceph-mon[34354]: pgmap v487: 129 pgs: 129 active+clean; 39 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 468 KiB/s wr, 106 op/s 2024-05-09T09:22:54.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:54 smithi103 ceph-mon[34421]: pgmap v487: 129 pgs: 129 active+clean; 39 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 468 KiB/s wr, 106 op/s 2024-05-09T09:22:54.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:54 smithi069 ceph-mon[30123]: pgmap v487: 129 pgs: 129 active+clean; 39 MiB data, 539 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 468 KiB/s wr, 106 op/s 2024-05-09T09:22:55.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 104184 3.28 MB/sec execute 53 sec latency 3193.178 ms 2024-05-09T09:22:56.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 106822 3.54 MB/sec execute 54 sec latency 11.172 ms 2024-05-09T09:22:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:56 smithi163 ceph-mon[34354]: pgmap v488: 129 pgs: 129 active+clean; 39 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 511 KiB/s wr, 110 op/s 2024-05-09T09:22:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:56 smithi103 ceph-mon[34421]: pgmap v488: 129 pgs: 129 active+clean; 39 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 511 KiB/s wr, 110 op/s 2024-05-09T09:22:56.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:56 smithi069 ceph-mon[30123]: pgmap v488: 129 pgs: 129 active+clean; 39 MiB data, 558 MiB used, 1.0 TiB / 1.0 TiB avail; 991 KiB/s rd, 511 KiB/s wr, 110 op/s 2024-05-09T09:22:57.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 107241 3.48 MB/sec execute 55 sec latency 566.823 ms 2024-05-09T09:22:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:22:57 smithi163 ceph-mon[34354]: pgmap v489: 129 pgs: 129 active+clean; 41 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 717 KiB/s wr, 141 op/s 2024-05-09T09:22:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:22:57 smithi103 ceph-mon[34421]: pgmap v489: 129 pgs: 129 active+clean; 41 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 717 KiB/s wr, 141 op/s 2024-05-09T09:22:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:22:57 smithi069 ceph-mon[30123]: pgmap v489: 129 pgs: 129 active+clean; 41 MiB data, 582 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 717 KiB/s wr, 141 op/s 2024-05-09T09:22:58.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 107241 3.42 MB/sec execute 56 sec latency 1566.874 ms 2024-05-09T09:22:59.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 107241 3.36 MB/sec execute 57 sec latency 2566.973 ms 2024-05-09T09:23:00.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 107709 3.33 MB/sec execute 58 sec latency 3318.411 ms 2024-05-09T09:23:00.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:00 smithi163 ceph-mon[34354]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 696 KiB/s wr, 121 op/s 2024-05-09T09:23:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:00 smithi103 ceph-mon[34421]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 696 KiB/s wr, 121 op/s 2024-05-09T09:23:00.759 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:00 smithi069 ceph-mon[30123]: pgmap v490: 129 pgs: 129 active+clean; 41 MiB data, 570 MiB used, 1.0 TiB / 1.0 TiB avail; 1015 KiB/s rd, 696 KiB/s wr, 121 op/s 2024-05-09T09:23:01.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110321 3.56 MB/sec execute 59 sec latency 13.711 ms 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.578+0000 7ff5df04d640 1 -- 172.21.15.69:0/4098624996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8101980 msgr2=0x7ff5d8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.578+0000 7ff5df04d640 1 --2- 172.21.15.69:0/4098624996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8101980 0x7ff5d8101e00 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4009900 tx=0x7ff5c402a410 comp rx=0 tx=0).stop 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.578+0000 7ff5df04d640 1 -- 172.21.15.69:0/4098624996 shutdown_connections 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.578+0000 7ff5df04d640 1 --2- 172.21.15.69:0/4098624996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8102340 0x7ff5d8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 --2- 172.21.15.69:0/4098624996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8101980 0x7ff5d8101e00 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 --2- 172.21.15.69:0/4098624996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8100790 0x7ff5d8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- 172.21.15.69:0/4098624996 >> 172.21.15.69:0/4098624996 conn(0x7ff5d80fbd00 msgr2=0x7ff5d80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- 172.21.15.69:0/4098624996 shutdown_connections 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- 172.21.15.69:0/4098624996 wait complete. 2024-05-09T09:23:01.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 Processor -- start 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- start start 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8100790 0x7ff5d8197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8101980 0x7ff5d8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d810c440 con 0x7ff5d8102340 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d810c2c0 con 0x7ff5d8101980 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5df04d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d810c5c0 con 0x7ff5d8100790 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.579+0000 7ff5dd5ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5d7fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8101980 0x7ff5d8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60560/0 (socket says 172.21.15.69:60560) 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 -- 172.21.15.69:0/2417132091 learned_addr learned my addr 172.21.15.69:0/2417132091 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8100790 0x7ff5d8197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 -- 172.21.15.69:0/2417132091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8100790 msgr2=0x7ff5d8197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8100790 0x7ff5d8197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 -- 172.21.15.69:0/2417132091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8101980 msgr2=0x7ff5d8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8101980 0x7ff5d8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 -- 172.21.15.69:0/2417132091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff5d819f710 con 0x7ff5d8102340 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000eaa0 tx=0x7ff5d000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5d5ffb640 1 -- 172.21.15.69:0/2417132091 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d000cdb0 con 0x7ff5d8102340 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5d5ffb640 1 -- 172.21.15.69:0/2417132091 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5d0005020 con 0x7ff5d8102340 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff5d819fa00 con 0x7ff5d8102340 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5d5ffb640 1 -- 172.21.15.69:0/2417132091 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00116b0 con 0x7ff5d8102340 2024-05-09T09:23:01.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.580+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff5d819fed0 con 0x7ff5d8102340 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5d5ffb640 1 -- 172.21.15.69:0/2417132091 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff5d0005560 con 0x7ff5d8102340 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 Processor -- start 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 -- start start 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5d5ffb640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 0x7ff5c80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755341b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a755344650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a75529c340 con 0x55a75533c660 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a755244290 con 0x55a7553421c0 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a755298300 con 0x55a75533d050 2024-05-09T09:23:01.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 0x7ff5c80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5d5ffb640 1 -- 172.21.15.69:0/2417132091 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7ff5d00cf590 con 0x7ff5d8102340 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.583+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60576/0 (socket says 172.21.15.69:60576) 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 -- 172.21.15.69:0/37510115 learned_addr learned my addr 172.21.15.69:0/37510115 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a755344650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755341b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 -- 172.21.15.69:0/37510115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 msgr2=0x55a755341b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755341b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 -- 172.21.15.69:0/37510115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 msgr2=0x55a755344650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a755344650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 -- 172.21.15.69:0/37510115 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a7552688e0 con 0x55a75533c660 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755341b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 0x7ff5c80a50f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4031440 tx=0x7ff5c4004490 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009c80 tx=0x7ff5cc011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=236d2b079a93ee4e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5c2ffd640 1 -- 172.21.15.69:0/37510115 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc01e070 con 0x55a75533c660 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5c2ffd640 1 -- 172.21.15.69:0/37510115 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc009e30 con 0x55a75533c660 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.584+0000 7ff5c2ffd640 1 -- 172.21.15.69:0/37510115 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc00eac0 con 0x55a75533c660 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- 172.21.15.69:0/37510115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 msgr2=0x55a75533ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc009c80 tx=0x7ff5cc011810 comp rx=0 tx=0).stop 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- 172.21.15.69:0/37510115 shutdown_connections 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a755344650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755341b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/37510115 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75533ca80 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- 172.21.15.69:0/37510115 >> 172.21.15.69:0/37510115 conn(0x55a7553338c0 msgr2=0x55a755333cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- 172.21.15.69:0/37510115 shutdown_connections 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- 172.21.15.69:0/37510115 wait complete. 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 -- start start 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.585+0000 7ff5e19fd740 1 Processor -- start 2024-05-09T09:23:01.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755346d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534dd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a755345080 con 0x55a75533c660 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a755257790 con 0x55a7553421c0 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a755345200 con 0x55a75533d050 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 10 client.0 _ll_register_callbacks cb 0x55a755292a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60584/0 (socket says 172.21.15.69:60584) 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 -- 172.21.15.69:0/4287485077 learned_addr learned my addr 172.21.15.69:0/4287485077 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:01.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755346d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 -- 172.21.15.69:0/4287485077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 msgr2=0x55a755346d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755346d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 -- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 msgr2=0x55a75534dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 -- 172.21.15.69:0/4287485077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a754fb0210 con 0x55a75533c660 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0097b0 tx=0x7ff5cc005720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5be3a2b68c11d11b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc01e040 con 0x55a75533c660 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7ff5cc004230 con 0x55a75533c660 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.586+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5cc009e30 con 0x55a75533c660 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff5cc033050 con 0x55a75533c660 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 10 client.16362 fetch_fsmap learned FSMap version 33 2024-05-09T09:23:01.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a75507c620 con 0x55a75533c660 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc009e30 con 0x55a75533c660 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7ff5cc004a70 con 0x55a75533c660 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 10 client.16362 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 10 client.16362 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 20 client.16362 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a755352040 0x55a755354450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 4 client.16362 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a7552a9a70 con 0x55a755352040 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.587+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a755352040 0x55a755354450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.588+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a755352040 0x55a755354450 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3f301c97e2bfa3b server_cookie=364a385fac4e26bd in_seq=0 out_seq=0 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.588+0000 7ff5b7fff640 10 client.16362 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.588+0000 7ff5b7fff640 1 -- 172.21.15.69:0/4287485077 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a7552a9a70 con 0x55a755352040 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5b7fff640 10 client.16362 handle_command_reply: tid=0 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 client.16362 shutdown 2024-05-09T09:23:01.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 msgr2=0x55a7553467f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0097b0 tx=0x7ff5cc005720 comp rx=0 tx=0).stop 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 shutdown_connections 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a755352040 0x55a755354450 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a755346d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4287485077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a7553467f0 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 >> 172.21.15.69:0/4287485077 conn(0x55a7553338c0 msgr2=0x55a755332a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5b7fff640 0 client.16362 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 shutdown_connections 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4287485077 wait complete. 2024-05-09T09:23:01.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.589+0000 7ff5e19fd740 20 client.16362 trim_cache size 0 max 16384 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 Processor -- start 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 -- start start 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a75533c660 0x55a755336f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7553421c0 0x55a755345e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81aaa20 con 0x55a75533d050 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81a82f0 con 0x55a75533c660 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81a6140 con 0x55a7553421c0 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60588/0 (socket says 172.21.15.69:60588) 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5d7fff640 1 -- 172.21.15.69:0/4178397544 learned_addr learned my addr 172.21.15.69:0/4178397544 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:01.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.604+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a75533c660 0x55a755336f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7553421c0 0x55a755345e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 -- 172.21.15.69:0/4178397544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7553421c0 msgr2=0x55a755345e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7553421c0 0x55a755345e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 -- 172.21.15.69:0/4178397544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a75533c660 msgr2=0x55a755336f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a75533c660 0x55a755336f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 -- 172.21.15.69:0/4178397544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a755334630 con 0x55a75533d050 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40314e0 tx=0x7ff5c4047000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5121e906ce0d4ad8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5c27fc640 1 -- 172.21.15.69:0/4178397544 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4035040 con 0x55a75533d050 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5c27fc640 1 -- 172.21.15.69:0/4178397544 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5c4055810 con 0x55a75533d050 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.605+0000 7ff5c27fc640 1 -- 172.21.15.69:0/4178397544 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4032900 con 0x55a75533d050 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4178397544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 msgr2=0x55a75534dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7ff5c40314e0 tx=0x7ff5c4047000 comp rx=0 tx=0).stop 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4178397544 shutdown_connections 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a7553421c0 0x55a755345e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533d050 0x55a75534dd30 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/4178397544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a75533c660 0x55a755336f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4178397544 >> 172.21.15.69:0/4178397544 conn(0x55a7553338c0 msgr2=0x55a755336840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4178397544 shutdown_connections 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- 172.21.15.69:0/4178397544 wait complete. 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 -- start start 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 Processor -- start 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.606+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a75534b4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534c460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81a67b0 con 0x55a75533c660 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81a7c20 con 0x55a7553421c0 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff5d81a6e80 con 0x55a75533d050 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 10 client.0 _ll_register_callbacks cb 0x7ff5cc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60598/0 (socket says 172.21.15.69:60598) 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 -- 172.21.15.69:0/485845763 learned_addr learned my addr 172.21.15.69:0/485845763 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a75534b4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dd5ea640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 -- 172.21.15.69:0/485845763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 msgr2=0x55a75534b4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a75534b4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 -- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 msgr2=0x55a75534c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 -- 172.21.15.69:0/485845763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a75533dd20 con 0x55a75533c660 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5dcde9640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0058c0 tx=0x7ff5cc00e3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=275e868140278ff8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5b7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc015020 con 0x55a75533c660 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff5c4055810 con 0x55a75533c660 2024-05-09T09:23:01.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(95..95 src has 1..95) ==== 9129+0+0 (secure 0 0 0) 0x7ff5cc003b10 con 0x55a75533c660 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=33) ==== 24+0+0 (secure 0 0 0) 0x7ff5cc03e050 con 0x55a75533c660 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 10 client.16368 fetch_fsmap learned FSMap version 33 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.607+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=33,monmap=4+}) -- 0x55a7553458d0 con 0x55a75533c660 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff5c4055810 con 0x55a75533c660 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 33) ==== 8346+0+0 (secure 0 0 0) 0x7ff5cc019430 con 0x55a75533c660 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 10 client.16368 fetch_fsmap finished waiting for FSMap version 33 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 10 client.16368 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 20 client.16368 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a7553b8640 0x55a7553baa60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 4 client.16368 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a755341f30 con 0x55a7553b8640 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a7553b8640 0x55a7553baa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5b7fff640 10 client.16368 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:01.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.608+0000 7ff5d7fff640 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a7553b8640 0x55a7553baa60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f77d7fc0c32c85 server_cookie=b92787e8bffc4f77 in_seq=0 out_seq=0 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5b7fff640 1 -- 172.21.15.69:0/485845763 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a755341f30 con 0x55a7553b8640 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5b7fff640 10 client.16368 handle_command_reply: tid=0 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 client.16368 shutdown 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 msgr2=0x55a75534af60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc0058c0 tx=0x7ff5cc00e3b0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 shutdown_connections 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a7553421c0 0x55a75534c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a7553b8640 0x55a7553baa60 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a75533d050 0x55a75534b4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5b7fff640 0 client.16368 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:01.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 --2- 172.21.15.69:0/485845763 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a75533c660 0x55a75534af60 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 >> 172.21.15.69:0/485845763 conn(0x55a7553338c0 msgr2=0x55a755332170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 shutdown_connections 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.661+0000 7ff5e19fd740 1 -- 172.21.15.69:0/485845763 wait complete. 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5e19fd740 20 client.16368 trim_cache size 0 max 16384 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:23:01.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 msgr2=0x7ff5c80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 0x7ff5c80a50f0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4031440 tx=0x7ff5c4004490 comp rx=0 tx=0).stop 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 msgr2=0x7ff5d819f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7ff5d000eaa0 tx=0x7ff5d000ef60 comp rx=0 tx=0).stop 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 shutdown_connections 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff5c80a2c30 0x7ff5c80a50f0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff5d8102340 0x7ff5d819f0a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff5d8101980 0x7ff5d8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 --2- 172.21.15.69:0/2417132091 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff5d8100790 0x7ff5d8197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.662+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 >> 172.21.15.69:0/2417132091 conn(0x7ff5d80fbd00 msgr2=0x7ff5d8068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.663+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 shutdown_connections 2024-05-09T09:23:01.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:01.663+0000 7ff5df04d640 1 -- 172.21.15.69:0/2417132091 wait complete. 2024-05-09T09:23:02.269 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:23:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:23:02.269 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:23:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:23:02.270 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:23:01 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/1442206729 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x5599e3d8da40 con 0x5599e3d60c00 2024-05-09T09:23:02.270 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:23:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:23:02.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110667 3.50 MB/sec execute 60 sec latency 596.123 ms 2024-05-09T09:23:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:02 smithi163 ceph-mon[34354]: pgmap v491: 129 pgs: 129 active+clean; 42 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 835 KiB/s wr, 134 op/s 2024-05-09T09:23:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:02 smithi163 ceph-mon[34354]: daemon mds.c restarted 2024-05-09T09:23:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:02 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:23:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:02 smithi103 ceph-mon[34421]: pgmap v491: 129 pgs: 129 active+clean; 42 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 835 KiB/s wr, 134 op/s 2024-05-09T09:23:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:02 smithi103 ceph-mon[34421]: daemon mds.c restarted 2024-05-09T09:23:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:02 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:23:02.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:02 smithi069 ceph-mon[30123]: pgmap v491: 129 pgs: 129 active+clean; 42 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 932 KiB/s rd, 835 KiB/s wr, 134 op/s 2024-05-09T09:23:02.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:02 smithi069 ceph-mon[30123]: daemon mds.c restarted 2024-05-09T09:23:02.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:02 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:23:03.482 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110667 3.45 MB/sec execute 61 sec latency 1596.156 ms 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/3445117907,v1:172.21.15.163:6837/3445117907] up:boot 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:23:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:23:03.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/3445117907,v1:172.21.15.163:6837/3445117907] up:boot 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:23:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:23:03.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:23:03.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:23:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: osdmap e96: 12 total, 12 up, 12 in 2024-05-09T09:23:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/3445117907,v1:172.21.15.163:6837/3445117907] up:boot 2024-05-09T09:23:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:23:03.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:23:03.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:23:03.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:23:03.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:23:03.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:23:04.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110667 3.39 MB/sec execute 62 sec latency 2596.266 ms 2024-05-09T09:23:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:04 smithi163 ceph-mon[34354]: pgmap v493: 129 pgs: 129 active+clean; 42 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-05-09T09:23:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:04 smithi103 ceph-mon[34421]: pgmap v493: 129 pgs: 129 active+clean; 42 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-05-09T09:23:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:04 smithi069 ceph-mon[30123]: pgmap v493: 129 pgs: 129 active+clean; 42 MiB data, 599 MiB used, 1.0 TiB / 1.0 TiB avail; 1007 KiB/s rd, 989 KiB/s wr, 128 op/s 2024-05-09T09:23:05.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110723 3.34 MB/sec execute 63 sec latency 3360.189 ms 2024-05-09T09:23:06.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110723 3.29 MB/sec execute 64 sec latency 1191.886 ms 2024-05-09T09:23:06.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:06 smithi163 ceph-mon[34354]: pgmap v494: 129 pgs: 129 active+clean; 43 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-05-09T09:23:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:06 smithi103 ceph-mon[34421]: pgmap v494: 129 pgs: 129 active+clean; 43 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-05-09T09:23:06.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:06 smithi069 ceph-mon[30123]: pgmap v494: 129 pgs: 129 active+clean; 43 MiB data, 609 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.1 MiB/s wr, 164 op/s 2024-05-09T09:23:07.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110723 3.24 MB/sec execute 65 sec latency 2191.954 ms 2024-05-09T09:23:08.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110723 3.19 MB/sec execute 66 sec latency 3192.063 ms 2024-05-09T09:23:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:08 smithi163 ceph-mon[34354]: pgmap v495: 129 pgs: 129 active+clean; 39 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:08 smithi103 ceph-mon[34421]: pgmap v495: 129 pgs: 129 active+clean; 39 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:08.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:08 smithi069 ceph-mon[30123]: pgmap v495: 129 pgs: 129 active+clean; 39 MiB data, 602 MiB used, 1.0 TiB / 1.0 TiB avail; 769 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:09.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 110723 3.14 MB/sec execute 67 sec latency 4192.158 ms 2024-05-09T09:23:10.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 111170 3.12 MB/sec execute 68 sec latency 4942.777 ms 2024-05-09T09:23:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:10 smithi163 ceph-mon[34354]: pgmap v496: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:10 smithi103 ceph-mon[34421]: pgmap v496: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:10.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:10 smithi069 ceph-mon[30123]: pgmap v496: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 599 KiB/s rd, 1.2 MiB/s wr, 142 op/s 2024-05-09T09:23:11.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 113868 3.32 MB/sec execute 69 sec latency 21.070 ms 2024-05-09T09:23:12.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114266 3.28 MB/sec execute 70 sec latency 594.404 ms 2024-05-09T09:23:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:12 smithi163 ceph-mon[34354]: pgmap v497: 129 pgs: 129 active+clean; 38 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:23:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:12 smithi103 ceph-mon[34421]: pgmap v497: 129 pgs: 129 active+clean; 38 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:23:12.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:12 smithi069 ceph-mon[30123]: pgmap v497: 129 pgs: 129 active+clean; 38 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 545 KiB/s rd, 1.1 MiB/s wr, 123 op/s 2024-05-09T09:23:13.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114266 3.23 MB/sec execute 71 sec latency 1594.496 ms 2024-05-09T09:23:13.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:13 smithi163 ceph-mon[34354]: pgmap v498: 129 pgs: 129 active+clean; 38 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:23:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:13 smithi103 ceph-mon[34421]: pgmap v498: 129 pgs: 129 active+clean; 38 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:23:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:13 smithi069 ceph-mon[30123]: pgmap v498: 129 pgs: 129 active+clean; 38 MiB data, 621 MiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 1.1 MiB/s wr, 103 op/s 2024-05-09T09:23:14.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114266 3.19 MB/sec execute 72 sec latency 2594.626 ms 2024-05-09T09:23:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 -- 172.21.15.69:0/3779322328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e41018c0 msgr2=0x7fc4e4101ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3779322328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e41018c0 0x7fc4e4101ce0 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4009900 tx=0x7fc4d402a410 comp rx=0 tx=0).stop 2024-05-09T09:23:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 -- 172.21.15.69:0/3779322328 shutdown_connections 2024-05-09T09:23:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3779322328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4103470 0x7fc4e4109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3779322328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e4102ab0 0x7fc4e4102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3779322328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e41018c0 0x7fc4e4101ce0 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 -- 172.21.15.69:0/3779322328 >> 172.21.15.69:0/3779322328 conn(0x7fc4e40fce50 msgr2=0x7fc4e40ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 -- 172.21.15.69:0/3779322328 shutdown_connections 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.759+0000 7fc4e889a640 1 -- 172.21.15.69:0/3779322328 wait complete. 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 Processor -- start 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 -- start start 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e41018c0 0x7fc4e4197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4102ab0 0x7fc4e4198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410d530 con 0x7fc4e4103470 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410d3b0 con 0x7fc4e4102ab0 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e889a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e410d6b0 con 0x7fc4e41018c0 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58754/0 (socket says 172.21.15.69:58754) 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.760+0000 7fc4e2d9d640 1 -- 172.21.15.69:0/3755623633 learned_addr learned my addr 172.21.15.69:0/3755623633 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e259c640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e41018c0 0x7fc4e4197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 -- 172.21.15.69:0/3755623633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e41018c0 msgr2=0x7fc4e4197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e41018c0 0x7fc4e4197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 -- 172.21.15.69:0/3755623633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4102ab0 msgr2=0x7fc4e4198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4102ab0 0x7fc4e4198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 -- 172.21.15.69:0/3755623633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc4e419f760 con 0x7fc4e4103470 2024-05-09T09:23:14.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e2d9d640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00eaa0 tx=0x7fc4dc00ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4db7fe640 1 -- 172.21.15.69:0/3755623633 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc00cdb0 con 0x7fc4e4103470 2024-05-09T09:23:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4db7fe640 1 -- 172.21.15.69:0/3755623633 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4dc005020 con 0x7fc4e4103470 2024-05-09T09:23:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc4e419fa50 con 0x7fc4e4103470 2024-05-09T09:23:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4db7fe640 1 -- 172.21.15.69:0/3755623633 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4dc0116b0 con 0x7fc4e4103470 2024-05-09T09:23:14.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.761+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc4e419ff50 con 0x7fc4e4103470 2024-05-09T09:23:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.763+0000 7fc4db7fe640 1 -- 172.21.15.69:0/3755623633 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc4dc005560 con 0x7fc4e4103470 2024-05-09T09:23:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 Processor -- start 2024-05-09T09:23:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 -- start start 2024-05-09T09:23:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213591c40 0x557213596740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213596db0 0x557213599240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5572134f5340 con 0x557213591250 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55721349d290 con 0x557213591c40 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5572134f1300 con 0x557213596db0 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4db7fe640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 0x7fc4d00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4dbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4e2d9d640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 0x7fc4d00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.764+0000 7fc4dbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58768/0 (socket says 172.21.15.69:58768) 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 -- 172.21.15.69:0/4172156260 learned_addr learned my addr 172.21.15.69:0/4172156260 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4e259c640 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213591c40 0x557213596740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213596db0 0x557213599240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4db7fe640 1 -- 172.21.15.69:0/3755623633 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fc4dc0cf6f0 con 0x7fc4e4103470 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 -- 172.21.15.69:0/4172156260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213596db0 msgr2=0x557213599240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213596db0 0x557213599240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.773 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 -- 172.21.15.69:0/4172156260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213591c40 msgr2=0x557213596740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213591c40 0x557213596740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 -- 172.21.15.69:0/4172156260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5572134c18e0 con 0x557213591250 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4e2d9d640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 0x7fc4d00a50f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0cd1e0 tx=0x7fc4dc0c6760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00f300 tx=0x7fc4cc00b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48f20d3472bb0019 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4d8ff9640 1 -- 172.21.15.69:0/4172156260 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc005020 con 0x557213591250 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4d8ff9640 1 -- 172.21.15.69:0/4172156260 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4cc0047a0 con 0x557213591250 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.765+0000 7fc4d8ff9640 1 -- 172.21.15.69:0/4172156260 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc012650 con 0x557213591250 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/4172156260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 msgr2=0x557213591670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc00f300 tx=0x7fc4cc00b4d0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/4172156260 shutdown_connections 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213596db0 0x557213599240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213591c40 0x557213596740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/4172156260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591250 0x557213591670 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/4172156260 >> 172.21.15.69:0/4172156260 conn(0x55721358c8c0 msgr2=0x55721358ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/4172156260 shutdown_connections 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/4172156260 wait complete. 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.766+0000 7fc4eb5a6740 1 -- start start 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 Processor -- start 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557213599d60 con 0x557213591c40 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5572134b0790 con 0x557213596db0 2024-05-09T09:23:14.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557213599ee0 con 0x557213591250 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4eb5a6740 10 client.0 _ll_register_callbacks cb 0x5572134eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58780/0 (socket says 172.21.15.69:58780) 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 -- 172.21.15.69:0/1709585028 learned_addr learned my addr 172.21.15.69:0/1709585028 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 -- 172.21.15.69:0/1709585028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 msgr2=0x55721359b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 -- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 msgr2=0x5572135a29b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a29b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 -- 172.21.15.69:0/1709585028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557213209210 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4e259c640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005940 tx=0x7fc4d4031a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31eb7f5af03f17cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.767+0000 7fc4c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4035040 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fc4d4004240 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4cc0047a0 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4d404e050 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 10 client.16386 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572132d5620 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4cc0047a0 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fc4d4004b10 con 0x557213591c40 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 10 client.16386 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:14.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 10 client.16386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 20 client.16386 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5572135a6f70 0x5572135a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 4 client.16386 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557213502a70 con 0x5572135a6f70 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.768+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5572135a6f70 0x5572135a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.769+0000 7fc4c9ffb640 10 client.16386 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.769+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5572135a6f70 0x5572135a9380 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e5ae2bd287a377 server_cookie=1827ca6258114267 in_seq=0 out_seq=0 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.769+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/1709585028 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557213502a70 con 0x5572135a6f70 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4c9ffb640 10 client.16386 handle_command_reply: tid=0 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 client.16386 shutdown 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 msgr2=0x55721359b970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4005940 tx=0x7fc4d4031a20 comp rx=0 tx=0).stop 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 shutdown_connections 2024-05-09T09:23:14.778 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a29b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x55721359b970 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5572135a6f70 0x5572135a9380 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/1709585028 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4c9ffb640 0 client.16386 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 >> 172.21.15.69:0/1709585028 conn(0x55721358c8c0 msgr2=0x55721358ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 shutdown_connections 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/1709585028 wait complete. 2024-05-09T09:23:14.779 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.770+0000 7fc4eb5a6740 20 client.16386 trim_cache size 0 max 16384 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 Processor -- start 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 -- start start 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721358cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x557213592e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41acca0 con 0x557213591c40 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41aa570 con 0x557213596db0 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41a83c0 con 0x557213591250 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58796/0 (socket says 172.21.15.69:58796) 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 -- 172.21.15.69:0/2986149584 learned_addr learned my addr 172.21.15.69:0/2986149584 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 -- 172.21.15.69:0/2986149584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 msgr2=0x55721358cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721358cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721358cca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x557213592e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 -- 172.21.15.69:0/2986149584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 msgr2=0x557213592e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x557213592e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 -- 172.21.15.69:0/2986149584 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55721358af00 con 0x557213591c40 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x557213592e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:14.791 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.785+0000 7fc4e259c640 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4031440 tx=0x7fc4d4004850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a186081446e344a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4cbfff640 1 -- 172.21.15.69:0/2986149584 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4035030 con 0x557213591c40 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4cbfff640 1 -- 172.21.15.69:0/2986149584 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d40054e0 con 0x557213591c40 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4cbfff640 1 -- 172.21.15.69:0/2986149584 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4004cb0 con 0x557213591c40 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/2986149584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 msgr2=0x5572135a2970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4031440 tx=0x7fc4d4004850 comp rx=0 tx=0).stop 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/2986149584 shutdown_connections 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x557213592e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a2970 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/2986149584 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721358cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/2986149584 >> 172.21.15.69:0/2986149584 conn(0x55721358c8c0 msgr2=0x557213596450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/2986149584 shutdown_connections 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.786+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/2986149584 wait complete. 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 -- start start 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 Processor -- start 2024-05-09T09:23:14.792 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359fc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a1190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41a8a30 con 0x557213591c40 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41a9ea0 con 0x557213596db0 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc4e41a9100 con 0x557213591250 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4eb5a6740 10 client.0 _ll_register_callbacks cb 0x7fc4d4009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58798/0 (socket says 172.21.15.69:58798) 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 -- 172.21.15.69:0/540408220 learned_addr learned my addr 172.21.15.69:0/540408220 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359fc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 -- 172.21.15.69:0/540408220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 msgr2=0x55721359fc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359fc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 -- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 msgr2=0x5572135a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 -- 172.21.15.69:0/540408220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5572135a1780 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4e259c640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4031a60 tx=0x7fc4d4004c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4788bae02527f4b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.787+0000 7fc4c9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc4d4060020 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fc4d4005170 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc4d40054e0 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fc4d4068050 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 10 client.16392 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x5572135a2a00 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc4d40054e0 con 0x557213591c40 2024-05-09T09:23:14.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fc4d4045cb0 con 0x557213591c40 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 10 client.16392 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 10 client.16392 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 20 client.16392 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55721360d940 0x55721360fd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 4 client.16392 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5572134fcdc0 con 0x55721360d940 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.788+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55721360d940 0x55721360fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.789+0000 7fc4c9ffb640 10 client.16392 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:14.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.789+0000 7fc4dbfff640 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55721360d940 0x55721360fd60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=34b8bdf9c69e91a5 server_cookie=94ba93327743c833 in_seq=0 out_seq=0 2024-05-09T09:23:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.836+0000 7fc4c9ffb640 1 -- 172.21.15.69:0/540408220 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5572134fcdc0 con 0x55721360d940 2024-05-09T09:23:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.836+0000 7fc4c9ffb640 10 client.16392 handle_command_reply: tid=0 2024-05-09T09:23:14.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.836+0000 7fc4eb5a6740 1 client.16392 shutdown 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.836+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 msgr2=0x5572135a0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.836+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7fc4d4031a60 tx=0x7fc4d4004c60 comp rx=0 tx=0).stop 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 shutdown_connections 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557213596db0 0x5572135a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557213591c40 0x5572135a0180 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55721360d940 0x55721360fd60 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 --2- 172.21.15.69:0/540408220 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557213591250 0x55721359fc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4c9ffb640 0 client.16392 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 >> 172.21.15.69:0/540408220 conn(0x55721358c8c0 msgr2=0x55721359b3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 shutdown_connections 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 1 -- 172.21.15.69:0/540408220 wait complete. 2024-05-09T09:23:14.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4eb5a6740 20 client.16392 trim_cache size 0 max 16384 2024-05-09T09:23:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:23:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:23:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:23:14.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 msgr2=0x7fc4d00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 0x7fc4d00a50f0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc0cd1e0 tx=0x7fc4dc0c6760 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 msgr2=0x7fc4e419f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7fc4dc00eaa0 tx=0x7fc4dc00ef60 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 shutdown_connections 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4d00a2c30 0x7fc4d00a50f0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc4e4103470 0x7fc4e419f0f0 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc4e4102ab0 0x7fc4e4198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 --2- 172.21.15.69:0/3755623633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc4e41018c0 0x7fc4e4197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 >> 172.21.15.69:0/3755623633 conn(0x7fc4e40fce50 msgr2=0x7fc4e4105d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 shutdown_connections 2024-05-09T09:23:14.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:14.837+0000 7fc4e889a640 1 -- 172.21.15.69:0/3755623633 wait complete. 2024-05-09T09:23:15.483 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114334 3.15 MB/sec execute 73 sec latency 3344.622 ms 2024-05-09T09:23:16.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114334 3.10 MB/sec execute 74 sec latency 1204.570 ms 2024-05-09T09:23:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:16 smithi163 ceph-mon[34354]: pgmap v499: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-05-09T09:23:16.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:16 smithi103 ceph-mon[34421]: pgmap v499: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-05-09T09:23:16.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:16 smithi069 ceph-mon[30123]: pgmap v499: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 1.1 MiB/s wr, 126 op/s 2024-05-09T09:23:17.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114334 3.06 MB/sec execute 75 sec latency 2204.675 ms 2024-05-09T09:23:18.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114334 3.02 MB/sec execute 76 sec latency 3204.733 ms 2024-05-09T09:23:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:18 smithi163 ceph-mon[34354]: pgmap v500: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 999 KiB/s wr, 103 op/s 2024-05-09T09:23:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:18 smithi103 ceph-mon[34421]: pgmap v500: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 999 KiB/s wr, 103 op/s 2024-05-09T09:23:18.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:18 smithi069 ceph-mon[30123]: pgmap v500: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 999 KiB/s wr, 103 op/s 2024-05-09T09:23:19.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114334 2.98 MB/sec execute 77 sec latency 4204.835 ms 2024-05-09T09:23:20.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 114795 2.97 MB/sec execute 78 sec latency 4978.819 ms 2024-05-09T09:23:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:20 smithi163 ceph-mon[34354]: pgmap v501: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 851 KiB/s wr, 104 op/s 2024-05-09T09:23:20.562 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:20 smithi069 ceph-mon[30123]: pgmap v501: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 851 KiB/s wr, 104 op/s 2024-05-09T09:23:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:20 smithi103 ceph-mon[34421]: pgmap v501: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 851 KiB/s wr, 104 op/s 2024-05-09T09:23:21.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117399 3.14 MB/sec execute 79 sec latency 25.109 ms 2024-05-09T09:23:22.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117756 3.11 MB/sec execute 80 sec latency 641.925 ms 2024-05-09T09:23:22.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:22 smithi163 ceph-mon[34354]: pgmap v502: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 708 KiB/s wr, 96 op/s 2024-05-09T09:23:22.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:22 smithi103 ceph-mon[34421]: pgmap v502: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 708 KiB/s wr, 96 op/s 2024-05-09T09:23:22.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:22 smithi069 ceph-mon[30123]: pgmap v502: 129 pgs: 129 active+clean; 39 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 708 KiB/s wr, 96 op/s 2024-05-09T09:23:23.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117756 3.07 MB/sec execute 81 sec latency 1641.959 ms 2024-05-09T09:23:24.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117756 3.03 MB/sec execute 82 sec latency 2642.072 ms 2024-05-09T09:23:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:24 smithi163 ceph-mon[34354]: pgmap v503: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 730 KiB/s wr, 103 op/s 2024-05-09T09:23:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:24 smithi103 ceph-mon[34421]: pgmap v503: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 730 KiB/s wr, 103 op/s 2024-05-09T09:23:24.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:24 smithi069 ceph-mon[30123]: pgmap v503: 129 pgs: 129 active+clean; 39 MiB data, 624 MiB used, 1.0 TiB / 1.0 TiB avail; 802 KiB/s rd, 730 KiB/s wr, 103 op/s 2024-05-09T09:23:25.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117880 3.00 MB/sec execute 83 sec latency 3389.280 ms 2024-05-09T09:23:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:25 smithi103 ceph-mon[34421]: pgmap v504: 129 pgs: 129 active+clean; 40 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 886 KiB/s wr, 121 op/s 2024-05-09T09:23:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:25 smithi069 ceph-mon[30123]: pgmap v504: 129 pgs: 129 active+clean; 40 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 886 KiB/s wr, 121 op/s 2024-05-09T09:23:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:25 smithi163 ceph-mon[34354]: pgmap v504: 129 pgs: 129 active+clean; 40 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 718 KiB/s rd, 886 KiB/s wr, 121 op/s 2024-05-09T09:23:26.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117880 2.96 MB/sec execute 84 sec latency 1138.829 ms 2024-05-09T09:23:27.484 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117880 2.93 MB/sec execute 85 sec latency 2138.949 ms 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 -- 172.21.15.69:0/4272658749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 msgr2=0x7fafdc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 --2- 172.21.15.69:0/4272658749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc100bb0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x7fafcc009900 tx=0x7fafcc02a410 comp rx=0 tx=0).stop 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 -- 172.21.15.69:0/4272658749 shutdown_connections 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 --2- 172.21.15.69:0/4272658749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 --2- 172.21.15.69:0/4272658749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 --2- 172.21.15.69:0/4272658749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc100bb0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 -- 172.21.15.69:0/4272658749 >> 172.21.15.69:0/4272658749 conn(0x7fafdc0fbd00 msgr2=0x7fafdc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 -- 172.21.15.69:0/4272658749 shutdown_connections 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.936+0000 7fafe124d640 1 -- 172.21.15.69:0/4272658749 wait complete. 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 Processor -- start 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 -- start start 2024-05-09T09:23:27.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc19a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc1a12f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10c400 con 0x7fafdc100790 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10c280 con 0x7fafdc102340 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafe124d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc10c580 con 0x7fafdc101980 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56546/0 (socket says 172.21.15.69:56546) 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 -- 172.21.15.69:0/2959952626 learned_addr learned my addr 172.21.15.69:0/2959952626 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdb59e640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc1a12f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc19a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 -- 172.21.15.69:0/2959952626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 msgr2=0x7fafdc19a280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc19a280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.959 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 -- 172.21.15.69:0/2959952626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 msgr2=0x7fafdc1a12f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc1a12f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdad9d640 1 -- 172.21.15.69:0/2959952626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fafdc1a1960 con 0x7fafdc100790 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.937+0000 7fafdb59e640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fafcc031610 tx=0x7fafcc0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafd8d99640 1 -- 172.21.15.69:0/2959952626 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc03f030 con 0x7fafdc100790 2024-05-09T09:23:27.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafd8d99640 1 -- 172.21.15.69:0/2959952626 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafcc003ce0 con 0x7fafdc100790 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fafdc1a1bf0 con 0x7fafdc100790 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafd8d99640 1 -- 172.21.15.69:0/2959952626 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc035070 con 0x7fafdc100790 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.938+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fafdc1a20e0 con 0x7fafdc100790 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.940+0000 7fafd8d99640 1 -- 172.21.15.69:0/2959952626 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fafcc042030 con 0x7fafdc100790 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 Processor -- start 2024-05-09T09:23:27.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 -- start start 2024-05-09T09:23:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157908670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f15790d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f15786c340 con 0x55f15790ddb0 2024-05-09T09:23:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f157814290 con 0x55f157908250 2024-05-09T09:23:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafe4036740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f157868300 con 0x55f157908c40 2024-05-09T09:23:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafd8d99640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 0x7fafc80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafd3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafd3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56550/0 (socket says 172.21.15.69:56550) 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafd3fff640 1 -- 172.21.15.69:0/1117745133 learned_addr learned my addr 172.21.15.69:0/1117745133 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafdb59e640 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157908670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.941+0000 7fafdb59e640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 0x7fafc80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd8d99640 1 -- 172.21.15.69:0/2959952626 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fafcc0efa10 con 0x7fafdc100790 2024-05-09T09:23:27.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafdad9d640 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f15790d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 -- 172.21.15.69:0/1117745133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 msgr2=0x55f15790d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f15790d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 -- 172.21.15.69:0/1117745133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 msgr2=0x55f157908670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157908670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 -- 172.21.15.69:0/1117745133 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f1578388e0 con 0x55f15790ddb0 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd3fff640 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fafc400bb40 tx=0x7fafc400d340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d198a6643348ac63 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafdb59e640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 0x7fafc80a50f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x55f1579094a0 tx=0x7fafd40094a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd17fa640 1 -- 172.21.15.69:0/1117745133 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc4017420 con 0x55f15790ddb0 2024-05-09T09:23:27.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd17fa640 1 -- 172.21.15.69:0/1117745133 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc4017aa0 con 0x55f15790ddb0 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.942+0000 7fafd17fa640 1 -- 172.21.15.69:0/1117745133 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafc400fba0 con 0x55f15790ddb0 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- 172.21.15.69:0/1117745133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 msgr2=0x55f157910240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fafc400bb40 tx=0x7fafc400d340 comp rx=0 tx=0).stop 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- 172.21.15.69:0/1117745133 shutdown_connections 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157910240 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f15790d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 --2- 172.21.15.69:0/1117745133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157908670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- 172.21.15.69:0/1117745133 >> 172.21.15.69:0/1117745133 conn(0x55f1579038c0 msgr2=0x55f157903cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- 172.21.15.69:0/1117745133 shutdown_connections 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- 172.21.15.69:0/1117745133 wait complete. 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 -- start start 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.943+0000 7fafe4036740 1 Processor -- start 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f157912920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f15790ddb0 0x55f157919960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f157910d60 con 0x55f157908250 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f157827790 con 0x55f157908c40 2024-05-09T09:23:27.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f157910ee0 con 0x55f15790ddb0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 10 client.0 _ll_register_callbacks cb 0x55f157862a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56554/0 (socket says 172.21.15.69:56554) 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 -- 172.21.15.69:0/2331263796 learned_addr learned my addr 172.21.15.69:0/2331263796 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f157912920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdb59e640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f15790ddb0 0x55f157919960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 -- 172.21.15.69:0/2331263796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f15790ddb0 msgr2=0x55f157919960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f15790ddb0 0x55f157919960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 -- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 msgr2=0x55f157912920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f157912920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 -- 172.21.15.69:0/2331263796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f157580210 con 0x55f157908250 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f157912920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafdad9d640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0b88c0 tx=0x7fafcc0bbcc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=807299494592f966 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafc27fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafcc049030 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fafcc003800 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.944+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafc4017aa0 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fafcc0bc5f0 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 10 client.16404 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f15764c620 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafc4017aa0 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fafcc0b8560 con 0x55f157908250 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 10 client.16404 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 10 client.16404 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 20 client.16404 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:27.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f15791dfc0 0x55f157920450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 4 client.16404 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f157879a70 con 0x55f15791dfc0 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.945+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f15791dfc0 0x55f157920450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafd3fff640 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f15791dfc0 0x55f157920450 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3856873c9039126 server_cookie=190a6ac411a583fe in_seq=0 out_seq=0 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafc27fc640 10 client.16404 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafc27fc640 1 -- 172.21.15.69:0/2331263796 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f157879a70 con 0x55f15791dfc0 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafc27fc640 10 client.16404 handle_command_reply: tid=0 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafe4036740 1 client.16404 shutdown 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 msgr2=0x55f1579123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.946+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fafcc0b88c0 tx=0x7fafcc0bbcc0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 shutdown_connections 2024-05-09T09:23:27.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f15791dfc0 0x55f157920450 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f15790ddb0 0x55f157919960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f157912920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafc27fc640 0 client.16404 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 --2- 172.21.15.69:0/2331263796 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f157908250 0x55f1579123e0 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 >> 172.21.15.69:0/2331263796 conn(0x55f1579038c0 msgr2=0x55f157902a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 shutdown_connections 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 1 -- 172.21.15.69:0/2331263796 wait complete. 2024-05-09T09:23:27.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.947+0000 7fafe4036740 20 client.16404 trim_cache size 0 max 16384 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.961+0000 7fafe4036740 1 Processor -- start 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.961+0000 7fafe4036740 1 -- start start 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908250 0x55f157925d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f1579262a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1aee20 con 0x55f15790ddb0 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ac6f0 con 0x55f157908c40 2024-05-09T09:23:27.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafe4036740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1aa540 con 0x55f157908250 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56558/0 (socket says 172.21.15.69:56558) 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 -- 172.21.15.69:0/401548453 learned_addr learned my addr 172.21.15.69:0/401548453 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdad9d640 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908250 0x55f157925d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafd3fff640 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f1579262a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 -- 172.21.15.69:0/401548453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908250 msgr2=0x55f157925d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908250 0x55f157925d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 -- 172.21.15.69:0/401548453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 msgr2=0x55f1579262a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f1579262a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.978 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 -- 172.21.15.69:0/401548453 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f157909020 con 0x55f15790ddb0 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafdb59e640 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fafd4012a20 tx=0x7fafd4009770 comp rx=0 tx=0).ready entity=mon.0 client_cookie=40be4cb718451ae8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.962+0000 7fafd0ff9640 1 -- 172.21.15.69:0/401548453 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4004b20 con 0x55f15790ddb0 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafd0ff9640 1 -- 172.21.15.69:0/401548453 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd4004cc0 con 0x55f15790ddb0 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafd0ff9640 1 -- 172.21.15.69:0/401548453 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd4005810 con 0x55f15790ddb0 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- 172.21.15.69:0/401548453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 msgr2=0x55f157919920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7fafd4012a20 tx=0x7fafd4009770 comp rx=0 tx=0).stop 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- 172.21.15.69:0/401548453 shutdown_connections 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157919920 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908c40 0x55f1579262a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 --2- 172.21.15.69:0/401548453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908250 0x55f157925d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- 172.21.15.69:0/401548453 >> 172.21.15.69:0/401548453 conn(0x55f1579038c0 msgr2=0x55f157909df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:27.979 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- 172.21.15.69:0/401548453 shutdown_connections 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- 172.21.15.69:0/401548453 wait complete. 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.963+0000 7fafe4036740 1 -- start start 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 Processor -- start 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157916b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f1579170d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1aabb0 con 0x55f15790ddb0 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ac020 con 0x55f157908250 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fafdc1ab280 con 0x55f157908c40 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 10 client.0 _ll_register_callbacks cb 0x7fafcc003e90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:27.980 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56564/0 (socket says 172.21.15.69:56564) 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 -- 172.21.15.69:0/3114660544 learned_addr learned my addr 172.21.15.69:0/3114660544 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafd3fff640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f1579170d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdad9d640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157916b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 -- 172.21.15.69:0/3114660544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 msgr2=0x55f1579170d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f1579170d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 -- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 msgr2=0x55f157916b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:27.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157916b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 -- 172.21.15.69:0/3114660544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f157918680 con 0x55f15790ddb0 2024-05-09T09:23:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafd3fff640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f1579170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:23:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdad9d640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157916b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:27.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafdb59e640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fafd40129f0 tx=0x7fafd4009850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b88492dfad7ddf36 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafc27fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fafd401b040 con 0x55f15790ddb0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fafd40109a0 con 0x55f15790ddb0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.964+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fafd4004cc0 con 0x55f15790ddb0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fafd403c050 con 0x55f15790ddb0 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 10 client.16410 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:27.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55f157919990 con 0x55f15790ddb0 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fafd4004cc0 con 0x55f15790ddb0 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fafd4029030 con 0x55f15790ddb0 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 10 client.16410 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 10 client.16410 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 20 client.16410 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:27.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f157984a10 0x55f157986e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 4 client.16410 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f157873dc0 con 0x55f157984a10 2024-05-09T09:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafdad9d640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f157984a10 0x55f157986e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafc27fc640 10 client.16410 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:27.985 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:27.965+0000 7fafdad9d640 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f157984a10 0x55f157986e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41fddb357a8f0841 server_cookie=989071e00408f2c9 in_seq=0 out_seq=0 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.046+0000 7fafc27fc640 1 -- 172.21.15.69:0/3114660544 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f157873dc0 con 0x55f157984a10 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.046+0000 7fafc27fc640 10 client.16410 handle_command_reply: tid=0 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.046+0000 7fafe4036740 1 client.16410 shutdown 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 msgr2=0x55f157918090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7fafd40129f0 tx=0x7fafd4009850 comp rx=0 tx=0).stop 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 shutdown_connections 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f15790ddb0 0x55f157918090 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f157984a10 0x55f157986e30 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f157908c40 0x55f1579170d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 --2- 172.21.15.69:0/3114660544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f157908250 0x55f157916b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafc27fc640 0 client.16410 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 >> 172.21.15.69:0/3114660544 conn(0x55f1579038c0 msgr2=0x55f157906870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 shutdown_connections 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 1 -- 172.21.15.69:0/3114660544 wait complete. 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe4036740 20 client.16410 trim_cache size 0 max 16384 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:23:28.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 msgr2=0x7fafc80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 0x7fafc80a50f0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x55f1579094a0 tx=0x7fafd40094a0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 msgr2=0x7fafdc199d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7fafcc031610 tx=0x7fafcc0043b0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 shutdown_connections 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fafc80a2c30 0x7fafc80a50f0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fafdc102340 0x7fafdc1a12f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fafdc101980 0x7fafdc19a280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 --2- 172.21.15.69:0/2959952626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fafdc100790 0x7fafdc199d40 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:28.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.047+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 >> 172.21.15.69:0/2959952626 conn(0x7fafdc0fbd00 msgr2=0x7fafdc068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.048+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 shutdown_connections 2024-05-09T09:23:28.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:28.048+0000 7fafe124d640 1 -- 172.21.15.69:0/2959952626 wait complete. 2024-05-09T09:23:28.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117880 2.89 MB/sec execute 86 sec latency 3139.048 ms 2024-05-09T09:23:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:28 smithi163 ceph-mon[34354]: pgmap v505: 129 pgs: 129 active+clean; 40 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 760 KiB/s wr, 94 op/s 2024-05-09T09:23:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:28 smithi103 ceph-mon[34421]: pgmap v505: 129 pgs: 129 active+clean; 40 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 760 KiB/s wr, 94 op/s 2024-05-09T09:23:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:28 smithi069 ceph-mon[30123]: pgmap v505: 129 pgs: 129 active+clean; 40 MiB data, 642 MiB used, 1.0 TiB / 1.0 TiB avail; 450 KiB/s rd, 760 KiB/s wr, 94 op/s 2024-05-09T09:23:29.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 117880 2.86 MB/sec execute 87 sec latency 4139.155 ms 2024-05-09T09:23:30.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 118296 2.85 MB/sec execute 88 sec latency 4887.283 ms 2024-05-09T09:23:30.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:30 smithi163 ceph-mon[34354]: pgmap v506: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 714 KiB/s wr, 88 op/s 2024-05-09T09:23:30.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:30 smithi103 ceph-mon[34421]: pgmap v506: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 714 KiB/s wr, 88 op/s 2024-05-09T09:23:30.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:30 smithi069 ceph-mon[30123]: pgmap v506: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 459 KiB/s rd, 714 KiB/s wr, 88 op/s 2024-05-09T09:23:31.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 120941 3.00 MB/sec execute 89 sec latency 16.996 ms 2024-05-09T09:23:32.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 122926 3.05 MB/sec execute 90 sec latency 11.875 ms 2024-05-09T09:23:32.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:32 smithi163 ceph-mon[34354]: pgmap v507: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 790 KiB/s wr, 77 op/s 2024-05-09T09:23:32.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:32 smithi103 ceph-mon[34421]: pgmap v507: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 790 KiB/s wr, 77 op/s 2024-05-09T09:23:32.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:32 smithi069 ceph-mon[30123]: pgmap v507: 129 pgs: 129 active+clean; 40 MiB data, 647 MiB used, 1.0 TiB / 1.0 TiB avail; 444 KiB/s rd, 790 KiB/s wr, 77 op/s 2024-05-09T09:23:33.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124739 3.15 MB/sec execute 91 sec latency 18.659 ms 2024-05-09T09:23:34.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124905 3.12 MB/sec execute 92 sec latency 869.248 ms 2024-05-09T09:23:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:34 smithi163 ceph-mon[34354]: pgmap v508: 129 pgs: 129 active+clean; 40 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 949 KiB/s wr, 93 op/s 2024-05-09T09:23:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:34 smithi103 ceph-mon[34421]: pgmap v508: 129 pgs: 129 active+clean; 40 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 949 KiB/s wr, 93 op/s 2024-05-09T09:23:34.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:34 smithi069 ceph-mon[30123]: pgmap v508: 129 pgs: 129 active+clean; 40 MiB data, 687 MiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 949 KiB/s wr, 93 op/s 2024-05-09T09:23:35.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124973 3.09 MB/sec execute 93 sec latency 1616.382 ms 2024-05-09T09:23:36.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124973 3.05 MB/sec execute 94 sec latency 1189.424 ms 2024-05-09T09:23:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:36 smithi163 ceph-mon[34354]: pgmap v509: 129 pgs: 129 active+clean; 42 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T09:23:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:36 smithi103 ceph-mon[34421]: pgmap v509: 129 pgs: 129 active+clean; 42 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T09:23:36.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:36 smithi069 ceph-mon[30123]: pgmap v509: 129 pgs: 129 active+clean; 42 MiB data, 698 MiB used, 1.0 TiB / 1.0 TiB avail; 864 KiB/s rd, 1.2 MiB/s wr, 138 op/s 2024-05-09T09:23:37.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124973 3.02 MB/sec execute 95 sec latency 2189.529 ms 2024-05-09T09:23:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:37 smithi163 ceph-mon[34354]: pgmap v510: 129 pgs: 129 active+clean; 42 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-05-09T09:23:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:37 smithi103 ceph-mon[34421]: pgmap v510: 129 pgs: 129 active+clean; 42 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-05-09T09:23:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:37 smithi069 ceph-mon[30123]: pgmap v510: 129 pgs: 129 active+clean; 42 MiB data, 703 MiB used, 1.0 TiB / 1.0 TiB avail; 816 KiB/s rd, 1.0 MiB/s wr, 118 op/s 2024-05-09T09:23:38.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124973 2.99 MB/sec execute 96 sec latency 3189.630 ms 2024-05-09T09:23:39.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 124973 2.96 MB/sec execute 97 sec latency 4189.763 ms 2024-05-09T09:23:40.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 125592 2.98 MB/sec execute 98 sec latency 4936.112 ms 2024-05-09T09:23:40.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:40 smithi163 ceph-mon[34354]: pgmap v511: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-09T09:23:40.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:40 smithi103 ceph-mon[34421]: pgmap v511: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-09T09:23:40.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:40 smithi069 ceph-mon[30123]: pgmap v511: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 861 KiB/s rd, 1.1 MiB/s wr, 120 op/s 2024-05-09T09:23:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 -- 172.21.15.69:0/619465910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c102340 msgr2=0x7fd51c108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 --2- 172.21.15.69:0/619465910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c102340 0x7fd51c108bd0 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7fd514009ca0 tx=0x7fd51402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 -- 172.21.15.69:0/619465910 shutdown_connections 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 --2- 172.21.15.69:0/619465910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c102340 0x7fd51c108bd0 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 --2- 172.21.15.69:0/619465910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c101980 0x7fd51c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 --2- 172.21.15.69:0/619465910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 0x7fd51c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 -- 172.21.15.69:0/619465910 >> 172.21.15.69:0/619465910 conn(0x7fd51c0fbd00 msgr2=0x7fd51c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 -- 172.21.15.69:0/619465910 shutdown_connections 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.140+0000 7fd523dce640 1 -- 172.21.15.69:0/619465910 wait complete. 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 Processor -- start 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 -- start start 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 0x7fd51c069a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c102340 0x7fd51c19ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c10c3f0 con 0x7fd51c101980 2024-05-09T09:23:41.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c10c270 con 0x7fd51c102340 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd523dce640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c10c570 con 0x7fd51c100790 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35342/0 (socket says 172.21.15.69:35342) 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 -- 172.21.15.69:0/4205495045 learned_addr learned my addr 172.21.15.69:0/4205495045 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd52236b640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c102340 0x7fd51c19ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521b6a640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 0x7fd51c069a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 -- 172.21.15.69:0/4205495045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 msgr2=0x7fd51c069a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 0x7fd51c069a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 -- 172.21.15.69:0/4205495045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c102340 msgr2=0x7fd51c19ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c102340 0x7fd51c19ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 -- 172.21.15.69:0/4205495045 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd51c19f6d0 con 0x7fd51c101980 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.141+0000 7fd521369640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00e9f0 tx=0x7fd50c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.142+0000 7fd512ffd640 1 -- 172.21.15.69:0/4205495045 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c00cd80 con 0x7fd51c101980 2024-05-09T09:23:41.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.142+0000 7fd512ffd640 1 -- 172.21.15.69:0/4205495045 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c010910 con 0x7fd51c101980 2024-05-09T09:23:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.142+0000 7fd512ffd640 1 -- 172.21.15.69:0/4205495045 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c018990 con 0x7fd51c101980 2024-05-09T09:23:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.142+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd51c19f9c0 con 0x7fd51c101980 2024-05-09T09:23:41.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.142+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd51c19ff10 con 0x7fd51c101980 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.144+0000 7fd512ffd640 1 -- 172.21.15.69:0/4205495045 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd50c010ac0 con 0x7fd51c101980 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 Processor -- start 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 -- start start 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd512ffd640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 0x7fd5080a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738af2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738afa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562738a56340 con 0x562738af2c40 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5627389fe290 con 0x562738af7db0 2024-05-09T09:23:41.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd526bb0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562738a52300 con 0x562738af2250 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.145+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd52236b640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 0x7fd5080a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35356/0 (socket says 172.21.15.69:35356) 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 -- 172.21.15.69:0/199419282 learned_addr learned my addr 172.21.15.69:0/199419282 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd52236b640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738afa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521b6a640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738af2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd512ffd640 1 -- 172.21.15.69:0/4205495045 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fd50c0cf6f0 con 0x7fd51c101980 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 -- 172.21.15.69:0/199419282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 msgr2=0x562738af2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738af2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 -- 172.21.15.69:0/199419282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 msgr2=0x562738afa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738afa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 -- 172.21.15.69:0/199419282 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562738a228e0 con 0x562738af2c40 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd52236b640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738afa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd52236b640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 0x7fd5080a50f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd514006010 tx=0x7fd51402f3a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.146+0000 7fd521369640 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097b0 tx=0x7fd50c09bb80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=625c092307493a41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.147+0000 7fd507fff640 1 -- 172.21.15.69:0/199419282 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0b4dd0 con 0x562738af2c40 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.147+0000 7fd507fff640 1 -- 172.21.15.69:0/199419282 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c09bc80 con 0x562738af2c40 2024-05-09T09:23:41.148 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.147+0000 7fd507fff640 1 -- 172.21.15.69:0/199419282 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0982f0 con 0x562738af2c40 2024-05-09T09:23:41.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.147+0000 7fd526bb0740 1 -- 172.21.15.69:0/199419282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 msgr2=0x562738af7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.147+0000 7fd526bb0740 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7fd50c0097b0 tx=0x7fd50c09bb80 comp rx=0 tx=0).stop 2024-05-09T09:23:41.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 -- 172.21.15.69:0/199419282 shutdown_connections 2024-05-09T09:23:41.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738afa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738af7740 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 --2- 172.21.15.69:0/199419282 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738af2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 -- 172.21.15.69:0/199419282 >> 172.21.15.69:0/199419282 conn(0x562738aed8c0 msgr2=0x562738aedcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 -- 172.21.15.69:0/199419282 shutdown_connections 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 -- 172.21.15.69:0/199419282 wait complete. 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 -- start start 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.148+0000 7fd526bb0740 1 Processor -- start 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd521b6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd521b6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35364/0 (socket says 172.21.15.69:35364) 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd521b6a640 1 -- 172.21.15.69:0/2301902472 learned_addr learned my addr 172.21.15.69:0/2301902472 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2c40 0x562738afca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738b03aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562738afad60 con 0x562738af2250 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562738a11790 con 0x562738af7db0 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.149+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562738afaee0 con 0x562738af2c40 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd526bb0740 10 client.0 _ll_register_callbacks cb 0x562738a4ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd521b6a640 1 -- 172.21.15.69:0/2301902472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2c40 msgr2=0x562738afca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd521b6a640 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2c40 0x562738afca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd521b6a640 1 -- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 msgr2=0x562738b03aa0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd521b6a640 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738b03aa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.150+0000 7fd521b6a640 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56273876a210 con 0x562738af2250 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.151+0000 7fd521b6a640 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd518010720 tx=0x7fd518010be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6808d714189f1abd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.151+0000 7fd5057fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.151+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd51800fbf0 con 0x562738af2250 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.151+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fd51800fd90 con 0x562738af2250 2024-05-09T09:23:41.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.151+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd50c09bc80 con 0x562738af2250 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd518024050 con 0x562738af2250 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 10 client.16422 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562738836620 con 0x562738af2250 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c09bc80 con 0x562738af2250 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fd518019400 con 0x562738af2250 2024-05-09T09:23:41.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 10 client.16422 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 10 client.16422 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 20 client.16422 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738b08160 0x562738b0a620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 4 client.16422 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.152+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562738a63a70 con 0x562738b08160 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.153+0000 7fd521369640 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738b08160 0x562738b0a620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.153+0000 7fd5057fa640 10 client.16422 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.153+0000 7fd521369640 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738b08160 0x562738b0a620 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67db0ca31ccac2e4 server_cookie=8bdd25c61843983e in_seq=0 out_seq=0 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd5057fa640 1 -- 172.21.15.69:0/2301902472 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562738a63a70 con 0x562738b08160 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd5057fa640 10 client.16422 handle_command_reply: tid=0 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 client.16422 shutdown 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 msgr2=0x562738afc520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7fd518010720 tx=0x7fd518010be0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 shutdown_connections 2024-05-09T09:23:41.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af7db0 0x562738b03aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738b08160 0x562738b0a620 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2c40 0x562738afca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 --2- 172.21.15.69:0/2301902472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738afc520 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd5057fa640 0 client.16422 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.154+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 >> 172.21.15.69:0/2301902472 conn(0x562738aed8c0 msgr2=0x562738aeca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.155+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 shutdown_connections 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.155+0000 7fd526bb0740 1 -- 172.21.15.69:0/2301902472 wait complete. 2024-05-09T09:23:41.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.155+0000 7fd526bb0740 20 client.16422 trim_cache size 0 max 16384 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 Processor -- start 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 -- start start 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738aed8c0 0x562738b0ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af2c40 0x562738b0f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1aef40 con 0x562738af2250 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1ac810 con 0x562738af2c40 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd526bb0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1aa660 con 0x562738aed8c0 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35378/0 (socket says 172.21.15.69:35378) 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 -- 172.21.15.69:0/1086396541 learned_addr learned my addr 172.21.15.69:0/1086396541 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521b6a640 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738aed8c0 0x562738b0ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd52236b640 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af2c40 0x562738b0f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 -- 172.21.15.69:0/1086396541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738aed8c0 msgr2=0x562738b0ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738aed8c0 0x562738b0ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 -- 172.21.15.69:0/1086396541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af2c40 msgr2=0x562738b0f3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af2c40 0x562738b0f3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 -- 172.21.15.69:0/1086396541 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562738aebf00 con 0x562738af2250 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.170+0000 7fd521369640 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd50c098650 tx=0x7fd50c0baea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e8854ad0ebd4540f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd5077fe640 1 -- 172.21.15.69:0/1086396541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c099d20 con 0x562738af2250 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd5077fe640 1 -- 172.21.15.69:0/1086396541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c0b4dd0 con 0x562738af2250 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd5077fe640 1 -- 172.21.15.69:0/1086396541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd50c0982f0 con 0x562738af2250 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 -- 172.21.15.69:0/1086396541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 msgr2=0x562738b03a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7fd50c098650 tx=0x7fd50c0baea0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 -- 172.21.15.69:0/1086396541 shutdown_connections 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738af2c40 0x562738b0f3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2250 0x562738b03a60 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 --2- 172.21.15.69:0/1086396541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738aed8c0 0x562738b0ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 -- 172.21.15.69:0/1086396541 >> 172.21.15.69:0/1086396541 conn(0x562738af7880 msgr2=0x562738af0f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 -- 172.21.15.69:0/1086396541 shutdown_connections 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.171+0000 7fd526bb0740 1 -- 172.21.15.69:0/1086396541 wait complete. 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 -- start start 2024-05-09T09:23:41.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 Processor -- start 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738aed8c0 0x562738b00fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738b014e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1aacd0 con 0x562738af2c40 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1ac140 con 0x562738aed8c0 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd51c1ab3a0 con 0x562738af2250 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd526bb0740 10 client.0 _ll_register_callbacks cb 0x7fd51800a780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35394/0 (socket says 172.21.15.69:35394) 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 -- 172.21.15.69:0/3339410226 learned_addr learned my addr 172.21.15.69:0/3339410226 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd521b6a640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738aed8c0 0x562738b00fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.176 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd521369640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738b014e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 -- 172.21.15.69:0/3339410226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 msgr2=0x562738b014e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738b014e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 -- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738aed8c0 msgr2=0x562738b00fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738aed8c0 0x562738b00fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd52236b640 1 -- 172.21.15.69:0/3339410226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562738b02a70 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.172+0000 7fd521369640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738b014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd52236b640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd514009ca0 tx=0x7fd514005d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590a60de8c933dbf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd514004720 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fd51402f610 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd50c0b4dd0 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fd51402f810 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 10 client.16428 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x562738b03e20 con 0x562738af2c40 2024-05-09T09:23:41.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd50c0b4dd0 con 0x562738af2c40 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fd514052070 con 0x562738af2c40 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 10 client.16428 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 10 client.16428 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 20 client.16428 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738af7db0 0x562738b70fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 4 client.16428 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.173+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562738a5ddc0 con 0x562738af7db0 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.174+0000 7fd521b6a640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738af7db0 0x562738b70fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.174+0000 7fd5057fa640 10 client.16428 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:41.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.174+0000 7fd521b6a640 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738af7db0 0x562738b70fe0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7749d13a41692d04 server_cookie=2091792edb1077a5 in_seq=0 out_seq=0 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd5057fa640 1 -- 172.21.15.69:0/3339410226 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562738a5ddc0 con 0x562738af7db0 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd5057fa640 10 client.16428 handle_command_reply: tid=0 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 client.16428 shutdown 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 msgr2=0x562738b02480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7fd514009ca0 tx=0x7fd514005d80 comp rx=0 tx=0).stop 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 shutdown_connections 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562738af2c40 0x562738b02480 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562738af7db0 0x562738b70fe0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562738af2250 0x562738b014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd5057fa640 0 client.16428 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 --2- 172.21.15.69:0/3339410226 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562738aed8c0 0x562738b00fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 >> 172.21.15.69:0/3339410226 conn(0x562738af7880 msgr2=0x562738aec170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 shutdown_connections 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 1 -- 172.21.15.69:0/3339410226 wait complete. 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.231+0000 7fd526bb0740 20 client.16428 trim_cache size 0 max 16384 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:23:41.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 msgr2=0x7fd5080a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 0x7fd5080a50f0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7fd514006010 tx=0x7fd51402f3a0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 msgr2=0x7fd51c19c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7fd50c00e9f0 tx=0x7fd50c00eeb0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 shutdown_connections 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5080a2c30 0x7fd5080a50f0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd51c102340 0x7fd51c19ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd51c101980 0x7fd51c19c7d0 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 --2- 172.21.15.69:0/4205495045 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd51c100790 0x7fd51c069a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 >> 172.21.15.69:0/4205495045 conn(0x7fd51c0fbd00 msgr2=0x7fd51c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:41.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 shutdown_connections 2024-05-09T09:23:41.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:41.232+0000 7fd523dce640 1 -- 172.21.15.69:0/4205495045 wait complete. 2024-05-09T09:23:41.485 INFO:tasks.workunit.client.0.smithi069.stdout: 1 128031 3.09 MB/sec execute 99 sec latency 33.705 ms 2024-05-09T09:23:42.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 129768 3.11 MB/sec execute 100 sec latency 15.056 ms 2024-05-09T09:23:42.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:42 smithi163 ceph-mon[34354]: pgmap v512: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-05-09T09:23:42.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:42 smithi103 ceph-mon[34421]: pgmap v512: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-05-09T09:23:42.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:42 smithi069 ceph-mon[30123]: pgmap v512: 129 pgs: 129 active+clean; 43 MiB data, 708 MiB used, 1.0 TiB / 1.0 TiB avail; 848 KiB/s rd, 1.0 MiB/s wr, 108 op/s 2024-05-09T09:23:43.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 131837 3.22 MB/sec execute 101 sec latency 10.351 ms 2024-05-09T09:23:44.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 131943 3.19 MB/sec execute 102 sec latency 930.379 ms 2024-05-09T09:23:44.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:44 smithi163 ceph-mon[34354]: pgmap v513: 129 pgs: 129 active+clean; 43 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 929 KiB/s wr, 128 op/s 2024-05-09T09:23:44.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:44 smithi103 ceph-mon[34421]: pgmap v513: 129 pgs: 129 active+clean; 43 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 929 KiB/s wr, 128 op/s 2024-05-09T09:23:44.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:44 smithi069 ceph-mon[30123]: pgmap v513: 129 pgs: 129 active+clean; 43 MiB data, 713 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 929 KiB/s wr, 128 op/s 2024-05-09T09:23:45.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 132256 3.16 MB/sec execute 103 sec latency 1676.767 ms 2024-05-09T09:23:46.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135041 3.30 MB/sec execute 104 sec latency 19.733 ms 2024-05-09T09:23:46.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:46 smithi163 ceph-mon[34354]: pgmap v514: 129 pgs: 129 active+clean; 45 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 991 KiB/s wr, 176 op/s 2024-05-09T09:23:46.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:46 smithi103 ceph-mon[34421]: pgmap v514: 129 pgs: 129 active+clean; 45 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 991 KiB/s wr, 176 op/s 2024-05-09T09:23:46.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:46 smithi069 ceph-mon[30123]: pgmap v514: 129 pgs: 129 active+clean; 45 MiB data, 722 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 991 KiB/s wr, 176 op/s 2024-05-09T09:23:47.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135487 3.28 MB/sec execute 105 sec latency 466.095 ms 2024-05-09T09:23:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:47 smithi163 ceph-mon[34354]: pgmap v515: 129 pgs: 129 active+clean; 45 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 139 op/s 2024-05-09T09:23:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:47 smithi103 ceph-mon[34421]: pgmap v515: 129 pgs: 129 active+clean; 45 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 139 op/s 2024-05-09T09:23:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:47 smithi069 ceph-mon[30123]: pgmap v515: 129 pgs: 129 active+clean; 45 MiB data, 726 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 779 KiB/s wr, 139 op/s 2024-05-09T09:23:48.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135487 3.25 MB/sec execute 106 sec latency 1466.230 ms 2024-05-09T09:23:49.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135487 3.22 MB/sec execute 107 sec latency 2466.265 ms 2024-05-09T09:23:50.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135543 3.19 MB/sec execute 108 sec latency 3211.174 ms 2024-05-09T09:23:50.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:50 smithi163 ceph-mon[34354]: pgmap v516: 129 pgs: 129 active+clean; 47 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 160 op/s 2024-05-09T09:23:50.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:50 smithi103 ceph-mon[34421]: pgmap v516: 129 pgs: 129 active+clean; 47 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 160 op/s 2024-05-09T09:23:50.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:50 smithi069 ceph-mon[30123]: pgmap v516: 129 pgs: 129 active+clean; 47 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 911 KiB/s wr, 160 op/s 2024-05-09T09:23:51.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135543 3.16 MB/sec execute 109 sec latency 1201.803 ms 2024-05-09T09:23:52.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135543 3.13 MB/sec execute 110 sec latency 2201.847 ms 2024-05-09T09:23:52.557 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:52 smithi163 ceph-mon[34354]: pgmap v517: 129 pgs: 129 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 800 KiB/s wr, 154 op/s 2024-05-09T09:23:52.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:52 smithi103 ceph-mon[34421]: pgmap v517: 129 pgs: 129 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 800 KiB/s wr, 154 op/s 2024-05-09T09:23:52.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:52 smithi069 ceph-mon[30123]: pgmap v517: 129 pgs: 129 active+clean; 46 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 800 KiB/s wr, 154 op/s 2024-05-09T09:23:53.486 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135543 3.10 MB/sec execute 111 sec latency 3201.929 ms 2024-05-09T09:23:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:53 smithi103 ceph-mon[34421]: pgmap v518: 129 pgs: 129 active+clean; 46 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 814 KiB/s wr, 157 op/s 2024-05-09T09:23:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:53 smithi069 ceph-mon[30123]: pgmap v518: 129 pgs: 129 active+clean; 46 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 814 KiB/s wr, 157 op/s 2024-05-09T09:23:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:53 smithi163 ceph-mon[34354]: pgmap v518: 129 pgs: 129 active+clean; 46 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 814 KiB/s wr, 157 op/s 2024-05-09T09:23:54.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.328+0000 7f6f1afc1640 1 -- 172.21.15.69:0/20656680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f141075a0 msgr2=0x7f6f141099f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.328+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/20656680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f141075a0 0x7f6f141099f0 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c009ca0 tx=0x7f6f0c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- 172.21.15.69:0/20656680 shutdown_connections 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/20656680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f141075a0 0x7f6f141099f0 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/20656680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14107060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/20656680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f14102340 0x7f6f14104730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- 172.21.15.69:0/20656680 >> 172.21.15.69:0/20656680 conn(0x7f6f140fbd30 msgr2=0x7f6f140fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- 172.21.15.69:0/20656680 shutdown_connections 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- 172.21.15.69:0/20656680 wait complete. 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 Processor -- start 2024-05-09T09:23:54.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- start start 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f141075a0 0x7f6f1419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d560 con 0x7f6f14102340 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d3e0 con 0x7f6f141075a0 2024-05-09T09:23:54.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f1afc1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f1410d6e0 con 0x7f6f14104c70 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.329+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46906/0 (socket says 172.21.15.69:46906) 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1221602719 learned_addr learned my addr 172.21.15.69:0/1221602719 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1221602719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 msgr2=0x7f6f14198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f13fff640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1221602719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f141075a0 msgr2=0x7f6f1419f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f1955e640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f141075a0 0x7f6f1419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f141075a0 0x7f6f1419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1221602719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6f1419f6e0 con 0x7f6f14102340 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f13fff640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400c890 tx=0x7f6f0400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f11ffb640 1 -- 172.21.15.69:0/1221602719 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04004220 con 0x7f6f14102340 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6f1419f9d0 con 0x7f6f14102340 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f11ffb640 1 -- 172.21.15.69:0/1221602719 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f040043c0 con 0x7f6f14102340 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.330+0000 7f6f11ffb640 1 -- 172.21.15.69:0/1221602719 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f04004a00 con 0x7f6f14102340 2024-05-09T09:23:54.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.331+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6f1419ff70 con 0x7f6f14102340 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.333+0000 7f6f11ffb640 1 -- 172.21.15.69:0/1221602719 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6f040079b0 con 0x7f6f14102340 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 Processor -- start 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 -- start start 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fe9660 0x559a22fe9a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ff1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559a22f49340 con 0x559a22fea050 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559a22ef1290 con 0x559a22fef1c0 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559a22f45300 con 0x559a22fe9660 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46908/0 (socket says 172.21.15.69:46908) 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1502218627 learned_addr learned my addr 172.21.15.69:0/1502218627 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f11ffb640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 0x7f6efc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f13fff640 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ff1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1955e640 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fe9660 0x559a22fe9a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f1955e640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 0x7f6efc0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1502218627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fe9660 msgr2=0x559a22fe9a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fe9660 0x559a22fe9a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1502218627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 msgr2=0x559a22ff1650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ff1650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.334+0000 7f6f18d5d640 1 -- 172.21.15.69:0/1502218627 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a22f158e0 con 0x559a22fea050 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6f11ffb640 1 -- 172.21.15.69:0/1221602719 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f6f040cf170 con 0x7f6f14102340 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6f04098020 tx=0x7f6f0409b2c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24bb3c90ee1e5de4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6f1955e640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 0x7f6efc0a50f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c02ade0 tx=0x7f6f0c0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6efaffd640 1 -- 172.21.15.69:0/1502218627 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0401d030 con 0x559a22fea050 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6efaffd640 1 -- 172.21.15.69:0/1502218627 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0409bd50 con 0x559a22fea050 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.335+0000 7f6efaffd640 1 -- 172.21.15.69:0/1502218627 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0409fa00 con 0x559a22fea050 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- 172.21.15.69:0/1502218627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 msgr2=0x559a22feeb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7f6f04098020 tx=0x7f6f0409b2c0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- 172.21.15.69:0/1502218627 shutdown_connections 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ff1650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fea050 0x559a22feeb50 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/1502218627 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fe9660 0x559a22fe9a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- 172.21.15.69:0/1502218627 >> 172.21.15.69:0/1502218627 conn(0x559a22fe08c0 msgr2=0x559a22fe0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- 172.21.15.69:0/1502218627 shutdown_connections 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- 172.21.15.69:0/1502218627 wait complete. 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 -- start start 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 Processor -- start 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.336+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff3d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ffad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559a22ff1fc0 con 0x559a22fe9660 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559a22f04790 con 0x559a22fef1c0 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559a22ff2140 con 0x559a22fea050 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 10 client.0 _ll_register_callbacks cb 0x559a22f3fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46912/0 (socket says 172.21.15.69:46912) 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 -- 172.21.15.69:0/3997041618 learned_addr learned my addr 172.21.15.69:0/3997041618 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f13fff640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff3d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1955e640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ffad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 -- 172.21.15.69:0/3997041618 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 msgr2=0x559a22ff3d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff3d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 -- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 msgr2=0x559a22ffad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ffad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 -- 172.21.15.69:0/3997041618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a22c5d210 con 0x559a22fe9660 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6f040058c0 tx=0x7f6f040d29e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf5292d8af06a13b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6ef3fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0401d040 con 0x559a22fe9660 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f6f04084be0 con 0x559a22fe9660 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f0409bd50 con 0x559a22fe9660 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6f04012eb0 con 0x559a22fe9660 2024-05-09T09:23:54.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 10 client.16440 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.337+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559a22d29620 con 0x559a22fe9660 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0409bd50 con 0x559a22fe9660 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f6f040a3e30 con 0x559a22fe9660 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 10 client.16440 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 10 client.16440 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 20 client.16440 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a22ffeef0 0x559a230012e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 4 client.16440 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559a22f56a70 con 0x559a22ffeef0 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f13fff640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a22ffeef0 0x559a230012e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6ef3fff640 10 client.16440 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.338+0000 7f6f13fff640 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a22ffeef0 0x559a230012e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=776fd0bbbe244d1a server_cookie=e24bfa164af3469e in_seq=0 out_seq=0 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6ef3fff640 1 -- 172.21.15.69:0/3997041618 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559a22f56a70 con 0x559a22ffeef0 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6ef3fff640 10 client.16440 handle_command_reply: tid=0 2024-05-09T09:23:54.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 client.16440 shutdown 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 msgr2=0x559a22ff3810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7f6f040058c0 tx=0x7f6f040d29e0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 shutdown_connections 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fef1c0 0x559a22ffad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a22ffeef0 0x559a230012e0 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff3d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/3997041618 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fe9660 0x559a22ff3810 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6ef3fff640 0 client.16440 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 >> 172.21.15.69:0/3997041618 conn(0x559a22fe08c0 msgr2=0x559a22fdfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 shutdown_connections 2024-05-09T09:23:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.339+0000 7f6f1ddad740 1 -- 172.21.15.69:0/3997041618 wait complete. 2024-05-09T09:23:54.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.340+0000 7f6f1ddad740 20 client.16440 trim_cache size 0 max 16384 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.354+0000 7f6f1ddad740 1 Processor -- start 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 -- start start 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a23006b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a23007080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141aaa50 con 0x559a22fef1c0 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141a8320 con 0x559a22fe9660 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141a6170 con 0x559a22fea050 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46926/0 (socket says 172.21.15.69:46926) 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 -- 172.21.15.69:0/2757773544 learned_addr learned my addr 172.21.15.69:0/2757773544 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a23006b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f13fff640 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a23007080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 -- 172.21.15.69:0/2757773544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 msgr2=0x559a23007080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a23007080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 -- 172.21.15.69:0/2757773544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 msgr2=0x559a23006b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a23006b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 -- 172.21.15.69:0/2757773544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559a22fdef00 con 0x559a22fef1c0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c031550 tx=0x7f6f0c03a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc39c9277aa730c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6efa7fc640 1 -- 172.21.15.69:0/2757773544 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c04f070 con 0x559a22fef1c0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6efa7fc640 1 -- 172.21.15.69:0/2757773544 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0c036040 con 0x559a22fef1c0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.355+0000 7f6efa7fc640 1 -- 172.21.15.69:0/2757773544 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c042040 con 0x559a22fef1c0 2024-05-09T09:23:54.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2757773544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 msgr2=0x559a22ffad20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c031550 tx=0x7f6f0c03a000 comp rx=0 tx=0).stop 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2757773544 shutdown_connections 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ffad20 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a23007080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2757773544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a23006b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2757773544 >> 172.21.15.69:0/2757773544 conn(0x559a22fe08c0 msgr2=0x559a22fe0ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2757773544 shutdown_connections 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2757773544 wait complete. 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 -- start start 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.356+0000 7f6f1ddad740 1 Processor -- start 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a22ff8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff86a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141a67e0 con 0x559a22fef1c0 2024-05-09T09:23:54.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141a7c50 con 0x559a22fe9660 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6f141a6eb0 con 0x559a22fea050 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1ddad740 10 client.0 _ll_register_callbacks cb 0x7f6f08009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46942/0 (socket says 172.21.15.69:46942) 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 -- 172.21.15.69:0/2122234996 learned_addr learned my addr 172.21.15.69:0/2122234996 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a22ff8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f13fff640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff86a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 -- 172.21.15.69:0/2122234996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 msgr2=0x559a22ff86a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff86a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 -- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 msgr2=0x559a22ff8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a22ff8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 -- 172.21.15.69:0/2122234996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559a22ff9ca0 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f13fff640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6f1955e640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c031ae0 tx=0x7f6f0c031bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cbda2f759382e5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6ef3fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.357+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6f0c04f050 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6f0c036040 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f6f0c0305c0 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f6f0c068050 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 10 client.16446 fetch_fsmap learned FSMap version 35 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x559a22ffb020 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6f0c036040 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f6f0c048050 con 0x559a22fef1c0 2024-05-09T09:23:54.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 10 client.16446 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 10 client.16446 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 20 client.16446 populate_metadata read hostname 'smithi069' 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a23065aa0 0x559a23067ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 4 client.16446 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559a22f50dc0 con 0x559a23065aa0 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.358+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a23065aa0 0x559a23067ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.359+0000 7f6ef3fff640 10 client.16446 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:54.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.359+0000 7f6f18d5d640 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a23065aa0 0x559a23067ec0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e22986611fc83529 server_cookie=e317b113c479e536 in_seq=0 out_seq=0 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6ef3fff640 1 -- 172.21.15.69:0/2122234996 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559a22f50dc0 con 0x559a23065aa0 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6ef3fff640 10 client.16446 handle_command_reply: tid=0 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 client.16446 shutdown 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 msgr2=0x559a22ff96b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c031ae0 tx=0x7f6f0c031bb0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 shutdown_connections 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559a22fef1c0 0x559a22ff96b0 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559a23065aa0 0x559a23067ec0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559a22fea050 0x559a22ff86a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 --2- 172.21.15.69:0/2122234996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559a22fe9660 0x559a22ff8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6ef3fff640 0 client.16446 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 >> 172.21.15.69:0/2122234996 conn(0x559a22fe08c0 msgr2=0x559a22fe3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 shutdown_connections 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 1 -- 172.21.15.69:0/2122234996 wait complete. 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.410+0000 7f6f1ddad740 20 client.16446 trim_cache size 0 max 16384 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:23:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 msgr2=0x7f6efc0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 0x7f6efc0a50f0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6f0c02ade0 tx=0x7f6f0c0095c0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 msgr2=0x7f6f14197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:23:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7f6f0400c890 tx=0x7f6f0400cd50 comp rx=0 tx=0).stop 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 shutdown_connections 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6efc0a2c30 0x7f6efc0a50f0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6f141075a0 0x7f6f1419f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6f14104c70 0x7f6f14198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 --2- 172.21.15.69:0/1221602719 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6f14102340 0x7f6f14197ac0 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 >> 172.21.15.69:0/1221602719 conn(0x7f6f140fbd30 msgr2=0x7f6f14102c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 shutdown_connections 2024-05-09T09:23:54.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:23:54.411+0000 7f6f1afc1640 1 -- 172.21.15.69:0/1221602719 wait complete. 2024-05-09T09:23:54.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 135543 3.07 MB/sec execute 112 sec latency 4202.044 ms 2024-05-09T09:23:55.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 136003 3.06 MB/sec execute 113 sec latency 4948.031 ms 2024-05-09T09:23:56.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 138629 3.18 MB/sec execute 114 sec latency 16.878 ms 2024-05-09T09:23:56.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:56 smithi163 ceph-mon[34354]: pgmap v519: 129 pgs: 129 active+clean; 46 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 141 op/s 2024-05-09T09:23:56.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:56 smithi103 ceph-mon[34421]: pgmap v519: 129 pgs: 129 active+clean; 46 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 141 op/s 2024-05-09T09:23:56.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:56 smithi069 ceph-mon[30123]: pgmap v519: 129 pgs: 129 active+clean; 46 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 803 KiB/s wr, 141 op/s 2024-05-09T09:23:57.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139034 3.16 MB/sec execute 115 sec latency 531.880 ms 2024-05-09T09:23:58.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139034 3.13 MB/sec execute 116 sec latency 1531.944 ms 2024-05-09T09:23:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:58 smithi163 ceph-mon[34354]: pgmap v520: 129 pgs: 129 active+clean; 47 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 705 KiB/s wr, 94 op/s 2024-05-09T09:23:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:58 smithi103 ceph-mon[34421]: pgmap v520: 129 pgs: 129 active+clean; 47 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 705 KiB/s wr, 94 op/s 2024-05-09T09:23:58.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:58 smithi069 ceph-mon[30123]: pgmap v520: 129 pgs: 129 active+clean; 47 MiB data, 743 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 705 KiB/s wr, 94 op/s 2024-05-09T09:23:59.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139034 3.10 MB/sec execute 117 sec latency 2531.985 ms 2024-05-09T09:23:59.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:23:59 smithi103 ceph-mon[34421]: pgmap v521: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 675 KiB/s wr, 109 op/s 2024-05-09T09:23:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:23:59 smithi069 ceph-mon[30123]: pgmap v521: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 675 KiB/s wr, 109 op/s 2024-05-09T09:23:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:23:59 smithi163 ceph-mon[34354]: pgmap v521: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 675 KiB/s wr, 109 op/s 2024-05-09T09:24:00.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139090 3.08 MB/sec execute 118 sec latency 3280.557 ms 2024-05-09T09:24:01.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139090 3.05 MB/sec execute 119 sec latency 1203.570 ms 2024-05-09T09:24:02.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139090 3.03 MB/sec execute 120 sec latency 2203.614 ms 2024-05-09T09:24:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:02 smithi163 ceph-mon[34354]: pgmap v522: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 498 KiB/s wr, 79 op/s 2024-05-09T09:24:02.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:02 smithi103 ceph-mon[34421]: pgmap v522: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 498 KiB/s wr, 79 op/s 2024-05-09T09:24:02.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:02 smithi069 ceph-mon[30123]: pgmap v522: 129 pgs: 129 active+clean; 44 MiB data, 742 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 498 KiB/s wr, 79 op/s 2024-05-09T09:24:03.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139090 3.00 MB/sec execute 121 sec latency 3203.668 ms 2024-05-09T09:24:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:03 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:24:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:03 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:24:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:03 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:24:04.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139090 2.98 MB/sec execute 122 sec latency 4203.756 ms 2024-05-09T09:24:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:04 smithi163 ceph-mon[34354]: pgmap v523: 129 pgs: 129 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 658 KiB/s wr, 83 op/s 2024-05-09T09:24:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:24:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:24:04.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:24:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:04 smithi103 ceph-mon[34421]: pgmap v523: 129 pgs: 129 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 658 KiB/s wr, 83 op/s 2024-05-09T09:24:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:24:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:24:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:24:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:04 smithi069 ceph-mon[30123]: pgmap v523: 129 pgs: 129 active+clean; 45 MiB data, 729 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 658 KiB/s wr, 83 op/s 2024-05-09T09:24:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:24:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:24:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:24:05.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 139488 2.97 MB/sec execute 123 sec latency 4951.341 ms 2024-05-09T09:24:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:05 smithi163 ceph-mon[34354]: pgmap v524: 129 pgs: 129 active+clean; 45 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 716 KiB/s wr, 89 op/s 2024-05-09T09:24:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:05 smithi103 ceph-mon[34421]: pgmap v524: 129 pgs: 129 active+clean; 45 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 716 KiB/s wr, 89 op/s 2024-05-09T09:24:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:05 smithi069 ceph-mon[30123]: pgmap v524: 129 pgs: 129 active+clean; 45 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 427 KiB/s rd, 716 KiB/s wr, 89 op/s 2024-05-09T09:24:06.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 142133 3.08 MB/sec execute 124 sec latency 19.900 ms 2024-05-09T09:24:07.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 142633 3.06 MB/sec execute 125 sec latency 410.117 ms 2024-05-09T09:24:07.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- 172.21.15.69:0/3762317499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8101890 msgr2=0x7f28c8101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3762317499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8101890 0x7f28c8101cb0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009900 tx=0x7f28bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- 172.21.15.69:0/3762317499 shutdown_connections 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3762317499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8103440 0x7f28c8109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3762317499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8102a80 0x7f28c8102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3762317499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8101890 0x7f28c8101cb0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- 172.21.15.69:0/3762317499 >> 172.21.15.69:0/3762317499 conn(0x7f28c80fce20 msgr2=0x7f28c80ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- 172.21.15.69:0/3762317499 shutdown_connections 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- 172.21.15.69:0/3762317499 wait complete. 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 Processor -- start 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.500+0000 7f28cfe76640 1 -- start start 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 0x7f28c8197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102a80 0x7f28c8198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c810d560 con 0x7f28c8103440 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c810d3e0 con 0x7f28c8102a80 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c810d6e0 con 0x7f28c8101890 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cdc12640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 0x7f28c8197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cdc12640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 0x7f28c8197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:60854/0 (socket says 172.21.15.69:60854) 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32774/0 (socket says 172.21.15.69:32774) 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cdc12640 1 -- 172.21.15.69:0/3790503276 learned_addr learned my addr 172.21.15.69:0/3790503276 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cd411640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102a80 0x7f28c8198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 -- 172.21.15.69:0/3790503276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 msgr2=0x7f28c8197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 0x7f28c8197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 -- 172.21.15.69:0/3790503276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102a80 msgr2=0x7f28c8198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102a80 0x7f28c8198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 -- 172.21.15.69:0/3790503276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c819f760 con 0x7f28c8103440 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28ce413640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28c000ca30 tx=0x7f28c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28baffd640 1 -- 172.21.15.69:0/3790503276 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c0004d20 con 0x7f28c8103440 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c819fa50 con 0x7f28c8103440 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28baffd640 1 -- 172.21.15.69:0/3790503276 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c0005020 con 0x7f28c8103440 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28baffd640 1 -- 172.21.15.69:0/3790503276 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c000f6b0 con 0x7f28c8103440 2024-05-09T09:24:07.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.501+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c819ffd0 con 0x7f28c8103440 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.504+0000 7f28baffd640 1 -- 172.21.15.69:0/3790503276 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f28c0005560 con 0x7f28c8103440 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.504+0000 7f28d2b89740 1 Processor -- start 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 -- start start 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2b050 0x556228d2fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d32650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556228c8a340 con 0x556228d2a660 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556228c32290 con 0x556228d301c0 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28d2b89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556228c86300 con 0x556228d2b050 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32788/0 (socket says 172.21.15.69:32788) 2024-05-09T09:24:07.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 -- 172.21.15.69:0/1606874740 learned_addr learned my addr 172.21.15.69:0/1606874740 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d32650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cdc12640 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2b050 0x556228d2fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28baffd640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 0x7f28b40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 -- 172.21.15.69:0/1606874740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2b050 msgr2=0x556228d2fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2b050 0x556228d2fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 -- 172.21.15.69:0/1606874740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 msgr2=0x556228d32650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d32650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 -- 172.21.15.69:0/1606874740 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556228c568e0 con 0x556228d2a660 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28ce413640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 0x7f28b40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28baffd640 1 -- 172.21.15.69:0/3790503276 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f28c00cf560 con 0x7f28c8103440 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28cd411640 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f28c40101f0 tx=0x7f28c40095d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8e84b54d1b61706 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.505+0000 7f28b3fff640 1 -- 172.21.15.69:0/1606874740 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4017610 con 0x556228d2a660 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28b3fff640 1 -- 172.21.15.69:0/1606874740 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c401c070 con 0x556228d2a660 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28b3fff640 1 -- 172.21.15.69:0/1606874740 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c40159d0 con 0x556228d2a660 2024-05-09T09:24:07.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 -- 172.21.15.69:0/1606874740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 msgr2=0x556228d2aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28ce413640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 0x7f28b40a50f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f28c009b740 tx=0x7f28c00054d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7f28c40101f0 tx=0x7f28c40095d0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 -- 172.21.15.69:0/1606874740 shutdown_connections 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d32650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2b050 0x556228d2fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1606874740 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2a660 0x556228d2aa80 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 -- 172.21.15.69:0/1606874740 >> 172.21.15.69:0/1606874740 conn(0x556228d218c0 msgr2=0x556228d21cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 -- 172.21.15.69:0/1606874740 shutdown_connections 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.506+0000 7f28d2b89740 1 -- 172.21.15.69:0/1606874740 wait complete. 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 -- start start 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 Processor -- start 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d34710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d3bc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556228d33020 con 0x556228d2b050 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556228c45790 con 0x556228d301c0 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556228d331a0 con 0x556228d2a660 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28d2b89740 10 client.0 _ll_register_callbacks cb 0x556228c80a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:32792/0 (socket says 172.21.15.69:32792) 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 -- 172.21.15.69:0/1934011470 learned_addr learned my addr 172.21.15.69:0/1934011470 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cd411640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d34710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cd411640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d3bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 -- 172.21.15.69:0/1934011470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 msgr2=0x556228d34710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d34710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 -- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 msgr2=0x556228d3bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d3bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cdc12640 1 -- 172.21.15.69:0/1934011470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55622899e210 con 0x556228d2b050 2024-05-09T09:24:07.517 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.507+0000 7f28cd411640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d34710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28cdc12640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009230 tx=0x7f28bc004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c427f11523ecc8c4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28bc035040 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c401c070 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f28bc0043f0 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f28bc005020 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 10 client.16458 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556228a6a620 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c401c070 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f28bc0035f0 con 0x556228d2b050 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 10 client.16458 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 10 client.16458 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 20 client.16458 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:07.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228d40250 0x556228d42690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 4 client.16458 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.508+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556228c97a70 con 0x556228d40250 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.509+0000 7f28cd411640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228d40250 0x556228d42690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.509+0000 7f28b17fa640 10 client.16458 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.509+0000 7f28cd411640 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228d40250 0x556228d42690 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8145f9b8ad05e8fa server_cookie=8e888d8893e820b4 in_seq=0 out_seq=0 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28b17fa640 1 -- 172.21.15.69:0/1934011470 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556228c97a70 con 0x556228d40250 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28b17fa640 10 client.16458 handle_command_reply: tid=0 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 client.16458 shutdown 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 msgr2=0x556228d34c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7f28bc009230 tx=0x7f28bc004040 comp rx=0 tx=0).stop 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 shutdown_connections 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d301c0 0x556228d3bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d2b050 0x556228d34c50 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228d40250 0x556228d42690 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 --2- 172.21.15.69:0/1934011470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d34710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28b17fa640 0 client.16458 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:07.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 >> 172.21.15.69:0/1934011470 conn(0x556228d218c0 msgr2=0x556228d20a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 shutdown_connections 2024-05-09T09:24:07.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 1 -- 172.21.15.69:0/1934011470 wait complete. 2024-05-09T09:24:07.520 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.510+0000 7f28d2b89740 20 client.16458 trim_cache size 0 max 16384 2024-05-09T09:24:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 Processor -- start 2024-05-09T09:24:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 -- start start 2024-05-09T09:24:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d2c1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d3bc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81aaae0 con 0x556228d301c0 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81a83b0 con 0x556228d2b050 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28d2b89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81a6200 con 0x556228d2a660 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57524/0 (socket says 172.21.15.69:57524) 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 -- 172.21.15.69:0/2877124142 learned_addr learned my addr 172.21.15.69:0/2877124142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cdc12640 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d3bc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 -- 172.21.15.69:0/2877124142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 msgr2=0x556228d2c1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d2c1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 -- 172.21.15.69:0/2877124142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 msgr2=0x556228d3bc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d3bc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 -- 172.21.15.69:0/2877124142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556228d42d10 con 0x556228d301c0 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28cd411640 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x556228d2b8b0 tx=0x7f28c40046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc0b6db774760fbe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28b37fe640 1 -- 172.21.15.69:0/2877124142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4028070 con 0x556228d301c0 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28b37fe640 1 -- 172.21.15.69:0/2877124142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4031070 con 0x556228d301c0 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.525+0000 7f28b37fe640 1 -- 172.21.15.69:0/2877124142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c402b040 con 0x556228d301c0 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- 172.21.15.69:0/2877124142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 msgr2=0x556228d2c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x556228d2b8b0 tx=0x7f28c40046c0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- 172.21.15.69:0/2877124142 shutdown_connections 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d2c710 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d3bc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 --2- 172.21.15.69:0/2877124142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d2c1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- 172.21.15.69:0/2877124142 >> 172.21.15.69:0/2877124142 conn(0x556228d218c0 msgr2=0x556228d21ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- 172.21.15.69:0/2877124142 shutdown_connections 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- 172.21.15.69:0/2877124142 wait complete. 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 -- start start 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.526+0000 7f28d2b89740 1 Processor -- start 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d39580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d39ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81a6870 con 0x556228d301c0 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81a7ce0 con 0x556228d2b050 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c81a6f40 con 0x556228d2a660 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 10 client.0 _ll_register_callbacks cb 0x7f28bc009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57532/0 (socket says 172.21.15.69:57532) 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 -- 172.21.15.69:0/3768759315 learned_addr learned my addr 172.21.15.69:0/3768759315 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cdc12640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d39ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d39580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 -- 172.21.15.69:0/3768759315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 msgr2=0x556228d39580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d39580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 -- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 msgr2=0x556228d39ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:07.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d39ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 -- 172.21.15.69:0/3768759315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556228d3b0c0 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cdc12640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d39ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28cd411640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f28c4001eb0 tx=0x7f28c4023040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4f988e3a13156cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28c4028070 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f28c402cde0 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.527+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28c4031070 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f28c402c560 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 10 client.16464 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x556228da6a20 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28c4031070 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f28c4015ba0 con 0x556228d301c0 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 10 client.16464 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:07.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 10 client.16464 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 20 client.16464 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228da7140 0x556228da9560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 4 client.16464 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556228c91dc0 con 0x556228da7140 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28cdc12640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228da7140 0x556228da9560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28cdc12640 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228da7140 0x556228da9560 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3486f68e3f2ee9df server_cookie=2ba0fd87fe46915d in_seq=0 out_seq=0 2024-05-09T09:24:07.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.528+0000 7f28b17fa640 10 client.16464 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28b17fa640 1 -- 172.21.15.69:0/3768759315 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556228c91dc0 con 0x556228da7140 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28b17fa640 10 client.16464 handle_command_reply: tid=0 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 client.16464 shutdown 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 msgr2=0x556228d3aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7f28c4001eb0 tx=0x7f28c4023040 comp rx=0 tx=0).stop 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 shutdown_connections 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556228d301c0 0x556228d3aad0 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556228d2b050 0x556228d39ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556228da7140 0x556228da9560 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 --2- 172.21.15.69:0/3768759315 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556228d2a660 0x556228d39580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28b17fa640 0 client.16464 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 >> 172.21.15.69:0/3768759315 conn(0x556228d218c0 msgr2=0x556228d346b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 shutdown_connections 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 1 -- 172.21.15.69:0/3768759315 wait complete. 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.572+0000 7f28d2b89740 20 client.16464 trim_cache size 0 max 16384 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:24:07.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:24:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 msgr2=0x7f28b40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 0x7f28b40a50f0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f28c009b740 tx=0x7f28c00054d0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 msgr2=0x7f28c819f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7f28c000ca30 tx=0x7f28c000ce30 comp rx=0 tx=0).stop 2024-05-09T09:24:07.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 shutdown_connections 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28b40a2c30 0x7f28b40a50f0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c8103440 0x7f28c819f0f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c8102a80 0x7f28c8198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 --2- 172.21.15.69:0/3790503276 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c8101890 0x7f28c8197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 >> 172.21.15.69:0/3790503276 conn(0x7f28c80fce20 msgr2=0x7f28c8105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 shutdown_connections 2024-05-09T09:24:07.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:07.573+0000 7f28cfe76640 1 -- 172.21.15.69:0/3790503276 wait complete. 2024-05-09T09:24:08.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 142633 3.04 MB/sec execute 126 sec latency 1410.149 ms 2024-05-09T09:24:08.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:08 smithi163 ceph-mon[34354]: pgmap v525: 129 pgs: 129 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 833 KiB/s wr, 91 op/s 2024-05-09T09:24:08.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:08 smithi103 ceph-mon[34421]: pgmap v525: 129 pgs: 129 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 833 KiB/s wr, 91 op/s 2024-05-09T09:24:08.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:08 smithi069 ceph-mon[30123]: pgmap v525: 129 pgs: 129 active+clean; 44 MiB data, 738 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 833 KiB/s wr, 91 op/s 2024-05-09T09:24:09.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 142633 3.01 MB/sec execute 127 sec latency 2410.192 ms 2024-05-09T09:24:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:09 smithi103 ceph-mon[34421]: pgmap v526: 129 pgs: 129 active+clean; 45 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 920 KiB/s wr, 110 op/s 2024-05-09T09:24:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:09 smithi069 ceph-mon[30123]: pgmap v526: 129 pgs: 129 active+clean; 45 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 920 KiB/s wr, 110 op/s 2024-05-09T09:24:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:09 smithi163 ceph-mon[34354]: pgmap v526: 129 pgs: 129 active+clean; 45 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 920 KiB/s wr, 110 op/s 2024-05-09T09:24:10.487 INFO:tasks.workunit.client.0.smithi069.stdout: 1 143066 3.00 MB/sec execute 128 sec latency 3154.489 ms 2024-05-09T09:24:11.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 145699 3.11 MB/sec execute 129 sec latency 19.556 ms 2024-05-09T09:24:12.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 146126 3.09 MB/sec execute 130 sec latency 508.207 ms 2024-05-09T09:24:12.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:12 smithi163 ceph-mon[34354]: pgmap v527: 129 pgs: 129 active+clean; 45 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-05-09T09:24:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:12 smithi103 ceph-mon[34421]: pgmap v527: 129 pgs: 129 active+clean; 45 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-05-09T09:24:12.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:12 smithi069 ceph-mon[30123]: pgmap v527: 129 pgs: 129 active+clean; 45 MiB data, 770 MiB used, 1.0 TiB / 1.0 TiB avail; 446 KiB/s rd, 890 KiB/s wr, 84 op/s 2024-05-09T09:24:13.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 146126 3.06 MB/sec execute 131 sec latency 1508.276 ms 2024-05-09T09:24:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:13 smithi103 ceph-mon[34421]: pgmap v528: 129 pgs: 129 active+clean; 46 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-09T09:24:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:13 smithi069 ceph-mon[30123]: pgmap v528: 129 pgs: 129 active+clean; 46 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-09T09:24:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:13 smithi163 ceph-mon[34354]: pgmap v528: 129 pgs: 129 active+clean; 46 MiB data, 757 MiB used, 1.0 TiB / 1.0 TiB avail; 676 KiB/s rd, 1.0 MiB/s wr, 104 op/s 2024-05-09T09:24:14.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 146126 3.04 MB/sec execute 132 sec latency 2508.311 ms 2024-05-09T09:24:15.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 146440 3.02 MB/sec execute 133 sec latency 3254.409 ms 2024-05-09T09:24:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:15 smithi103 ceph-mon[34421]: pgmap v529: 129 pgs: 129 active+clean; 47 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 954 KiB/s wr, 125 op/s 2024-05-09T09:24:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:15 smithi069 ceph-mon[30123]: pgmap v529: 129 pgs: 129 active+clean; 47 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 954 KiB/s wr, 125 op/s 2024-05-09T09:24:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:15 smithi163 ceph-mon[34354]: pgmap v529: 129 pgs: 129 active+clean; 47 MiB data, 767 MiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 954 KiB/s wr, 125 op/s 2024-05-09T09:24:16.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 149110 3.13 MB/sec execute 134 sec latency 11.273 ms 2024-05-09T09:24:17.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 150124 3.13 MB/sec execute 135 sec latency 43.163 ms 2024-05-09T09:24:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:17 smithi103 ceph-mon[34421]: pgmap v530: 129 pgs: 129 active+clean; 48 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-05-09T09:24:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:17 smithi069 ceph-mon[30123]: pgmap v530: 129 pgs: 129 active+clean; 48 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-05-09T09:24:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:17 smithi163 ceph-mon[34354]: pgmap v530: 129 pgs: 129 active+clean; 48 MiB data, 751 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 1.0 MiB/s wr, 130 op/s 2024-05-09T09:24:18.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 152843 3.23 MB/sec execute 136 sec latency 12.961 ms 2024-05-09T09:24:19.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 154465 3.25 MB/sec execute 137 sec latency 25.212 ms 2024-05-09T09:24:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:19 smithi103 ceph-mon[34421]: pgmap v531: 129 pgs: 129 active+clean; 50 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-09T09:24:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:19 smithi069 ceph-mon[30123]: pgmap v531: 129 pgs: 129 active+clean; 50 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-09T09:24:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:19 smithi163 ceph-mon[34354]: pgmap v531: 129 pgs: 129 active+clean; 50 MiB data, 746 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 1.4 MiB/s wr, 181 op/s 2024-05-09T09:24:20.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 156600 3.32 MB/sec execute 138 sec latency 9.537 ms 2024-05-09T09:24:20.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 -- 172.21.15.69:0/2341020729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 msgr2=0x7ff8c0101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/2341020729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c0101e00 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009900 tx=0x7ff8b002a410 comp rx=0 tx=0).stop 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 -- 172.21.15.69:0/2341020729 shutdown_connections 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/2341020729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 0x7ff8c0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/2341020729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c0101e00 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/2341020729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 -- 172.21.15.69:0/2341020729 >> 172.21.15.69:0/2341020729 conn(0x7ff8c00fbd00 msgr2=0x7ff8c00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.663+0000 7ff8c6b42640 1 -- 172.21.15.69:0/2341020729 shutdown_connections 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.664+0000 7ff8c6b42640 1 -- 172.21.15.69:0/2341020729 wait complete. 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.664+0000 7ff8c6b42640 1 Processor -- start 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.664+0000 7ff8c6b42640 1 -- start start 2024-05-09T09:24:20.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c019c820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 0x7ff8c019ee40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c010c3f0 con 0x7ff8c0101980 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c010c270 con 0x7ff8c0100790 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c6b42640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c010c570 con 0x7ff8c0102340 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c019c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35040/0 (socket says 172.21.15.69:35040) 2024-05-09T09:24:20.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 -- 172.21.15.69:0/4113828663 learned_addr learned my addr 172.21.15.69:0/4113828663 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c50df640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 0x7ff8c019ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 -- 172.21.15.69:0/4113828663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 msgr2=0x7ff8c019ee40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 0x7ff8c019ee40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 -- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 msgr2=0x7ff8c019c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c019c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 -- 172.21.15.69:0/4113828663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff8c019f750 con 0x7ff8c0100790 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.665+0000 7ff8c48de640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400e9c0 tx=0x7ff8b400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.666+0000 7ff8bdffb640 1 -- 172.21.15.69:0/4113828663 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b400cd80 con 0x7ff8c0100790 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.666+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff8c019f9e0 con 0x7ff8c0100790 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.666+0000 7ff8bdffb640 1 -- 172.21.15.69:0/4113828663 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b40035f0 con 0x7ff8c0100790 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.666+0000 7ff8bdffb640 1 -- 172.21.15.69:0/4113828663 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b4010690 con 0x7ff8c0100790 2024-05-09T09:24:20.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.666+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff8c019ff00 con 0x7ff8c0100790 2024-05-09T09:24:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8bdffb640 1 -- 172.21.15.69:0/4113828663 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff8b40037a0 con 0x7ff8c0100790 2024-05-09T09:24:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 Processor -- start 2024-05-09T09:24:20.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 -- start start 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7250 0x55fcdbff7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdbffc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbf5b340 con 0x55fcdbff7250 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbf03290 con 0x55fcdbffcdb0 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c9855740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbf57300 con 0x55fcdbff7c40 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8bdffb640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 0x7ff8a80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdbffc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35042/0 (socket says 172.21.15.69:35042) 2024-05-09T09:24:20.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.669+0000 7ff8bffff640 1 -- 172.21.15.69:0/535741884 learned_addr learned my addr 172.21.15.69:0/535741884 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8c50df640 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7250 0x55fcdbff7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8c50df640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 0x7ff8a80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bdffb640 1 -- 172.21.15.69:0/4113828663 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7ff8b40cf840 con 0x7ff8c0100790 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 -- 172.21.15.69:0/535741884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 msgr2=0x55fcdbffc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdbffc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 -- 172.21.15.69:0/535741884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7250 msgr2=0x55fcdbff7670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7250 0x55fcdbff7670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 -- 172.21.15.69:0/535741884 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdbf278e0 con 0x55fcdbffcdb0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8c50df640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 0x7ff8a80a50f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8005c40 tx=0x7ff8b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8bffff640 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff8b002a930 tx=0x7ff8b0005de0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=58b9ab652861dd73 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8a6ffd640 1 -- 172.21.15.69:0/535741884 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0040070 con 0x55fcdbffcdb0 2024-05-09T09:24:20.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8a6ffd640 1 -- 172.21.15.69:0/535741884 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b0004030 con 0x55fcdbffcdb0 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.670+0000 7ff8a6ffd640 1 -- 172.21.15.69:0/535741884 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0035040 con 0x55fcdbffcdb0 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 -- 172.21.15.69:0/535741884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 msgr2=0x55fcdbfff240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7ff8b002a930 tx=0x7ff8b0005de0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 -- 172.21.15.69:0/535741884 shutdown_connections 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdbfff240 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdbffc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 --2- 172.21.15.69:0/535741884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7250 0x55fcdbff7670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 -- 172.21.15.69:0/535741884 >> 172.21.15.69:0/535741884 conn(0x55fcdbff28c0 msgr2=0x55fcdbff2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 -- 172.21.15.69:0/535741884 shutdown_connections 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.671+0000 7ff8c9855740 1 -- 172.21.15.69:0/535741884 wait complete. 2024-05-09T09:24:20.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 -- start start 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 Processor -- start 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdc001920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbfffd00 con 0x55fcdbffcdb0 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbf16790 con 0x55fcdbff7250 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fcdbfffe80 con 0x55fcdbff7c40 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c9855740 10 client.0 _ll_register_callbacks cb 0x55fcdbf51a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35054/0 (socket says 172.21.15.69:35054) 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdc001920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.672+0000 7ff8c48de640 1 -- 172.21.15.69:0/2400003670 learned_addr learned my addr 172.21.15.69:0/2400003670 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c50df640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 -- 172.21.15.69:0/2400003670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 msgr2=0x55fcdc001920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdc001920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 -- 172.21.15.69:0/2400003670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 msgr2=0x55fcdc008960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 -- 172.21.15.69:0/2400003670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdbc6f210 con 0x55fcdbff7250 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c50df640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdc001920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4098180 tx=0x7ff8b409c4e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c8b24e8b41bf3388 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b40a4990 con 0x55fcdbff7250 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7ff8b40bf030 con 0x55fcdbff7250 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b0004030 con 0x55fcdbff7250 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff8b4003b30 con 0x55fcdbff7250 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c9855740 10 client.25793 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:20.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fcdbd3b620 con 0x55fcdbff7250 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.673+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b0004030 con 0x55fcdbff7250 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7ff8b409ac70 con 0x55fcdbff7250 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 10 client.25793 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 10 client.25793 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 20 client.25793 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc00ceb0 0x55fcdc00f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 4 client.25793 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcdbf68a70 con 0x55fcdc00ceb0 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc00ceb0 0x55fcdc00f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc00ceb0 0x55fcdc00f340 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16de69b4ec62866e server_cookie=63b54e59998220e4 in_seq=0 out_seq=0 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.674+0000 7ff89ffff640 10 client.25793 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff89ffff640 1 -- 172.21.15.69:0/2400003670 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcdbf68a70 con 0x55fcdc00ceb0 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff89ffff640 10 client.25793 handle_command_reply: tid=0 2024-05-09T09:24:20.681 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 client.25793 shutdown 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 msgr2=0x55fcdc0013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7ff8b4098180 tx=0x7ff8b409c4e0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 shutdown_connections 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc00ceb0 0x55fcdc00f340 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7c40 0x55fcdc001920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2400003670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7250 0x55fcdc0013e0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff89ffff640 0 client.25793 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 >> 172.21.15.69:0/2400003670 conn(0x55fcdbff28c0 msgr2=0x55fcdbff1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 shutdown_connections 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 1 -- 172.21.15.69:0/2400003670 wait complete. 2024-05-09T09:24:20.682 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.675+0000 7ff8c9855740 20 client.25793 trim_cache size 0 max 16384 2024-05-09T09:24:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 Processor -- start 2024-05-09T09:24:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 -- start start 2024-05-09T09:24:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.692 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01aaaa0 con 0x55fcdbffcdb0 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01a8370 con 0x55fcdbff7c40 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c9855740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01a61c0 con 0x55fcdbff7250 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c50df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35068/0 (socket says 172.21.15.69:35068) 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8c48de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:50852/0 (socket says 172.21.15.69:50852) 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.692+0000 7ff8bffff640 1 -- 172.21.15.69:0/1682680108 learned_addr learned my addr 172.21.15.69:0/1682680108 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 -- 172.21.15.69:0/1682680108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 msgr2=0x55fcdc014c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 -- 172.21.15.69:0/1682680108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 msgr2=0x55fcdc008ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 -- 172.21.15.69:0/1682680108 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcdbff8020 con 0x55fcdbff7c40 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8c48de640 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8bffff640 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009900 tx=0x7ff8b0031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5d0a056c85ad7ce5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8a67fc640 1 -- 172.21.15.69:0/1682680108 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0040040 con 0x55fcdbff7c40 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8a67fc640 1 -- 172.21.15.69:0/1682680108 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b0004040 con 0x55fcdbff7c40 2024-05-09T09:24:20.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.693+0000 7ff8a67fc640 1 -- 172.21.15.69:0/1682680108 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0035040 con 0x55fcdbff7c40 2024-05-09T09:24:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- 172.21.15.69:0/1682680108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 msgr2=0x55fcdc015190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff8b0009900 tx=0x7ff8b0031400 comp rx=0 tx=0).stop 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- 172.21.15.69:0/1682680108 shutdown_connections 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbffcdb0 0x55fcdc008ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbff7c40 0x55fcdc015190 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- 172.21.15.69:0/1682680108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc014c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- 172.21.15.69:0/1682680108 >> 172.21.15.69:0/1682680108 conn(0x55fcdbff28c0 msgr2=0x55fcdbffc470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- 172.21.15.69:0/1682680108 shutdown_connections 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- 172.21.15.69:0/1682680108 wait complete. 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- start start 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 Processor -- start 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc005450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdc006890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01a6830 con 0x55fcdbff7c40 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01a7ca0 con 0x55fcdbffcdb0 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff8c01a6f00 con 0x55fcdbff7250 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.694+0000 7ff8c9855740 10 client.0 _ll_register_callbacks cb 0x7ff8b4003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53014/0 (socket says 172.21.15.69:53014) 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 -- 172.21.15.69:0/2424142299 learned_addr learned my addr 172.21.15.69:0/2424142299 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c50df640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdc006890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc005450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 -- 172.21.15.69:0/2424142299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 msgr2=0x55fcdc005450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc005450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 -- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 msgr2=0x55fcdc006890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdc006890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 -- 172.21.15.69:0/2424142299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcdc006e80 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c50df640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdc006890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8bffff640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00314e0 tx=0x7ff8b0005de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2423fcf9dcdb74fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff89ffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff8b0040030 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7ff8b00043f0 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff8b0004040 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7ff8b0065050 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c9855740 10 client.16482 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55fcdc008190 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.695+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff8b0004040 con 0x55fcdbff7c40 2024-05-09T09:24:20.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7ff8b0060030 con 0x55fcdbff7c40 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 10 client.16482 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 10 client.16482 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 20 client.16482 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc0738b0 0x55fcdc075ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 4 client.16482 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fcdbf62dc0 con 0x55fcdc0738b0 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc0738b0 0x55fcdc075ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff8c48de640 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc0738b0 0x55fcdc075ca0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e1c1911d4f53ece server_cookie=21d09ab28576b515 in_seq=0 out_seq=0 2024-05-09T09:24:20.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.696+0000 7ff89ffff640 10 client.16482 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff89ffff640 1 -- 172.21.15.69:0/2424142299 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fcdbf62dc0 con 0x55fcdc0738b0 2024-05-09T09:24:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff89ffff640 10 client.16482 handle_command_reply: tid=0 2024-05-09T09:24:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 client.16482 shutdown 2024-05-09T09:24:20.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 msgr2=0x55fcdc005990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7ff8b00314e0 tx=0x7ff8b0005de0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 shutdown_connections 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcdbffcdb0 0x55fcdc006890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcdbff7c40 0x55fcdc005990 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcdc0738b0 0x55fcdc075ca0 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 --2- 172.21.15.69:0/2424142299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcdbff7250 0x55fcdc005450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff89ffff640 0 client.16482 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.768+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 >> 172.21.15.69:0/2424142299 conn(0x55fcdbff28c0 msgr2=0x55fcdc008ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 shutdown_connections 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c9855740 1 -- 172.21.15.69:0/2424142299 wait complete. 2024-05-09T09:24:20.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c9855740 20 client.16482 trim_cache size 0 max 16384 2024-05-09T09:24:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:24:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:24:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:24:20.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 msgr2=0x7ff8a80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 0x7ff8a80a50f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7ff8b8005c40 tx=0x7ff8b8009500 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 msgr2=0x7ff8c0069a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff8b400e9c0 tx=0x7ff8b400ee80 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 shutdown_connections 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff8a80a2c30 0x7ff8a80a50f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff8c0102340 0x7ff8c019ee40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff8c0101980 0x7ff8c019c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 --2- 172.21.15.69:0/4113828663 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff8c0100790 0x7ff8c0069a50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 >> 172.21.15.69:0/4113828663 conn(0x7ff8c00fbd00 msgr2=0x7ff8c0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:20.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 shutdown_connections 2024-05-09T09:24:20.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:20.769+0000 7ff8c6b42640 1 -- 172.21.15.69:0/4113828663 wait complete. 2024-05-09T09:24:21.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 158584 3.36 MB/sec execute 139 sec latency 65.915 ms 2024-05-09T09:24:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:21 smithi103 ceph-mon[34421]: pgmap v532: 129 pgs: 129 active+clean; 50 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-09T09:24:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:21 smithi069 ceph-mon[30123]: pgmap v532: 129 pgs: 129 active+clean; 50 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-09T09:24:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:21 smithi163 ceph-mon[34354]: pgmap v532: 129 pgs: 129 active+clean; 50 MiB data, 747 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 1.3 MiB/s wr, 152 op/s 2024-05-09T09:24:22.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 160362 3.41 MB/sec execute 140 sec latency 15.221 ms 2024-05-09T09:24:23.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 163263 3.51 MB/sec execute 141 sec latency 13.768 ms 2024-05-09T09:24:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:23 smithi103 ceph-mon[34421]: pgmap v533: 129 pgs: 129 active+clean; 50 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T09:24:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:23 smithi069 ceph-mon[30123]: pgmap v533: 129 pgs: 129 active+clean; 50 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T09:24:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:23 smithi163 ceph-mon[34354]: pgmap v533: 129 pgs: 129 active+clean; 50 MiB data, 758 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 1.4 MiB/s wr, 197 op/s 2024-05-09T09:24:24.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 164636 3.54 MB/sec execute 142 sec latency 11.894 ms 2024-05-09T09:24:25.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 167086 3.61 MB/sec execute 143 sec latency 14.241 ms 2024-05-09T09:24:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:25 smithi103 ceph-mon[34421]: pgmap v534: 129 pgs: 129 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-05-09T09:24:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:25 smithi069 ceph-mon[30123]: pgmap v534: 129 pgs: 129 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-05-09T09:24:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:25 smithi163 ceph-mon[34354]: pgmap v534: 129 pgs: 129 active+clean; 49 MiB data, 766 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 MiB/s rd, 1.5 MiB/s wr, 266 op/s 2024-05-09T09:24:26.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 168933 3.62 MB/sec execute 144 sec latency 18.611 ms 2024-05-09T09:24:27.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 170745 3.69 MB/sec execute 145 sec latency 50.476 ms 2024-05-09T09:24:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:27 smithi103 ceph-mon[34421]: pgmap v535: 129 pgs: 129 active+clean; 51 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 262 op/s 2024-05-09T09:24:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:27 smithi069 ceph-mon[30123]: pgmap v535: 129 pgs: 129 active+clean; 51 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 262 op/s 2024-05-09T09:24:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:27 smithi163 ceph-mon[34354]: pgmap v535: 129 pgs: 129 active+clean; 51 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 1.6 MiB/s wr, 262 op/s 2024-05-09T09:24:28.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 173250 3.75 MB/sec execute 146 sec latency 13.259 ms 2024-05-09T09:24:29.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 174772 3.78 MB/sec execute 147 sec latency 13.860 ms 2024-05-09T09:24:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:29 smithi103 ceph-mon[34421]: pgmap v536: 129 pgs: 129 active+clean; 54 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 317 op/s 2024-05-09T09:24:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:29 smithi069 ceph-mon[30123]: pgmap v536: 129 pgs: 129 active+clean; 54 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 317 op/s 2024-05-09T09:24:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:29 smithi163 ceph-mon[34354]: pgmap v536: 129 pgs: 129 active+clean; 54 MiB data, 778 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.8 MiB/s wr, 317 op/s 2024-05-09T09:24:30.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 177599 3.87 MB/sec execute 148 sec latency 19.048 ms 2024-05-09T09:24:31.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 179061 3.88 MB/sec execute 149 sec latency 15.480 ms 2024-05-09T09:24:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:31 smithi103 ceph-mon[34421]: pgmap v537: 129 pgs: 129 active+clean; 54 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 259 op/s 2024-05-09T09:24:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:31 smithi069 ceph-mon[30123]: pgmap v537: 129 pgs: 129 active+clean; 54 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 259 op/s 2024-05-09T09:24:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:31 smithi163 ceph-mon[34354]: pgmap v537: 129 pgs: 129 active+clean; 54 MiB data, 784 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 1.4 MiB/s wr, 259 op/s 2024-05-09T09:24:32.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 181376 3.95 MB/sec execute 150 sec latency 13.464 ms 2024-05-09T09:24:33.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 183680 4.00 MB/sec execute 151 sec latency 13.898 ms 2024-05-09T09:24:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:33 smithi103 ceph-mon[34421]: pgmap v538: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 309 op/s 2024-05-09T09:24:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:33 smithi069 ceph-mon[30123]: pgmap v538: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 309 op/s 2024-05-09T09:24:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:33 smithi163 ceph-mon[34354]: pgmap v538: 129 pgs: 129 active+clean; 56 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 MiB/s rd, 1.6 MiB/s wr, 309 op/s 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 -- 172.21.15.69:0/4062230108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac103450 msgr2=0x7f12ac109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 --2- 172.21.15.69:0/4062230108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac103450 0x7f12ac109ce0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7f129c009900 tx=0x7f129c02a410 comp rx=0 tx=0).stop 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 -- 172.21.15.69:0/4062230108 shutdown_connections 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 --2- 172.21.15.69:0/4062230108 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac103450 0x7f12ac109ce0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 --2- 172.21.15.69:0/4062230108 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 --2- 172.21.15.69:0/4062230108 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac1018a0 0x7f12ac101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 -- 172.21.15.69:0/4062230108 >> 172.21.15.69:0/4062230108 conn(0x7f12ac0fcdf0 msgr2=0x7f12ac0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 -- 172.21.15.69:0/4062230108 shutdown_connections 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.859+0000 7f12b0916640 1 -- 172.21.15.69:0/4062230108 wait complete. 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 Processor -- start 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 -- start start 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac103450 0x7f12ac19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac10d570 con 0x7f12ac1018a0 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac10d3f0 con 0x7f12ac102a90 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12b0916640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac10d6f0 con 0x7f12ac103450 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12aa59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12aa59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34414/0 (socket says 172.21.15.69:34414) 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12aa59c640 1 -- 172.21.15.69:0/2687843781 learned_addr learned my addr 172.21.15.69:0/2687843781 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.860+0000 7f12aad9d640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac103450 0x7f12ac19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 -- 172.21.15.69:0/2687843781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac103450 msgr2=0x7f12ac19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac103450 0x7f12ac19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 -- 172.21.15.69:0/2687843781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 msgr2=0x7f12ac197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 -- 172.21.15.69:0/2687843781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f12ac19f680 con 0x7f12ac1018a0 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12aa59c640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f12a400eaa0 tx=0x7f12a400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12a37fe640 1 -- 172.21.15.69:0/2687843781 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4016070 con 0x7f12ac1018a0 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f12ac19f970 con 0x7f12ac1018a0 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12a37fe640 1 -- 172.21.15.69:0/2687843781 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12a4003e50 con 0x7f12ac1018a0 2024-05-09T09:24:33.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12a37fe640 1 -- 172.21.15.69:0/2687843781 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a40116d0 con 0x7f12ac1018a0 2024-05-09T09:24:33.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.861+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f12ac19fe70 con 0x7f12ac1018a0 2024-05-09T09:24:33.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.863+0000 7f12a37fe640 1 -- 172.21.15.69:0/2687843781 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f12a4019070 con 0x7f12ac1018a0 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.863+0000 7f12b361c740 1 Processor -- start 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 -- start start 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e69ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e69d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c89e5f5340 con 0x55c89e695660 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c89e59d290 con 0x55c89e696050 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12b361c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c89e5f1300 con 0x55c89e69b1c0 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aa59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e69ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34424/0 (socket says 172.21.15.69:34424) 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aad9d640 1 -- 172.21.15.69:0/3710582958 learned_addr learned my addr 172.21.15.69:0/3710582958 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e69d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a37fe640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 0x7f12980a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 -- 172.21.15.69:0/3710582958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 msgr2=0x55c89e69ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e69ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 -- 172.21.15.69:0/3710582958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 msgr2=0x55c89e695a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12a9d9b640 1 -- 172.21.15.69:0/3710582958 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89e5c18e0 con 0x55c89e69b1c0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aad9d640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.864+0000 7f12aa59c640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e69ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12aad9d640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 0x7f12980a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12a37fe640 1 -- 172.21.15.69:0/2687843781 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f12a40cf370 con 0x7f12ac1018a0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e69d650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1294009fa0 tx=0x7f12940118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fbc56fc1cf466878 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12a0ff9640 1 -- 172.21.15.69:0/3710582958 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294004030 con 0x55c89e69b1c0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12a0ff9640 1 -- 172.21.15.69:0/3710582958 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1294005500 con 0x55c89e69b1c0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.865+0000 7f12a0ff9640 1 -- 172.21.15.69:0/3710582958 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1294004e10 con 0x55c89e69b1c0 2024-05-09T09:24:33.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- 172.21.15.69:0/3710582958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 msgr2=0x55c89e69d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12aad9d640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 0x7f12980a50f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f129c031440 tx=0x7f129c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e69d650 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f1294009fa0 tx=0x7f12940118d0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- 172.21.15.69:0/3710582958 shutdown_connections 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e69d650 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e69ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- 172.21.15.69:0/3710582958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e695a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- 172.21.15.69:0/3710582958 >> 172.21.15.69:0/3710582958 conn(0x55c89e68c8c0 msgr2=0x55c89e68ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- 172.21.15.69:0/3710582958 shutdown_connections 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- 172.21.15.69:0/3710582958 wait complete. 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- start start 2024-05-09T09:24:33.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 Processor -- start 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e697db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e699300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c89e69e0e0 con 0x55c89e696050 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c89e5b0790 con 0x55c89e69b1c0 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c89e69e260 con 0x55c89e695660 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12b361c740 10 client.0 _ll_register_callbacks cb 0x55c89e5eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12a9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12aad9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e699300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12a9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34438/0 (socket says 172.21.15.69:34438) 2024-05-09T09:24:33.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12a9d9b640 1 -- 172.21.15.69:0/379910505 learned_addr learned my addr 172.21.15.69:0/379910505 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.866+0000 7f12aa59c640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e697db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 -- 172.21.15.69:0/379910505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 msgr2=0x55c89e697db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e697db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 -- 172.21.15.69:0/379910505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 msgr2=0x55c89e6982f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 -- 172.21.15.69:0/379910505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89e309210 con 0x55c89e69b1c0 2024-05-09T09:24:33.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12aad9d640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e699300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55c89e6968b0 tx=0x7f129c02a410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9dc3cff54e072ad6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f1291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129c035040 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1294005500 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f129c031dc0 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f129c058050 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12b361c740 10 client.25811 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c89e3d5620 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.867+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1294005500 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f129c03e840 con 0x55c89e69b1c0 2024-05-09T09:24:33.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 10 client.25811 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 10 client.25811 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 20 client.25811 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e6af720 0x55c89e6b1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 4 client.25811 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55c89e602a70 con 0x55c89e6af720 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12aa59c640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e6af720 0x55c89e6b1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f12aa59c640 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e6af720 0x55c89e6b1b50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10ba642537203481 server_cookie=d1a1360e97467335 in_seq=0 out_seq=0 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.868+0000 7f1291ffb640 10 client.25811 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f1291ffb640 1 -- 172.21.15.69:0/379910505 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55c89e602a70 con 0x55c89e6af720 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f1291ffb640 10 client.25811 handle_command_reply: tid=0 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 client.25811 shutdown 2024-05-09T09:24:33.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 msgr2=0x55c89e699300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e699300 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55c89e6968b0 tx=0x7f129c02a410 comp rx=0 tx=0).stop 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 shutdown_connections 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e699300 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6982f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e6af720 0x55c89e6b1b50 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 --2- 172.21.15.69:0/379910505 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e697db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f1291ffb640 0 client.25811 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 >> 172.21.15.69:0/379910505 conn(0x55c89e68c8c0 msgr2=0x55c89e68ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 shutdown_connections 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 1 -- 172.21.15.69:0/379910505 wait complete. 2024-05-09T09:24:33.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.869+0000 7f12b361c740 20 client.25811 trim_cache size 0 max 16384 2024-05-09T09:24:33.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.884+0000 7f12b361c740 1 Processor -- start 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.884+0000 7f12b361c740 1 -- start start 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e68cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e6b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1aa970 con 0x55c89e696050 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1a8240 con 0x55c89e69b1c0 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12b361c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1a6090 con 0x55c89e695660 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34440/0 (socket says 172.21.15.69:34440) 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 -- 172.21.15.69:0/4661872 learned_addr learned my addr 172.21.15.69:0/4661872 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12aa59c640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e68cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12aad9d640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e6b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 -- 172.21.15.69:0/4661872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 msgr2=0x55c89e68cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e68cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 -- 172.21.15.69:0/4661872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 msgr2=0x55c89e6b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e6b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 -- 172.21.15.69:0/4661872 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c89e68af00 con 0x55c89e696050 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12aa59c640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e68cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12aad9d640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e6b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1294009730 tx=0x7f1294009d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee975e92ed2a565d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f1293fff640 1 -- 172.21.15.69:0/4661872 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129401d040 con 0x55c89e696050 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f1293fff640 1 -- 172.21.15.69:0/4661872 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12940053c0 con 0x55c89e696050 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.885+0000 7f1293fff640 1 -- 172.21.15.69:0/4661872 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f129400fa30 con 0x55c89e696050 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- 172.21.15.69:0/4661872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 msgr2=0x55c89e6b7460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f1294009730 tx=0x7f1294009d00 comp rx=0 tx=0).stop 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- 172.21.15.69:0/4661872 shutdown_connections 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e69b1c0 0x55c89e6b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e696050 0x55c89e6b7460 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 --2- 172.21.15.69:0/4661872 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e695660 0x55c89e68cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- 172.21.15.69:0/4661872 >> 172.21.15.69:0/4661872 conn(0x55c89e68c8c0 msgr2=0x55c89e694c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- 172.21.15.69:0/4661872 shutdown_connections 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- 172.21.15.69:0/4661872 wait complete. 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 -- start start 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.886+0000 7f12b361c740 1 Processor -- start 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e6aa4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e6ab480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1a6700 con 0x55c89e695660 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1a7b70 con 0x55c89e696050 2024-05-09T09:24:33.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f12ac1a6dd0 con 0x55c89e69b1c0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 10 client.0 _ll_register_callbacks cb 0x7f129c004160 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34446/0 (socket says 172.21.15.69:34446) 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 -- 172.21.15.69:0/1585999968 learned_addr learned my addr 172.21.15.69:0/1585999968 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e6aa4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aad9d640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e6ab480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 -- 172.21.15.69:0/1585999968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 msgr2=0x55c89e6ab480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e6ab480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 -- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 msgr2=0x55c89e6aa4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e6aa4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 -- 172.21.15.69:0/1585999968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c89e6aba70 con 0x55c89e695660 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12aa59c640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f12a40ccbb0 tx=0x7f12a40d2ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8d209b9c7b106d0b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f1291ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:24:33.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f12a4020040 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f12a4099ca0 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f12940053c0 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f12a409fca0 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 10 client.16500 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.887+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55c89e715cd0 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f12940053c0 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f12a400f040 con 0x55c89e695660 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 10 client.16500 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 10 client.16500 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 20 client.16500 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e716510 0x55c89e718930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 4 client.16500 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c89e5fcdc0 con 0x55c89e716510 2024-05-09T09:24:33.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e716510 0x55c89e718930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f12a9d9b640 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e716510 0x55c89e718930 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd3649c1f972675a server_cookie=d85228a0128d0ca1 in_seq=0 out_seq=0 2024-05-09T09:24:33.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.888+0000 7f1291ffb640 10 client.16500 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:33.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.995+0000 7f1291ffb640 1 -- 172.21.15.69:0/1585999968 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c89e5fcdc0 con 0x55c89e716510 2024-05-09T09:24:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.995+0000 7f1291ffb640 10 client.16500 handle_command_reply: tid=0 2024-05-09T09:24:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 client.16500 shutdown 2024-05-09T09:24:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 msgr2=0x55c89e6a9f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f12a40ccbb0 tx=0x7f12a40d2ee0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 shutdown_connections 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c89e716510 0x55c89e718930 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c89e69b1c0 0x55c89e6ab480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c89e696050 0x55c89e6aa4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 --2- 172.21.15.69:0/1585999968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c89e695660 0x55c89e6a9f80 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f1291ffb640 0 client.16500 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 >> 172.21.15.69:0/1585999968 conn(0x55c89e68c8c0 msgr2=0x55c89e691630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 shutdown_connections 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 1 -- 172.21.15.69:0/1585999968 wait complete. 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b361c740 20 client.16500 trim_cache size 0 max 16384 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:24:33.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:24:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:24:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:24:33.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 msgr2=0x7f12980a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.996+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 0x7f12980a50f0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7f129c031440 tx=0x7f129c009260 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 msgr2=0x7f12ac197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7f12a400eaa0 tx=0x7f12a400ef60 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 shutdown_connections 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f12980a2c30 0x7f12980a50f0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f12ac103450 0x7f12ac19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f12ac102a90 0x7f12ac197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 --2- 172.21.15.69:0/2687843781 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f12ac1018a0 0x7f12ac197a60 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 >> 172.21.15.69:0/2687843781 conn(0x7f12ac0fcdf0 msgr2=0x7f12ac105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 shutdown_connections 2024-05-09T09:24:33.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:33.997+0000 7f12b0916640 1 -- 172.21.15.69:0/2687843781 wait complete. 2024-05-09T09:24:34.488 INFO:tasks.workunit.client.0.smithi069.stdout: 1 185177 4.03 MB/sec execute 152 sec latency 67.472 ms 2024-05-09T09:24:35.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 187947 4.11 MB/sec execute 153 sec latency 12.207 ms 2024-05-09T09:24:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:35 smithi103 ceph-mon[34421]: pgmap v539: 129 pgs: 129 active+clean; 58 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 304 op/s 2024-05-09T09:24:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:35 smithi069 ceph-mon[30123]: pgmap v539: 129 pgs: 129 active+clean; 58 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 304 op/s 2024-05-09T09:24:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:35 smithi163 ceph-mon[34354]: pgmap v539: 129 pgs: 129 active+clean; 58 MiB data, 780 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 304 op/s 2024-05-09T09:24:36.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 189227 4.11 MB/sec execute 154 sec latency 14.800 ms 2024-05-09T09:24:37.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 191681 4.18 MB/sec execute 155 sec latency 85.135 ms 2024-05-09T09:24:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:37 smithi103 ceph-mon[34421]: pgmap v540: 129 pgs: 129 active+clean; 61 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 306 op/s 2024-05-09T09:24:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:37 smithi069 ceph-mon[30123]: pgmap v540: 129 pgs: 129 active+clean; 61 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 306 op/s 2024-05-09T09:24:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:37 smithi163 ceph-mon[34354]: pgmap v540: 129 pgs: 129 active+clean; 61 MiB data, 832 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.6 MiB/s wr, 306 op/s 2024-05-09T09:24:38.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 193035 4.20 MB/sec execute 156 sec latency 15.076 ms 2024-05-09T09:24:39.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 195469 4.26 MB/sec execute 157 sec latency 21.895 ms 2024-05-09T09:24:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:39 smithi103 ceph-mon[34421]: pgmap v541: 129 pgs: 129 active+clean; 57 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 319 op/s 2024-05-09T09:24:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:39 smithi069 ceph-mon[30123]: pgmap v541: 129 pgs: 129 active+clean; 57 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 319 op/s 2024-05-09T09:24:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:39 smithi163 ceph-mon[34354]: pgmap v541: 129 pgs: 129 active+clean; 57 MiB data, 839 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 1.6 MiB/s wr, 319 op/s 2024-05-09T09:24:40.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 197442 4.28 MB/sec execute 158 sec latency 22.280 ms 2024-05-09T09:24:41.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 199208 4.33 MB/sec execute 159 sec latency 11.173 ms 2024-05-09T09:24:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:41 smithi103 ceph-mon[34421]: pgmap v542: 129 pgs: 129 active+clean; 58 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-05-09T09:24:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:41 smithi069 ceph-mon[30123]: pgmap v542: 129 pgs: 129 active+clean; 58 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-05-09T09:24:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:41 smithi163 ceph-mon[34354]: pgmap v542: 129 pgs: 129 active+clean; 58 MiB data, 842 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 280 op/s 2024-05-09T09:24:42.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 201412 4.37 MB/sec execute 160 sec latency 27.407 ms 2024-05-09T09:24:43.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 202977 4.40 MB/sec execute 161 sec latency 14.663 ms 2024-05-09T09:24:43.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:43 smithi163 ceph-mon[34354]: pgmap v543: 129 pgs: 129 active+clean; 61 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 328 op/s 2024-05-09T09:24:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:43 smithi103 ceph-mon[34421]: pgmap v543: 129 pgs: 129 active+clean; 61 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 328 op/s 2024-05-09T09:24:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:43 smithi069 ceph-mon[30123]: pgmap v543: 129 pgs: 129 active+clean; 61 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 1.7 MiB/s wr, 328 op/s 2024-05-09T09:24:44.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 205876 4.48 MB/sec execute 162 sec latency 13.762 ms 2024-05-09T09:24:45.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 207185 4.49 MB/sec execute 163 sec latency 14.120 ms 2024-05-09T09:24:45.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:45 smithi103 ceph-mon[34421]: pgmap v544: 129 pgs: 129 active+clean; 64 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-05-09T09:24:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:45 smithi069 ceph-mon[30123]: pgmap v544: 129 pgs: 129 active+clean; 64 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-05-09T09:24:46.815 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:45 smithi163 ceph-mon[34354]: pgmap v544: 129 pgs: 129 active+clean; 64 MiB data, 884 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-05-09T09:24:46.816 INFO:tasks.workunit.client.0.smithi069.stdout: 1 209615 4.55 MB/sec execute 164 sec latency 14.333 ms 2024-05-09T09:24:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.089+0000 7f72e561c640 1 -- 172.21.15.69:0/1529604398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e01018e0 msgr2=0x7f72e0101d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.089+0000 7f72e561c640 1 --2- 172.21.15.69:0/1529604398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e01018e0 0x7f72e0101d00 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f72d0009900 tx=0x7f72d002a410 comp rx=0 tx=0).stop 2024-05-09T09:24:47.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.089+0000 7f72e561c640 1 -- 172.21.15.69:0/1529604398 shutdown_connections 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 --2- 172.21.15.69:0/1529604398 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e0103490 0x7f72e0109d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 --2- 172.21.15.69:0/1529604398 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0102ad0 0x7f72e0102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 --2- 172.21.15.69:0/1529604398 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e01018e0 0x7f72e0101d00 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 -- 172.21.15.69:0/1529604398 >> 172.21.15.69:0/1529604398 conn(0x7f72e00fce50 msgr2=0x7f72e00ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 -- 172.21.15.69:0/1529604398 shutdown_connections 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 -- 172.21.15.69:0/1529604398 wait complete. 2024-05-09T09:24:47.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.090+0000 7f72e561c640 1 Processor -- start 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 -- start start 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 0x7f72e00755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 0x7f72e0073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d5b0 con 0x7f72e0103490 2024-05-09T09:24:47.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d430 con 0x7f72e01018e0 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72e561c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72e010d730 con 0x7f72e0102ad0 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 0x7f72e0073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34632/0 (socket says 172.21.15.69:34632) 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72df7fe640 1 -- 172.21.15.69:0/485598706 learned_addr learned my addr 172.21.15.69:0/485598706 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 0x7f72e00755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 -- 172.21.15.69:0/485598706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 msgr2=0x7f72e00755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 0x7f72e00755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 -- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 msgr2=0x7f72e0074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72deffd640 1 -- 172.21.15.69:0/485598706 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72e00749f0 con 0x7f72e0102ad0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72df7fe640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.091+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 0x7f72e00755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72deffd640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 0x7f72e0073c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f72c800e9f0 tx=0x7f72c800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72dcff9640 1 -- 172.21.15.69:0/485598706 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c800cd80 con 0x7f72e0102ad0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72e01a1c50 con 0x7f72e0102ad0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72dcff9640 1 -- 172.21.15.69:0/485598706 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c80035f0 con 0x7f72e0102ad0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72dcff9640 1 -- 172.21.15.69:0/485598706 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c8010670 con 0x7f72e0102ad0 2024-05-09T09:24:47.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.092+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72e01a21a0 con 0x7f72e0102ad0 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.094+0000 7f72dcff9640 1 -- 172.21.15.69:0/485598706 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f72c80037a0 con 0x7f72e0102ad0 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.095+0000 7f72e8409740 1 Processor -- start 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.095+0000 7f72dcff9640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 0x7f72cc0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.095+0000 7f72e8409740 1 -- start start 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.095+0000 7f72df7fe640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 0x7f72cc0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.095+0000 7f72dcff9640 1 -- 172.21.15.69:0/485598706 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f72c80cf670 con 0x7f72e0102ad0 2024-05-09T09:24:47.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 0x564874c14670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c19db0 0x564874c1c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564874b78340 con 0x564874c14c40 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564874b20290 con 0x564874c19db0 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72e8409740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564874b74300 con 0x564874c14250 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72d6ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 0x564874c14670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34646/0 (socket says 172.21.15.69:34646) 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 -- 172.21.15.69:0/3002180145 learned_addr learned my addr 172.21.15.69:0/3002180145 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:47.098 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72d6ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 0x564874c14670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53228/0 (socket says 172.21.15.69:53228) 2024-05-09T09:24:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72df7fe640 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c19db0 0x564874c1c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72df7fe640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 0x7f72cc0a5170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f72d80049d0 tx=0x7f72d8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 -- 172.21.15.69:0/3002180145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 msgr2=0x564874c14670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 0x564874c14670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 -- 172.21.15.69:0/3002180145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c19db0 msgr2=0x564874c1c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c19db0 0x564874c1c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 -- 172.21.15.69:0/3002180145 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564874b448e0 con 0x564874c14c40 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72deffd640 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f72c80a4290 tx=0x7f72c809b5b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d545e4b46a3fe4ee server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72d57fa640 1 -- 172.21.15.69:0/3002180145 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80b3020 con 0x564874c14c40 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72d57fa640 1 -- 172.21.15.69:0/3002180145 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c80a47c0 con 0x564874c14c40 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.096+0000 7f72d57fa640 1 -- 172.21.15.69:0/3002180145 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72c80981c0 con 0x564874c14c40 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 -- 172.21.15.69:0/3002180145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 msgr2=0x564874c19740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f72c80a4290 tx=0x7f72c809b5b0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 -- 172.21.15.69:0/3002180145 shutdown_connections 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c19db0 0x564874c1c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c14c40 0x564874c19740 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 --2- 172.21.15.69:0/3002180145 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c14250 0x564874c14670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 -- 172.21.15.69:0/3002180145 >> 172.21.15.69:0/3002180145 conn(0x564874c0f8c0 msgr2=0x564874c0fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 -- 172.21.15.69:0/3002180145 shutdown_connections 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.097+0000 7f72e8409740 1 -- 172.21.15.69:0/3002180145 wait complete. 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 -- start start 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 Processor -- start 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c1e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c1ec10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564874c1cd60 con 0x564874c23c40 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564874b33790 con 0x564874c14250 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564874c1cee0 con 0x564874c19db0 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72e8409740 10 client.0 _ll_register_callbacks cb 0x564874b6ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34654/0 (socket says 172.21.15.69:34654) 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72d6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c1e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 -- 172.21.15.69:0/2660830883 learned_addr learned my addr 172.21.15.69:0/2660830883 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72deffd640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c1ec10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 -- 172.21.15.69:0/2660830883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 msgr2=0x564874c1ec10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c1ec10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 -- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 msgr2=0x564874c1e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c1e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 -- 172.21.15.69:0/2660830883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56487488c210 con 0x564874c23c40 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72deffd640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c1ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f72d801c440 tx=0x7f72d8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=556de74def2324c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.098+0000 7f72c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d802a060 con 0x564874c23c40 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f72d8009040 con 0x564874c23c40 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72c80a47c0 con 0x564874c23c40 2024-05-09T09:24:47.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f72d8031050 con 0x564874c23c40 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 10 client.16512 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564874958620 con 0x564874c23c40 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72c80a47c0 con 0x564874c23c40 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f72d8025930 con 0x564874c23c40 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 10 client.16512 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 10 client.16512 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 20 client.16512 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c2ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 4 client.16512 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564874b85a70 con 0x564874c2a5f0 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.099+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c2ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72c67fc640 10 client.16512 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:47.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c2ca20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=db9fe561ce9ce619 server_cookie=e25a57e6291bd4cd in_seq=0 out_seq=0 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72c67fc640 1 -- 172.21.15.69:0/2660830883 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564874b85a70 con 0x564874c2a5f0 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72c67fc640 10 client.16512 handle_command_reply: tid=0 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 client.16512 shutdown 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 msgr2=0x564874c26030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f72d801c440 tx=0x7f72d8004ab0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 shutdown_connections 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c26030 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c2ca20 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72c67fc640 0 client.16512 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c1ec10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 --2- 172.21.15.69:0/2660830883 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c1e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.100+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 >> 172.21.15.69:0/2660830883 conn(0x564874c0f8c0 msgr2=0x564874c0ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.101+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 shutdown_connections 2024-05-09T09:24:47.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.101+0000 7f72e8409740 1 -- 172.21.15.69:0/2660830883 wait complete. 2024-05-09T09:24:47.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.101+0000 7f72e8409740 20 client.16512 trim_cache size 0 max 16384 2024-05-09T09:24:47.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.116+0000 7f72e8409740 1 Processor -- start 2024-05-09T09:24:47.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.116+0000 7f72e8409740 1 -- start start 2024-05-09T09:24:47.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c160a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c165e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01aef40 con 0x564874c23c40 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ac810 con 0x564874c14250 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72e8409740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01aa660 con 0x564874c19db0 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34670/0 (socket says 172.21.15.69:34670) 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 -- 172.21.15.69:0/2838178670 learned_addr learned my addr 172.21.15.69:0/2838178670 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72deffd640 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c165e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c160a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 -- 172.21.15.69:0/2838178670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 msgr2=0x564874c165e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c165e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 -- 172.21.15.69:0/2838178670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 msgr2=0x564874c160a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c160a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 -- 172.21.15.69:0/2838178670 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564874c1d010 con 0x564874c23c40 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.117+0000 7f72df7fe640 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f72d800f620 tx=0x7f72d8009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=36d423906785f4cf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.118+0000 7f72d4ff9640 1 -- 172.21.15.69:0/2838178670 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d802a040 con 0x564874c23c40 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.118+0000 7f72d4ff9640 1 -- 172.21.15.69:0/2838178670 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8012df0 con 0x564874c23c40 2024-05-09T09:24:47.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.118+0000 7f72d4ff9640 1 -- 172.21.15.69:0/2838178670 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d801ba80 con 0x564874c23c40 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.118+0000 7f72e8409740 1 -- 172.21.15.69:0/2838178670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 msgr2=0x564874c0e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.118+0000 7f72e8409740 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f72d800f620 tx=0x7f72d8009f40 comp rx=0 tx=0).stop 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- 172.21.15.69:0/2838178670 shutdown_connections 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c0e170 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c165e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- 172.21.15.69:0/2838178670 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c160a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- 172.21.15.69:0/2838178670 >> 172.21.15.69:0/2838178670 conn(0x564874c0f8c0 msgr2=0x564874c1e6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- 172.21.15.69:0/2838178670 shutdown_connections 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- 172.21.15.69:0/2838178670 wait complete. 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- start start 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 Processor -- start 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c3bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01aacd0 con 0x564874c23c40 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ac140 con 0x564874c14250 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72e01ab3a0 con 0x564874c19db0 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.119+0000 7f72e8409740 10 client.0 _ll_register_callbacks cb 0x7f72d800fdf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:57028/0 (socket says 172.21.15.69:57028) 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72deffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c3bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 -- 172.21.15.69:0/3895420180 learned_addr learned my addr 172.21.15.69:0/3895420180 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72df7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34686/0 (socket says 172.21.15.69:34686) 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 -- 172.21.15.69:0/3895420180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 msgr2=0x564874c3bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c3bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 -- 172.21.15.69:0/3895420180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 msgr2=0x564874c3ccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 -- 172.21.15.69:0/3895420180 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564874c3d2a0 con 0x564874c14250 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72df7fe640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72d6ffd640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006010 tx=0x7f72d0005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5eca3b64e30d01e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72c67fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:24:47.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72d0035040 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f72d0046510 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.120+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f72d8012df0 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f72d0056050 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 10 client.25835 fetch_fsmap learned FSMap version 35 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x564874c908c0 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72d8012df0 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f72d00317d0 con 0x564874c14250 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 10 client.25835 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 10 client.25835 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 20 client.25835 populate_metadata read hostname 'smithi069' 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c932a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 4 client.25835 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564874b7fdc0 con 0x564874c2a5f0 2024-05-09T09:24:47.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.121+0000 7f72deffd640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c932a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:24:47.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.122+0000 7f72deffd640 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c932a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bd7175be520f5414 server_cookie=a607d15efe68d9f5 in_seq=0 out_seq=0 2024-05-09T09:24:47.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.122+0000 7f72c67fc640 10 client.25835 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:47.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.234+0000 7f72c67fc640 1 -- 172.21.15.69:0/3895420180 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564874b7fdc0 con 0x564874c2a5f0 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.234+0000 7f72c67fc640 10 client.25835 handle_command_reply: tid=0 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.234+0000 7f72e8409740 1 client.25835 shutdown 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 msgr2=0x564874c3b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f72d0006010 tx=0x7f72d0005280 comp rx=0 tx=0).stop 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 shutdown_connections 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564874c23c40 0x564874c3ccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564874c2a5f0 0x564874c932a0 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564874c19db0 0x564874c3bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72c67fc640 0 client.25835 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:24:47.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 --2- 172.21.15.69:0/3895420180 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564874c14250 0x564874c3b760 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 >> 172.21.15.69:0/3895420180 conn(0x564874c0f8c0 msgr2=0x564874c1e5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 shutdown_connections 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 1 -- 172.21.15.69:0/3895420180 wait complete. 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e8409740 20 client.25835 trim_cache size 0 max 16384 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:24:47.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 msgr2=0x7f72cc0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 0x7f72cc0a5170 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f72d80049d0 tx=0x7f72d8009500 comp rx=0 tx=0).stop 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 msgr2=0x7f72e0073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 0x7f72e0073c20 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f72c800e9f0 tx=0x7f72c800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 shutdown_connections 2024-05-09T09:24:47.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72cc0a2cb0 0x7f72cc0a5170 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72e0103490 0x7f72e0074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72e0102ad0 0x7f72e0073c20 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 --2- 172.21.15.69:0/485598706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72e01018e0 0x7f72e00755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.235+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 >> 172.21.15.69:0/485598706 conn(0x7f72e00fce50 msgr2=0x7f72e0105d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.236+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 shutdown_connections 2024-05-09T09:24:47.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:24:47.236+0000 7f72e561c640 1 -- 172.21.15.69:0/485598706 wait complete. 2024-05-09T09:24:47.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 210836 4.55 MB/sec execute 165 sec latency 73.637 ms 2024-05-09T09:24:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:47 smithi103 ceph-mon[34421]: pgmap v545: 129 pgs: 129 active+clean; 64 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 320 op/s 2024-05-09T09:24:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:47 smithi069 ceph-mon[30123]: pgmap v545: 129 pgs: 129 active+clean; 64 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 320 op/s 2024-05-09T09:24:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:47 smithi163 ceph-mon[34354]: pgmap v545: 129 pgs: 129 active+clean; 64 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.8 MiB/s wr, 320 op/s 2024-05-09T09:24:48.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 213195 4.61 MB/sec execute 166 sec latency 29.859 ms 2024-05-09T09:24:49.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 215275 4.62 MB/sec execute 167 sec latency 14.220 ms 2024-05-09T09:24:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:49 smithi103 ceph-mon[34421]: pgmap v546: 129 pgs: 129 active+clean; 66 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 305 op/s 2024-05-09T09:24:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:49 smithi069 ceph-mon[30123]: pgmap v546: 129 pgs: 129 active+clean; 66 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 305 op/s 2024-05-09T09:24:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:49 smithi163 ceph-mon[34354]: pgmap v546: 129 pgs: 129 active+clean; 66 MiB data, 918 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 MiB/s rd, 1.7 MiB/s wr, 305 op/s 2024-05-09T09:24:50.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 217016 4.67 MB/sec execute 168 sec latency 11.327 ms 2024-05-09T09:24:51.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 219443 4.71 MB/sec execute 169 sec latency 13.917 ms 2024-05-09T09:24:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:51 smithi103 ceph-mon[34421]: pgmap v547: 129 pgs: 129 active+clean; 67 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 312 op/s 2024-05-09T09:24:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:51 smithi069 ceph-mon[30123]: pgmap v547: 129 pgs: 129 active+clean; 67 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 312 op/s 2024-05-09T09:24:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:51 smithi163 ceph-mon[34354]: pgmap v547: 129 pgs: 129 active+clean; 67 MiB data, 937 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 1.6 MiB/s wr, 312 op/s 2024-05-09T09:24:52.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 220775 4.73 MB/sec execute 170 sec latency 70.383 ms 2024-05-09T09:24:53.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 223511 4.80 MB/sec execute 171 sec latency 23.773 ms 2024-05-09T09:24:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:53 smithi103 ceph-mon[34421]: pgmap v548: 129 pgs: 129 active+clean; 64 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 329 op/s 2024-05-09T09:24:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:53 smithi069 ceph-mon[30123]: pgmap v548: 129 pgs: 129 active+clean; 64 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 329 op/s 2024-05-09T09:24:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:53 smithi163 ceph-mon[34354]: pgmap v548: 129 pgs: 129 active+clean; 64 MiB data, 939 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 MiB/s rd, 1.8 MiB/s wr, 329 op/s 2024-05-09T09:24:54.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 224898 4.81 MB/sec execute 172 sec latency 14.543 ms 2024-05-09T09:24:55.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 227426 4.86 MB/sec execute 173 sec latency 16.318 ms 2024-05-09T09:24:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:55 smithi103 ceph-mon[34421]: pgmap v549: 129 pgs: 129 active+clean; 67 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 355 op/s 2024-05-09T09:24:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:55 smithi069 ceph-mon[30123]: pgmap v549: 129 pgs: 129 active+clean; 67 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 355 op/s 2024-05-09T09:24:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:55 smithi163 ceph-mon[34354]: pgmap v549: 129 pgs: 129 active+clean; 67 MiB data, 965 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 2.0 MiB/s wr, 355 op/s 2024-05-09T09:24:56.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 229256 4.87 MB/sec execute 174 sec latency 15.254 ms 2024-05-09T09:24:57.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 231078 4.92 MB/sec execute 175 sec latency 10.701 ms 2024-05-09T09:24:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:57 smithi103 ceph-mon[34421]: pgmap v550: 129 pgs: 129 active+clean; 68 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 322 op/s 2024-05-09T09:24:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:57 smithi069 ceph-mon[30123]: pgmap v550: 129 pgs: 129 active+clean; 68 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 322 op/s 2024-05-09T09:24:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:57 smithi163 ceph-mon[34354]: pgmap v550: 129 pgs: 129 active+clean; 68 MiB data, 979 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 2.1 MiB/s wr, 322 op/s 2024-05-09T09:24:58.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 233022 4.93 MB/sec execute 176 sec latency 35.310 ms 2024-05-09T09:24:59.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 234753 4.97 MB/sec execute 177 sec latency 14.557 ms 2024-05-09T09:24:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:24:59 smithi103 ceph-mon[34421]: pgmap v551: 129 pgs: 129 active+clean; 71 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 359 op/s 2024-05-09T09:24:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:24:59 smithi069 ceph-mon[30123]: pgmap v551: 129 pgs: 129 active+clean; 71 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 359 op/s 2024-05-09T09:24:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:24:59 smithi163 ceph-mon[34354]: pgmap v551: 129 pgs: 129 active+clean; 71 MiB data, 980 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 2.5 MiB/s wr, 359 op/s 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2805917500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4102340 msgr2=0x7f74c4108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2805917500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4102340 0x7f74c4108bd0 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f74bc009ca0 tx=0x7f74bc02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2805917500 shutdown_connections 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2805917500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4102340 0x7f74c4108bd0 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2805917500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4101980 0x7f74c4101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2805917500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4100790 0x7f74c4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2805917500 >> 172.21.15.69:0/2805917500 conn(0x7f74c40fbd00 msgr2=0x7f74c40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2805917500 shutdown_connections 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2805917500 wait complete. 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.330+0000 7f74c9bb7640 1 Processor -- start 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 -- start start 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4100790 0x7f74c4069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4102340 0x7f74c419ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410c410 con 0x7f74c4101980 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410c290 con 0x7f74c4102340 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c9bb7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f74c410c590 con 0x7f74c4100790 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50308/0 (socket says 172.21.15.69:50308) 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c37fe640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4100790 0x7f74c4069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 -- 172.21.15.69:0/2424447852 learned_addr learned my addr 172.21.15.69:0/2424447852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c3fff640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4102340 0x7f74c419ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 -- 172.21.15.69:0/2424447852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4100790 msgr2=0x7f74c4069a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4100790 0x7f74c4069a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 -- 172.21.15.69:0/2424447852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4102340 msgr2=0x7f74c419ee30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4102340 0x7f74c419ee30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 -- 172.21.15.69:0/2424447852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f74c419f740 con 0x7f74c4101980 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.331+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9f0 tx=0x7f74b800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.332+0000 7f74c0ff9640 1 -- 172.21.15.69:0/2424447852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b800cd80 con 0x7f74c4101980 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.332+0000 7f74c0ff9640 1 -- 172.21.15.69:0/2424447852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b8005500 con 0x7f74c4101980 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.332+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f74c419f9d0 con 0x7f74c4101980 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.332+0000 7f74c0ff9640 1 -- 172.21.15.69:0/2424447852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b8010690 con 0x7f74c4101980 2024-05-09T09:25:00.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.332+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f74c419fed0 con 0x7f74c4101980 2024-05-09T09:25:00.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.334+0000 7f74c0ff9640 1 -- 172.21.15.69:0/2424447852 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f74b80046c0 con 0x7f74c4101980 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 Processor -- start 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 -- start start 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x56036052e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x560360536240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560360492340 con 0x560360533db0 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56036043a290 con 0x56036052e250 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56036048e300 con 0x56036052ec40 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x56036052e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x560360536240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:58608/0 (socket says 172.21.15.69:58608) 2024-05-09T09:25:00.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 -- 172.21.15.69:0/4094718956 learned_addr learned my addr 172.21.15.69:0/4094718956 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x56036052e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54254/0 (socket says 172.21.15.69:54254) 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c0ff9640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 0x7f74ac0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 -- 172.21.15.69:0/4094718956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 msgr2=0x56036052e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x56036052e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 -- 172.21.15.69:0/4094718956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 msgr2=0x560360536240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x560360536240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c37fe640 1 -- 172.21.15.69:0/4094718956 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56036045e8e0 con 0x56036052ec40 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c3fff640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 0x7f74ac0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.335+0000 7f74c0ff9640 1 -- 172.21.15.69:0/2424447852 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f74b80cf7e0 con 0x7f74c4101980 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74c37fe640 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009ca0 tx=0x7f74b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4eb047a2739e33c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74a9ffb640 1 -- 172.21.15.69:0/4094718956 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4004220 con 0x56036052ec40 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74a9ffb640 1 -- 172.21.15.69:0/4094718956 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b40043c0 con 0x56036052ec40 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74a9ffb640 1 -- 172.21.15.69:0/4094718956 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b40198d0 con 0x56036052ec40 2024-05-09T09:25:00.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 -- 172.21.15.69:0/4094718956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 msgr2=0x560360533740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74c3fff640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 0x7f74ac0a50f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005e30 tx=0x7f74bc0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f74b4009ca0 tx=0x7f74b4011810 comp rx=0 tx=0).stop 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 -- 172.21.15.69:0/4094718956 shutdown_connections 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x560360536240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052ec40 0x560360533740 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/4094718956 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x56036052e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 -- 172.21.15.69:0/4094718956 >> 172.21.15.69:0/4094718956 conn(0x5603605298c0 msgr2=0x560360529cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 -- 172.21.15.69:0/4094718956 shutdown_connections 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.336+0000 7f74cc9a4740 1 -- 172.21.15.69:0/4094718956 wait complete. 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 -- start start 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 Processor -- start 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x560360536d60 con 0x56036052ec40 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56036044d790 con 0x56036052e250 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x560360536ee0 con 0x560360533db0 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74cc9a4740 10 client.0 _ll_register_callbacks cb 0x560360488a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50320/0 (socket says 172.21.15.69:50320) 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54262/0 (socket says 172.21.15.69:54262) 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.337+0000 7f74c2ffd640 1 -- 172.21.15.69:0/1578038718 learned_addr learned my addr 172.21.15.69:0/1578038718 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 -- 172.21.15.69:0/1578038718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 msgr2=0x56036053f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 -- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 msgr2=0x560360538400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 -- 172.21.15.69:0/1578038718 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5603601a6210 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c37fe640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f74b809bc50 tx=0x7f74b80056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61fe4887729fb8c8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b809b800 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74b40043c0 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f74b80a44a0 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f74b809b620 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 10 client.16530 fetch_fsmap learned FSMap version 35 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x560360272620 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b40043c0 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f74b80a4b80 con 0x56036052ec40 2024-05-09T09:25:00.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 10 client.16530 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 10 client.16530 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 20 client.16530 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560360543f80 0x5603605463b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 4 client.16530 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.338+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56036049fa70 con 0x560360543f80 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.339+0000 7f74c37fe640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560360543f80 0x5603605463b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.339+0000 7f74a2ffd640 10 client.16530 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:00.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.339+0000 7f74c37fe640 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560360543f80 0x5603605463b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d292d59001bdd41c server_cookie=88881d531e5c4a95 in_seq=0 out_seq=0 2024-05-09T09:25:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74a2ffd640 1 -- 172.21.15.69:0/1578038718 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56036049fa70 con 0x560360543f80 2024-05-09T09:25:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74a2ffd640 10 client.16530 handle_command_reply: tid=0 2024-05-09T09:25:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 client.16530 shutdown 2024-05-09T09:25:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 msgr2=0x560360538940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7f74b809bc50 tx=0x7f74b80056b0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 shutdown_connections 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x560360543f80 0x5603605463b0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74a2ffd640 0 client.16530 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052ec40 0x560360538940 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/1578038718 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052e250 0x560360538400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 >> 172.21.15.69:0/1578038718 conn(0x5603605298c0 msgr2=0x560360528a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 shutdown_connections 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 1 -- 172.21.15.69:0/1578038718 wait complete. 2024-05-09T09:25:00.347 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.341+0000 7f74cc9a4740 20 client.16530 trim_cache size 0 max 16384 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 Processor -- start 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 -- start start 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 0x5603605361a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41acc70 con 0x560360533db0 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41aa540 con 0x56036052ec40 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41a8390 con 0x56036052e250 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 0x5603605361a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50322/0 (socket says 172.21.15.69:50322) 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c3fff640 1 -- 172.21.15.69:0/943639833 learned_addr learned my addr 172.21.15.69:0/943639833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:00.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 -- 172.21.15.69:0/943639833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 msgr2=0x56036053f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 -- 172.21.15.69:0/943639833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 msgr2=0x56036052c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 -- 172.21.15.69:0/943639833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x560360527f00 con 0x56036052e250 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c3fff640 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.357+0000 7f74c37fe640 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 0x5603605361a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x56036052a830 tx=0x7f74b4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=426e99774c818c23 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74a97fa640 1 -- 172.21.15.69:0/943639833 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4030070 con 0x56036052e250 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74a97fa640 1 -- 172.21.15.69:0/943639833 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b4011d10 con 0x56036052e250 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74a97fa640 1 -- 172.21.15.69:0/943639833 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b40052a0 con 0x56036052e250 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 -- 172.21.15.69:0/943639833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 msgr2=0x5603605361a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 0x5603605361a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x56036052a830 tx=0x7f74b4011810 comp rx=0 tx=0).stop 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 -- 172.21.15.69:0/943639833 shutdown_connections 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x560360533db0 0x56036052c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/943639833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56036052e250 0x5603605361a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 -- 172.21.15.69:0/943639833 >> 172.21.15.69:0/943639833 conn(0x5603605298c0 msgr2=0x56036052fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 -- 172.21.15.69:0/943639833 shutdown_connections 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.358+0000 7f74cc9a4740 1 -- 172.21.15.69:0/943639833 wait complete. 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 -- start start 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 Processor -- start 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053d460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41a8a00 con 0x56036052e250 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41a9e70 con 0x56036052ec40 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f74c41a90d0 con 0x560360533db0 2024-05-09T09:25:00.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74cc9a4740 10 client.0 _ll_register_callbacks cb 0x7f74b800cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74c2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053d460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74c37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50334/0 (socket says 172.21.15.69:50334) 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74c37fe640 1 -- 172.21.15.69:0/535978531 learned_addr learned my addr 172.21.15.69:0/535978531 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.359+0000 7f74c3fff640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 -- 172.21.15.69:0/535978531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 msgr2=0x56036053e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 -- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 msgr2=0x56036053d460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053d460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 -- 172.21.15.69:0/535978531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56036053ea60 con 0x56036052e250 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74c37fe640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f74b4000c00 tx=0x7f74b4005970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c445fffd8e3e1198 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:00.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f74b4039070 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7f74b4003a80 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f74b4011d10 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7f74b4047050 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 10 client.16536 fetch_fsmap learned FSMap version 35 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x56036053fd70 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f74b4011d10 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7f74b402c5c0 con 0x56036052e250 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 10 client.16536 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 10 client.16536 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 20 client.16536 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5603605aabf0 0x5603605ad010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 4 client.16536 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.360+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x560360499dc0 con 0x5603605aabf0 2024-05-09T09:25:00.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.361+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5603605aabf0 0x5603605ad010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.361+0000 7f74c2ffd640 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5603605aabf0 0x5603605ad010 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22f3c024d5c8df33 server_cookie=bf167d46be9e5cb9 in_seq=0 out_seq=0 2024-05-09T09:25:00.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.361+0000 7f74a2ffd640 10 client.16536 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74a2ffd640 1 -- 172.21.15.69:0/535978531 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x560360499dc0 con 0x5603605aabf0 2024-05-09T09:25:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74a2ffd640 10 client.16536 handle_command_reply: tid=0 2024-05-09T09:25:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 client.16536 shutdown 2024-05-09T09:25:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 msgr2=0x56036053cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7f74b4000c00 tx=0x7f74b4005970 comp rx=0 tx=0).stop 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 shutdown_connections 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5603605aabf0 0x5603605ad010 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x560360533db0 0x56036053e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56036052ec40 0x56036053d460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74a2ffd640 0 client.16536 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 --2- 172.21.15.69:0/535978531 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56036052e250 0x56036053cf20 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 >> 172.21.15.69:0/535978531 conn(0x5603605298c0 msgr2=0x56036052c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 shutdown_connections 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 1 -- 172.21.15.69:0/535978531 wait complete. 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.461+0000 7f74cc9a4740 20 client.16536 trim_cache size 0 max 16384 2024-05-09T09:25:00.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:25:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:25:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:25:00.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 msgr2=0x7f74ac0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 0x7f74ac0a50f0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f74bc005e30 tx=0x7f74bc0095c0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 msgr2=0x7f74c419c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7f74b800e9f0 tx=0x7f74b800edf0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 shutdown_connections 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f74ac0a2c30 0x7f74ac0a50f0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f74c4102340 0x7f74c419ee30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f74c4101980 0x7f74c419c810 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 --2- 172.21.15.69:0/2424447852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f74c4100790 0x7f74c4069a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 >> 172.21.15.69:0/2424447852 conn(0x7f74c40fbd00 msgr2=0x7f74c4104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 shutdown_connections 2024-05-09T09:25:00.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:00.462+0000 7f74c9bb7640 1 -- 172.21.15.69:0/2424447852 wait complete. 2024-05-09T09:25:00.489 INFO:tasks.workunit.client.0.smithi069.stdout: 1 237079 5.01 MB/sec execute 178 sec latency 45.749 ms 2024-05-09T09:25:01.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 238488 5.02 MB/sec execute 179 sec latency 15.879 ms 2024-05-09T09:25:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:01 smithi103 ceph-mon[34421]: pgmap v552: 129 pgs: 129 active+clean; 71 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:25:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:01 smithi069 ceph-mon[30123]: pgmap v552: 129 pgs: 129 active+clean; 71 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:25:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:01 smithi163 ceph-mon[34354]: pgmap v552: 129 pgs: 129 active+clean; 71 MiB data, 964 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:25:02.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 240902 5.07 MB/sec execute 180 sec latency 43.212 ms 2024-05-09T09:25:03.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 241960 5.07 MB/sec execute 181 sec latency 51.408 ms 2024-05-09T09:25:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:03 smithi103 ceph-mon[34421]: pgmap v553: 129 pgs: 129 active+clean; 72 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2024-05-09T09:25:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:03 smithi069 ceph-mon[30123]: pgmap v553: 129 pgs: 129 active+clean; 72 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2024-05-09T09:25:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:03 smithi163 ceph-mon[34354]: pgmap v553: 129 pgs: 129 active+clean; 72 MiB data, 953 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.8 MiB/s wr, 317 op/s 2024-05-09T09:25:04.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 244601 5.12 MB/sec execute 182 sec latency 71.917 ms 2024-05-09T09:25:04.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:25:04.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:25:04.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:25:05.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 245760 5.12 MB/sec execute 183 sec latency 14.793 ms 2024-05-09T09:25:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:25:05.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:05 smithi103 ceph-mon[34421]: pgmap v554: 129 pgs: 129 active+clean; 70 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-05-09T09:25:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:25:05.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:05 smithi069 ceph-mon[30123]: pgmap v554: 129 pgs: 129 active+clean; 70 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-05-09T09:25:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:25:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:25:05.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:25:05.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:25:05.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:25:05.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:25:05.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:05 smithi163 ceph-mon[34354]: pgmap v554: 129 pgs: 129 active+clean; 70 MiB data, 983 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 358 op/s 2024-05-09T09:25:06.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 248570 5.19 MB/sec execute 184 sec latency 11.427 ms 2024-05-09T09:25:07.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 249848 5.19 MB/sec execute 185 sec latency 19.768 ms 2024-05-09T09:25:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:07 smithi103 ceph-mon[34421]: pgmap v555: 129 pgs: 129 active+clean; 72 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:25:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:07 smithi069 ceph-mon[30123]: pgmap v555: 129 pgs: 129 active+clean; 72 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:25:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:07 smithi163 ceph-mon[34354]: pgmap v555: 129 pgs: 129 active+clean; 72 MiB data, 1017 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:25:08.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 252110 5.24 MB/sec execute 186 sec latency 34.382 ms 2024-05-09T09:25:09.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 253398 5.24 MB/sec execute 187 sec latency 36.181 ms 2024-05-09T09:25:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:09 smithi103 ceph-mon[34421]: pgmap v556: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:25:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:09 smithi069 ceph-mon[30123]: pgmap v556: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:25:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:09 smithi163 ceph-mon[34354]: pgmap v556: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:25:10.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 255808 5.29 MB/sec execute 188 sec latency 19.177 ms 2024-05-09T09:25:11.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 257864 5.30 MB/sec execute 189 sec latency 12.610 ms 2024-05-09T09:25:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:11 smithi103 ceph-mon[34421]: pgmap v557: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.5 MiB/s wr, 290 op/s 2024-05-09T09:25:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:11 smithi069 ceph-mon[30123]: pgmap v557: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.5 MiB/s wr, 290 op/s 2024-05-09T09:25:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:11 smithi163 ceph-mon[34354]: pgmap v557: 129 pgs: 129 active+clean; 73 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 930 KiB/s rd, 2.5 MiB/s wr, 290 op/s 2024-05-09T09:25:12.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 259562 5.33 MB/sec execute 190 sec latency 11.773 ms 2024-05-09T09:25:13.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 261576 5.35 MB/sec execute 191 sec latency 28.810 ms 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 -- 172.21.15.69:0/3332029156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584100790 msgr2=0x7fe584100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 --2- 172.21.15.69:0/3332029156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584100790 0x7fe584100bb0 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fe580009900 tx=0x7fe58002a410 comp rx=0 tx=0).stop 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 -- 172.21.15.69:0/3332029156 shutdown_connections 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 --2- 172.21.15.69:0/3332029156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584102340 0x7fe584108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 --2- 172.21.15.69:0/3332029156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 --2- 172.21.15.69:0/3332029156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584100790 0x7fe584100bb0 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.560+0000 7fe58b84c640 1 -- 172.21.15.69:0/3332029156 >> 172.21.15.69:0/3332029156 conn(0x7fe5840fbd00 msgr2=0x7fe5840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- 172.21.15.69:0/3332029156 shutdown_connections 2024-05-09T09:25:13.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- 172.21.15.69:0/3332029156 wait complete. 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 Processor -- start 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- start start 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584100790 0x7fe5841046d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410c3e0 con 0x7fe584102340 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410c260 con 0x7fe584100790 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe58b84c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe58410c560 con 0x7fe584101980 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe589de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe589de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51386/0 (socket says 172.21.15.69:51386) 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe589de9640 1 -- 172.21.15.69:0/2202706168 learned_addr learned my addr 172.21.15.69:0/2202706168 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.561+0000 7fe5895e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584100790 0x7fe5841046d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe588de7640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 -- 172.21.15.69:0/2202706168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 msgr2=0x7fe584102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 -- 172.21.15.69:0/2202706168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584100790 msgr2=0x7fe5841046d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584100790 0x7fe5841046d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 -- 172.21.15.69:0/2202706168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe584103b60 con 0x7fe584102340 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe588de7640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe589de9640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00eb10 tx=0x7fe57c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe57a7fc640 1 -- 172.21.15.69:0/2202706168 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c004b20 con 0x7fe584102340 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe57a7fc640 1 -- 172.21.15.69:0/2202706168 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c004cc0 con 0x7fe584102340 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe57a7fc640 1 -- 172.21.15.69:0/2202706168 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c010760 con 0x7fe584102340 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe5841a1c80 con 0x7fe584102340 2024-05-09T09:25:13.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.562+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe5841a21a0 con 0x7fe584102340 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.564+0000 7fe57a7fc640 1 -- 172.21.15.69:0/2202706168 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe57c005020 con 0x7fe584102340 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 Processor -- start 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 -- start start 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 0x55ee7d2a8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe57a7fc640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 0x7fe5700a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2ad740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d20c340 con 0x55ee7d2a8250 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d1b4290 con 0x55ee7d2a8c40 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe58e631740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d208300 con 0x55ee7d2addb0 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe5895e8640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 0x7fe5700a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe588de7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 0x55ee7d2a8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe57a7fc640 1 -- 172.21.15.69:0/2202706168 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fe57c0cf980 con 0x7fe584102340 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe589de9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe588de7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 0x55ee7d2a8670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51394/0 (socket says 172.21.15.69:51394) 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.565+0000 7fe588de7640 1 -- 172.21.15.69:0/1100962335 learned_addr learned my addr 172.21.15.69:0/1100962335 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe588de7640 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2ad740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 -- 172.21.15.69:0/1100962335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 msgr2=0x55ee7d2ad740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2ad740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 -- 172.21.15.69:0/1100962335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 msgr2=0x55ee7d2a8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 0x55ee7d2a8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 -- 172.21.15.69:0/1100962335 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee7d1d88e0 con 0x55ee7d2addb0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe5895e8640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 0x7fe5700a5170 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe580005ab0 tx=0x7fe580000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe589de9640 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b0240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cd150 tx=0x7fe57c004f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=519d16f6fa2a62b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe56f7fe640 1 -- 172.21.15.69:0/1100962335 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c09b7c0 con 0x55ee7d2addb0 2024-05-09T09:25:13.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe56f7fe640 1 -- 172.21.15.69:0/1100962335 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a5990 con 0x55ee7d2addb0 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.566+0000 7fe56f7fe640 1 -- 172.21.15.69:0/1100962335 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c0a94e0 con 0x55ee7d2addb0 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- 172.21.15.69:0/1100962335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 msgr2=0x55ee7d2b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b0240 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cd150 tx=0x7fe57c004f20 comp rx=0 tx=0).stop 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- 172.21.15.69:0/1100962335 shutdown_connections 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b0240 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2ad740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 --2- 172.21.15.69:0/1100962335 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8250 0x55ee7d2a8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- 172.21.15.69:0/1100962335 >> 172.21.15.69:0/1100962335 conn(0x55ee7d2a38c0 msgr2=0x55ee7d2a3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- 172.21.15.69:0/1100962335 shutdown_connections 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- 172.21.15.69:0/1100962335 wait complete. 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 -- start start 2024-05-09T09:25:13.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.567+0000 7fe58e631740 1 Processor -- start 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b2380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b9900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d2b0d60 con 0x55ee7d2a8c40 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d1c7790 con 0x55ee7d2a8250 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ee7d2b0ee0 con 0x55ee7d2addb0 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 10 client.0 _ll_register_callbacks cb 0x55ee7d202a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe588de7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe588de7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51400/0 (socket says 172.21.15.69:51400) 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b9900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe588de7640 1 -- 172.21.15.69:0/1830432549 learned_addr learned my addr 172.21.15.69:0/1830432549 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:13.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe5895e8640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b2380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 -- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 msgr2=0x55ee7d2b2380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b2380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 -- 172.21.15.69:0/1830432549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 msgr2=0x55ee7d2b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 -- 172.21.15.69:0/1830432549 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee7cf20210 con 0x55ee7d2addb0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe588de7640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe589de9640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b9900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09c020 tx=0x7fe57c09bd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29d020e697002451 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe56cff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c09c560 con 0x55ee7d2addb0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fe57c0a4840 con 0x55ee7d2addb0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.568+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe57c0a5990 con 0x55ee7d2addb0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe57c0a0210 con 0x55ee7d2addb0 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 10 client.25789 fetch_fsmap learned FSMap version 35 2024-05-09T09:25:13.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ee7cfec620 con 0x55ee7d2addb0 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a5990 con 0x55ee7d2addb0 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fe57c005460 con 0x55ee7d2addb0 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 10 client.25789 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 10 client.25789 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 20 client.25789 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d2bda60 0x55ee7d2bfe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 4 client.25789 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ee7d219a70 con 0x55ee7d2bda60 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.569+0000 7fe5895e8640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d2bda60 0x55ee7d2bfe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.570+0000 7fe56cff9640 10 client.25789 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.570+0000 7fe5895e8640 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d2bda60 0x55ee7d2bfe50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c831f7fef05c8a8d server_cookie=173eaabcc97c6b21 in_seq=0 out_seq=0 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.570+0000 7fe56cff9640 1 -- 172.21.15.69:0/1830432549 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ee7d219a70 con 0x55ee7d2bda60 2024-05-09T09:25:13.577 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.570+0000 7fe56cff9640 10 client.25789 handle_command_reply: tid=0 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.570+0000 7fe58e631740 1 client.25789 shutdown 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 msgr2=0x55ee7d2b9900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b9900 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fe57c09c020 tx=0x7fe57c09bd30 comp rx=0 tx=0).stop 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 shutdown_connections 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d2bda60 0x55ee7d2bfe50 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b9900 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 --2- 172.21.15.69:0/1830432549 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b2380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe56cff9640 0 client.25789 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 >> 172.21.15.69:0/1830432549 conn(0x55ee7d2a38c0 msgr2=0x55ee7d2a2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 shutdown_connections 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 1 -- 172.21.15.69:0/1830432549 wait complete. 2024-05-09T09:25:13.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.571+0000 7fe58e631740 20 client.25789 trim_cache size 0 max 16384 2024-05-09T09:25:13.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 Processor -- start 2024-05-09T09:25:13.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 -- start start 2024-05-09T09:25:13.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2a8250 0x55ee7d2c55e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2c5b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841aeee0 con 0x55ee7d2addb0 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841ac7b0 con 0x55ee7d2a8c40 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.588+0000 7fe58e631740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841aa600 con 0x55ee7d2a8250 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.589+0000 7fe589de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.589+0000 7fe589de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51410/0 (socket says 172.21.15.69:51410) 2024-05-09T09:25:13.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.589+0000 7fe589de9640 1 -- 172.21.15.69:0/326200246 learned_addr learned my addr 172.21.15.69:0/326200246 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.589+0000 7fe5895e8640 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2a8250 0x55ee7d2c55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 -- 172.21.15.69:0/326200246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2a8250 msgr2=0x55ee7d2c55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2a8250 0x55ee7d2c55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 -- 172.21.15.69:0/326200246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 msgr2=0x55ee7d2c5b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2c5b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 -- 172.21.15.69:0/326200246 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ee7d2a4630 con 0x55ee7d2addb0 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe589de9640 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cd470 tx=0x7fe57c009a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87339c437de40c3d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.590+0000 7fe56effd640 1 -- 172.21.15.69:0/326200246 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c09c560 con 0x55ee7d2addb0 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.591+0000 7fe56effd640 1 -- 172.21.15.69:0/326200246 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a4ad0 con 0x55ee7d2addb0 2024-05-09T09:25:13.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.591+0000 7fe56effd640 1 -- 172.21.15.69:0/326200246 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe57c005570 con 0x55ee7d2addb0 2024-05-09T09:25:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 -- 172.21.15.69:0/326200246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 msgr2=0x55ee7d2b98c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fe57c0cd470 tx=0x7fe57c009a70 comp rx=0 tx=0).stop 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 -- 172.21.15.69:0/326200246 shutdown_connections 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b98c0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2c5b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 --2- 172.21.15.69:0/326200246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2a8250 0x55ee7d2c55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.592+0000 7fe58e631740 1 -- 172.21.15.69:0/326200246 >> 172.21.15.69:0/326200246 conn(0x55ee7d2a38c0 msgr2=0x55ee7d2a9dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.593+0000 7fe58e631740 1 -- 172.21.15.69:0/326200246 shutdown_connections 2024-05-09T09:25:13.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.593+0000 7fe58e631740 1 -- 172.21.15.69:0/326200246 wait complete. 2024-05-09T09:25:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.593+0000 7fe58e631740 1 -- start start 2024-05-09T09:25:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.593+0000 7fe58e631740 1 Processor -- start 2024-05-09T09:25:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b8550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841aac70 con 0x55ee7d2a8c40 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841ac0e0 con 0x55ee7d2a8250 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe5841ab340 con 0x55ee7d2addb0 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe58e631740 10 client.0 _ll_register_callbacks cb 0x7fe57c0979e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe5895e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe5895e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b7160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:42284/0 (socket says 172.21.15.69:42284) 2024-05-09T09:25:13.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.594+0000 7fe5895e8640 1 -- 172.21.15.69:0/1384803259 learned_addr learned my addr 172.21.15.69:0/1384803259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe589de9640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b8550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 -- 172.21.15.69:0/1384803259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 msgr2=0x55ee7d2b8550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b8550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 -- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 msgr2=0x55ee7d2b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 -- 172.21.15.69:0/1384803259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ee7d2b8b90 con 0x55ee7d2a8c40 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe589de9640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe588de7640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b76a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x55ee7d2a7a00 tx=0x7fe574012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc44fdfbb9ec2d27 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.595+0000 7fe56cff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe574011440 con 0x55ee7d2a8c40 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(96..96 src has 1..96) ==== 9129+0+0 (secure 0 0 0) 0x7fe5740115e0 con 0x55ee7d2a8c40 2024-05-09T09:25:13.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe57c0a4ad0 con 0x55ee7d2a8c40 2024-05-09T09:25:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) ==== 24+0+0 (secure 0 0 0) 0x7fe57402a050 con 0x55ee7d2a8c40 2024-05-09T09:25:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe58e631740 10 client.16554 fetch_fsmap learned FSMap version 35 2024-05-09T09:25:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.596+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=4+}) -- 0x55ee7d32c760 con 0x55ee7d2a8c40 2024-05-09T09:25:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe57c0a4ad0 con 0x55ee7d2a8c40 2024-05-09T09:25:13.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 35) ==== 8346+0+0 (secure 0 0 0) 0x7fe57401e8c0 con 0x55ee7d2a8c40 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 10 client.16554 fetch_fsmap finished waiting for FSMap version 35 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 10 client.16554 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 20 client.16554 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d32d140 0x55ee7d32f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 4 client.16554 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.597+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ee7d213dc0 con 0x55ee7d32d140 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.598+0000 7fe5895e8640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d32d140 0x55ee7d32f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.598+0000 7fe5895e8640 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d32d140 0x55ee7d32f560 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=55171bebebf93419 server_cookie=fddce7a3ec67b7f1 in_seq=0 out_seq=0 2024-05-09T09:25:13.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.598+0000 7fe56cff9640 10 client.16554 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:13.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:13 smithi163 ceph-mon[34354]: pgmap v558: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-05-09T09:25:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:13 smithi103 ceph-mon[34421]: pgmap v558: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-05-09T09:25:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.703+0000 7fe56cff9640 1 -- 172.21.15.69:0/1384803259 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ee7d213dc0 con 0x55ee7d32d140 2024-05-09T09:25:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.703+0000 7fe56cff9640 10 client.16554 handle_command_reply: tid=0 2024-05-09T09:25:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.703+0000 7fe58e631740 1 client.16554 shutdown 2024-05-09T09:25:13.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 msgr2=0x55ee7d2b76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b76a0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x55ee7d2a7a00 tx=0x7fe574012820 comp rx=0 tx=0).stop 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 shutdown_connections 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ee7d32d140 0x55ee7d32f560 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ee7d2addb0 0x55ee7d2b8550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ee7d2a8c40 0x55ee7d2b76a0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe56cff9640 0 client.16554 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 --2- 172.21.15.69:0/1384803259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ee7d2a8250 0x55ee7d2b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 >> 172.21.15.69:0/1384803259 conn(0x55ee7d2a38c0 msgr2=0x55ee7d2b2010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 shutdown_connections 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 1 -- 172.21.15.69:0/1384803259 wait complete. 2024-05-09T09:25:13.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58e631740 20 client.16554 trim_cache size 0 max 16384 2024-05-09T09:25:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:25:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:25:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:25:13.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 msgr2=0x7fe5700a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.704+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 0x7fe5700a5170 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7fe580005ab0 tx=0x7fe580000bc0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 msgr2=0x7fe5841032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x7fe57c00eb10 tx=0x7fe57c00c2d0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 shutdown_connections 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe5700a2cb0 0x7fe5700a5170 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe584102340 0x7fe5841032a0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe584101980 0x7fe584102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 --2- 172.21.15.69:0/2202706168 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe584100790 0x7fe5841046d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 >> 172.21.15.69:0/2202706168 conn(0x7fe5840fbd00 msgr2=0x7fe5840fd770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:13.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 shutdown_connections 2024-05-09T09:25:13.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:13.705+0000 7fe58b84c640 1 -- 172.21.15.69:0/2202706168 wait complete. 2024-05-09T09:25:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:13 smithi069 ceph-mon[30123]: pgmap v558: 129 pgs: 129 active+clean; 74 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 350 op/s 2024-05-09T09:25:14.305 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:25:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:25:14.305 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:25:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:25:14.305 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:25:13 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/2022616100 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x561d24c81a40 con 0x561d24c54c00 2024-05-09T09:25:14.305 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:25:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:25:14.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 263132 5.38 MB/sec execute 192 sec latency 59.017 ms 2024-05-09T09:25:14.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:14 smithi103 ceph-mon[34421]: daemon mds.c restarted 2024-05-09T09:25:14.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:14 smithi069 ceph-mon[30123]: daemon mds.c restarted 2024-05-09T09:25:14.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:14 smithi163 ceph-mon[34354]: daemon mds.c restarted 2024-05-09T09:25:15.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 265686 5.42 MB/sec execute 193 sec latency 23.443 ms 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/3019686067,v1:172.21.15.163:6837/3019686067] up:boot 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:25:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:15 smithi103 ceph-mon[34421]: pgmap v560: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 302 op/s 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/3019686067,v1:172.21.15.163:6837/3019686067] up:boot 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:25:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:15 smithi069 ceph-mon[30123]: pgmap v560: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 302 op/s 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: osdmap e97: 12 total, 12 up, 12 in 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/3019686067,v1:172.21.15.163:6837/3019686067] up:boot 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:25:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:15 smithi163 ceph-mon[34354]: pgmap v560: 129 pgs: 129 active+clean; 76 MiB data, 1.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 302 op/s 2024-05-09T09:25:16.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 267119 5.43 MB/sec execute 194 sec latency 13.669 ms 2024-05-09T09:25:17.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 269838 5.49 MB/sec execute 195 sec latency 20.467 ms 2024-05-09T09:25:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:17 smithi103 ceph-mon[34421]: pgmap v561: 129 pgs: 129 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:25:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:17 smithi069 ceph-mon[30123]: pgmap v561: 129 pgs: 129 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:25:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:17 smithi163 ceph-mon[34354]: pgmap v561: 129 pgs: 129 active+clean; 74 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:25:18.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 271037 5.49 MB/sec execute 196 sec latency 29.792 ms 2024-05-09T09:25:19.490 INFO:tasks.workunit.client.0.smithi069.stdout: 1 273488 5.53 MB/sec execute 197 sec latency 36.519 ms 2024-05-09T09:25:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:19 smithi103 ceph-mon[34421]: pgmap v562: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 350 op/s 2024-05-09T09:25:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:19 smithi069 ceph-mon[30123]: pgmap v562: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 350 op/s 2024-05-09T09:25:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:19 smithi163 ceph-mon[34354]: pgmap v562: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 350 op/s 2024-05-09T09:25:20.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 275292 5.53 MB/sec execute 198 sec latency 13.761 ms 2024-05-09T09:25:21.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 277234 5.57 MB/sec execute 199 sec latency 12.106 ms 2024-05-09T09:25:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:21 smithi103 ceph-mon[34421]: pgmap v563: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:25:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:21 smithi069 ceph-mon[30123]: pgmap v563: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:25:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:21 smithi163 ceph-mon[34354]: pgmap v563: 129 pgs: 129 active+clean; 69 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:25:22.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 279517 5.60 MB/sec execute 200 sec latency 32.250 ms 2024-05-09T09:25:23.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 280862 5.61 MB/sec execute 201 sec latency 70.274 ms 2024-05-09T09:25:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:23 smithi103 ceph-mon[34421]: pgmap v564: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-05-09T09:25:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:23 smithi069 ceph-mon[30123]: pgmap v564: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-05-09T09:25:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:23 smithi163 ceph-mon[34354]: pgmap v564: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 378 op/s 2024-05-09T09:25:24.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 283597 5.66 MB/sec execute 202 sec latency 26.011 ms 2024-05-09T09:25:25.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 284834 5.66 MB/sec execute 203 sec latency 16.945 ms 2024-05-09T09:25:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:25 smithi103 ceph-mon[34421]: pgmap v565: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 313 op/s 2024-05-09T09:25:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:25 smithi069 ceph-mon[30123]: pgmap v565: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 313 op/s 2024-05-09T09:25:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:25 smithi163 ceph-mon[34354]: pgmap v565: 129 pgs: 129 active+clean; 73 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.5 MiB/s wr, 313 op/s 2024-05-09T09:25:26.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 287578 5.72 MB/sec execute 204 sec latency 14.763 ms 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 -- 172.21.15.69:0/172875429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 msgr2=0x7f8f9c101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/172875429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c101e20 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f8f90009900 tx=0x7f8f9002a410 comp rx=0 tx=0).stop 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 -- 172.21.15.69:0/172875429 shutdown_connections 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/172875429 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/172875429 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c101e20 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/172875429 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 -- 172.21.15.69:0/172875429 >> 172.21.15.69:0/172875429 conn(0x7f8f9c0fbd20 msgr2=0x7f8f9c0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 -- 172.21.15.69:0/172875429 shutdown_connections 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.793+0000 7f8fa1a12640 1 -- 172.21.15.69:0/172875429 wait complete. 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 Processor -- start 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 -- start start 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c1a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c10c480 con 0x7f8f9c1019a0 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c10c300 con 0x7f8f9c102360 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8fa1a12640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c10c600 con 0x7f8f9c1007b0 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.794+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35888/0 (socket says 172.21.15.69:35888) 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/968779236 learned_addr learned my addr 172.21.15.69:0/968779236 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9affd640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c1a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/968779236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 msgr2=0x7f8f9c199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/968779236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 msgr2=0x7f8f9c1a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c1a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/968779236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8f9c1a1920 con 0x7f8f9c1019a0 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9affd640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8f90031440 tx=0x7f8f90032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8fa0a10640 1 -- 172.21.15.69:0/968779236 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f9003f030 con 0x7f8f9c1019a0 2024-05-09T09:25:26.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8fa0a10640 1 -- 172.21.15.69:0/968779236 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f90003ce0 con 0x7f8f9c1019a0 2024-05-09T09:25:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8f9c1a1bb0 con 0x7f8f9c1019a0 2024-05-09T09:25:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8fa0a10640 1 -- 172.21.15.69:0/968779236 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f90035070 con 0x7f8f9c1019a0 2024-05-09T09:25:26.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.795+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8f9c1a20a0 con 0x7f8f9c1019a0 2024-05-09T09:25:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa0a10640 1 -- 172.21.15.69:0/968779236 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8f90042030 con 0x7f8f9c1019a0 2024-05-09T09:25:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 Processor -- start 2024-05-09T09:25:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 -- start start 2024-05-09T09:25:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 0x55dcd0c78740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0bd7340 con 0x55dcd0c73250 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0b7f290 con 0x55dcd0c73c40 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0bd3300 con 0x55dcd0c78db0 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35890/0 (socket says 172.21.15.69:35890) 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9b7fe640 1 -- 172.21.15.69:0/3398154426 learned_addr learned my addr 172.21.15.69:0/3398154426 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 0x55dcd0c78740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8fa0a10640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 0x7f8f840a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9affd640 1 -- 172.21.15.69:0/3398154426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 msgr2=0x55dcd0c7b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9affd640 1 -- 172.21.15.69:0/3398154426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 msgr2=0x55dcd0c73670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.798+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9affd640 1 -- 172.21.15.69:0/3398154426 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd0ba38e0 con 0x55dcd0c73c40 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 0x7f8f840a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8fa0a10640 1 -- 172.21.15.69:0/968779236 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f8f900ef8a0 con 0x7f8f9c1019a0 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 0x55dcd0c78740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c101800 tx=0x7f8f8c0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1c0af0d3b46677b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f81ffb640 1 -- 172.21.15.69:0/3398154426 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c01e070 con 0x55dcd0c73c40 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 0x7f8f840a50f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55dcd0c744a0 tx=0x7f8f94004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f81ffb640 1 -- 172.21.15.69:0/3398154426 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c0035f0 con 0x55dcd0c73c40 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.799+0000 7f8f81ffb640 1 -- 172.21.15.69:0/3398154426 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c00f7a0 con 0x55dcd0c73c40 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3398154426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 msgr2=0x55dcd0c78740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 0x55dcd0c78740 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c101800 tx=0x7f8f8c0118d0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3398154426 shutdown_connections 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73c40 0x55dcd0c78740 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3398154426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c73670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3398154426 >> 172.21.15.69:0/3398154426 conn(0x55dcd0c6e8c0 msgr2=0x55dcd0c6ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3398154426 shutdown_connections 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3398154426 wait complete. 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- start start 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 Processor -- start 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c84a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0c7bd60 con 0x55dcd0c73c40 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0b92790 con 0x55dcd0c73250 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dcd0c7bee0 con 0x55dcd0c78db0 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.800+0000 7f8fa47f7740 10 client.0 _ll_register_callbacks cb 0x55dcd0bcda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35906/0 (socket says 172.21.15.69:35906) 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36730/0 (socket says 172.21.15.69:36730) 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/843051114 learned_addr learned my addr 172.21.15.69:0/843051114 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c84a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 -- 172.21.15.69:0/843051114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 msgr2=0x55dcd0c84a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c84a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 -- 172.21.15.69:0/843051114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 msgr2=0x55dcd0c7d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 -- 172.21.15.69:0/843051114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd08eb210 con 0x55dcd0c73250 2024-05-09T09:25:26.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f9affd640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55dcd0c6f830 tx=0x7f8f8c004290 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d97272e26be28c82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f7affd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c01e040 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f8c0035f0 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f8f8c005040 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8f8c032050 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8fa47f7740 10 client.25886 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.801+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dcd09b7620 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c0035f0 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f8f8c021030 con 0x55dcd0c73250 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 10 client.25886 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 10 client.25886 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 20 client.25886 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:26.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0c88ce0 0x55dcd0c8b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 4 client.25886 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcd0be4a70 con 0x55dcd0c88ce0 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0c88ce0 0x55dcd0c8b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8f7affd640 10 client.25886 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.802+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0c88ce0 0x55dcd0c8b0d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b6b28de22e3ab391 server_cookie=c5e2b70a3736625c in_seq=0 out_seq=0 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8f7affd640 1 -- 172.21.15.69:0/843051114 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcd0be4a70 con 0x55dcd0c88ce0 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8f7affd640 10 client.25886 handle_command_reply: tid=0 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 client.25886 shutdown 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 msgr2=0x55dcd0c7d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x55dcd0c6f830 tx=0x7f8f8c004290 comp rx=0 tx=0).stop 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 shutdown_connections 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0c88ce0 0x55dcd0c8b0d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0c84a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8f7affd640 0 client.25886 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c7d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/843051114 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c7d490 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.803+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 >> 172.21.15.69:0/843051114 conn(0x55dcd0c6e8c0 msgr2=0x55dcd0c6da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.804+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 shutdown_connections 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.804+0000 7f8fa47f7740 1 -- 172.21.15.69:0/843051114 wait complete. 2024-05-09T09:25:26.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.804+0000 7f8fa47f7740 20 client.25886 trim_cache size 0 max 16384 2024-05-09T09:25:26.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 Processor -- start 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 -- start start 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 0x55dcd0c71db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7cab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1aee20 con 0x55dcd0c73250 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ac6f0 con 0x55dcd0c78db0 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1aa540 con 0x55dcd0c73c40 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35912/0 (socket says 172.21.15.69:35912) 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 -- 172.21.15.69:0/3909287369 learned_addr learned my addr 172.21.15.69:0/3909287369 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7cab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 0x55dcd0c71db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 -- 172.21.15.69:0/3909287369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 msgr2=0x55dcd0c71db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 0x55dcd0c71db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 -- 172.21.15.69:0/3909287369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c78db0 msgr2=0x55dcd0c7cab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7cab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 -- 172.21.15.69:0/3909287369 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcd0c8b440 con 0x55dcd0c73250 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 0x55dcd0c71db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f9affd640 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011910 tx=0x7f8f8c009e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ca4c9600aae8fda4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.819+0000 7f8f817fa640 1 -- 172.21.15.69:0/3909287369 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c03b070 con 0x55dcd0c73250 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8f817fa640 1 -- 172.21.15.69:0/3909287369 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c016030 con 0x55dcd0c73250 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8f817fa640 1 -- 172.21.15.69:0/3909287369 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f8c01e040 con 0x55dcd0c73250 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3909287369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 msgr2=0x55dcd0c71870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f8f8c011910 tx=0x7f8f8c009e20 comp rx=0 tx=0).stop 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3909287369 shutdown_connections 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c78db0 0x55dcd0c7cab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c73c40 0x55dcd0c71db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/3909287369 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73250 0x55dcd0c71870 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3909287369 >> 172.21.15.69:0/3909287369 conn(0x55dcd0c6e8c0 msgr2=0x55dcd0c78460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3909287369 shutdown_connections 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.820+0000 7f8fa47f7740 1 -- 172.21.15.69:0/3909287369 wait complete. 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 -- start start 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 Processor -- start 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c80e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0ced1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1aabb0 con 0x55dcd0c73c40 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ac020 con 0x55dcd0c73250 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8f9c1ab280 con 0x55dcd0c78db0 2024-05-09T09:25:26.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8fa47f7740 10 client.0 _ll_register_callbacks cb 0x7f8f8c01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35928/0 (socket says 172.21.15.69:35928) 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/4260172614 learned_addr learned my addr 172.21.15.69:0/4260172614 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9affd640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c80e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0ced1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/4260172614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 msgr2=0x55dcd0ced1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0ced1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 msgr2=0x55dcd0c80e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c80e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9a7fc640 1 -- 172.21.15.69:0/4260172614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcd0ced730 con 0x55dcd0c73c40 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.821+0000 7f8f9b7fe640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0ced1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f9a7fc640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006010 tx=0x7f8f90005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f9bcfcbb65f8d301 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:26.827 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8f900f5020 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f8f900b8410 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8f8c016030 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f8f90049040 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 10 client.16575 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dcd0ceea30 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8f8c016030 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f8f9003b460 con 0x55dcd0c73c40 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 10 client.16575 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 10 client.16575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 20 client.16575 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0cef3e0 0x55dcd0cf1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 4 client.16575 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dcd0bdedc0 con 0x55dcd0cef3e0 2024-05-09T09:25:26.828 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.822+0000 7f8f9affd640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0cef3e0 0x55dcd0cf1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.823+0000 7f8f7affd640 10 client.16575 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:26.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.823+0000 7f8f9affd640 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0cef3e0 0x55dcd0cf1800 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4e02797850e11cf3 server_cookie=3d48fc5a103c03d3 in_seq=0 out_seq=0 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8f7affd640 1 -- 172.21.15.69:0/4260172614 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dcd0bdedc0 con 0x55dcd0cef3e0 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8f7affd640 10 client.16575 handle_command_reply: tid=0 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 client.16575 shutdown 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 msgr2=0x55dcd0c81360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f8f90006010 tx=0x7f8f90005c90 comp rx=0 tx=0).stop 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 shutdown_connections 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcd0cef3e0 0x55dcd0cf1800 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcd0c78db0 0x55dcd0ced1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8f7affd640 0 client.16575 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcd0c73c40 0x55dcd0c81360 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 --2- 172.21.15.69:0/4260172614 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcd0c73250 0x55dcd0c80e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 >> 172.21.15.69:0/4260172614 conn(0x55dcd0c6e8c0 msgr2=0x55dcd0c71870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 shutdown_connections 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 1 -- 172.21.15.69:0/4260172614 wait complete. 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.932+0000 7f8fa47f7740 20 client.16575 trim_cache size 0 max 16384 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:25:26.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 msgr2=0x7f8f840a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 0x7f8f840a50f0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x55dcd0c744a0 tx=0x7f8f94004a40 comp rx=0 tx=0).stop 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 msgr2=0x7f8f9c19a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f8f90031440 tx=0x7f8f90032ce0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 shutdown_connections 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8f840a2c30 0x7f8f840a50f0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8f9c102360 0x7f8f9c1a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8f9c1019a0 0x7f8f9c19a250 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 --2- 172.21.15.69:0/968779236 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8f9c1007b0 0x7f8f9c199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 >> 172.21.15.69:0/968779236 conn(0x7f8f9c0fbd20 msgr2=0x7f8f9c0fd6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 shutdown_connections 2024-05-09T09:25:26.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:26.933+0000 7f8fa1a12640 1 -- 172.21.15.69:0/968779236 wait complete. 2024-05-09T09:25:27.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 288862 5.71 MB/sec execute 205 sec latency 20.167 ms 2024-05-09T09:25:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:27 smithi103 ceph-mon[34421]: pgmap v566: 129 pgs: 129 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-05-09T09:25:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:27 smithi069 ceph-mon[30123]: pgmap v566: 129 pgs: 129 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-05-09T09:25:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:27 smithi163 ceph-mon[34354]: pgmap v566: 129 pgs: 129 active+clean; 75 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.9 MiB/s wr, 360 op/s 2024-05-09T09:25:28.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 291263 5.75 MB/sec execute 206 sec latency 65.480 ms 2024-05-09T09:25:29.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 293221 5.76 MB/sec execute 207 sec latency 33.633 ms 2024-05-09T09:25:29.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:29 smithi103 ceph-mon[34421]: pgmap v567: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 338 op/s 2024-05-09T09:25:29.872 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:29 smithi163 ceph-mon[34354]: pgmap v567: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 338 op/s 2024-05-09T09:25:29.872 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:29 smithi069 ceph-mon[30123]: pgmap v567: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.5 MiB/s wr, 338 op/s 2024-05-09T09:25:30.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 294998 5.79 MB/sec execute 208 sec latency 15.288 ms 2024-05-09T09:25:31.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 297363 5.82 MB/sec execute 209 sec latency 17.874 ms 2024-05-09T09:25:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:31 smithi103 ceph-mon[34421]: pgmap v568: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-05-09T09:25:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:31 smithi069 ceph-mon[30123]: pgmap v568: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-05-09T09:25:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:31 smithi163 ceph-mon[34354]: pgmap v568: 129 pgs: 129 active+clean; 78 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.4 MiB/s wr, 318 op/s 2024-05-09T09:25:32.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 298860 5.83 MB/sec execute 210 sec latency 18.903 ms 2024-05-09T09:25:33.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 301635 5.88 MB/sec execute 211 sec latency 44.248 ms 2024-05-09T09:25:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:33 smithi103 ceph-mon[34421]: pgmap v569: 129 pgs: 129 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 421 op/s 2024-05-09T09:25:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:33 smithi069 ceph-mon[30123]: pgmap v569: 129 pgs: 129 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 421 op/s 2024-05-09T09:25:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:33 smithi163 ceph-mon[34354]: pgmap v569: 129 pgs: 129 active+clean; 76 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 3.2 MiB/s wr, 421 op/s 2024-05-09T09:25:34.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 302709 5.87 MB/sec execute 212 sec latency 26.715 ms 2024-05-09T09:25:35.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 305368 5.92 MB/sec execute 213 sec latency 18.323 ms 2024-05-09T09:25:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:35 smithi103 ceph-mon[34421]: pgmap v570: 129 pgs: 129 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:25:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:35 smithi069 ceph-mon[30123]: pgmap v570: 129 pgs: 129 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:25:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:35 smithi163 ceph-mon[34354]: pgmap v570: 129 pgs: 129 active+clean; 78 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:25:36.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 306955 5.92 MB/sec execute 214 sec latency 20.039 ms 2024-05-09T09:25:37.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 309079 5.96 MB/sec execute 215 sec latency 9.881 ms 2024-05-09T09:25:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:37 smithi103 ceph-mon[34421]: pgmap v571: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:25:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:37 smithi069 ceph-mon[30123]: pgmap v571: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:25:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:37 smithi163 ceph-mon[34354]: pgmap v571: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:25:38.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 311533 5.99 MB/sec execute 216 sec latency 26.785 ms 2024-05-09T09:25:39.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 312892 5.99 MB/sec execute 217 sec latency 54.329 ms 2024-05-09T09:25:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:39 smithi103 ceph-mon[34421]: pgmap v572: 129 pgs: 129 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 344 op/s 2024-05-09T09:25:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:39 smithi069 ceph-mon[30123]: pgmap v572: 129 pgs: 129 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 344 op/s 2024-05-09T09:25:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:39 smithi163 ceph-mon[34354]: pgmap v572: 129 pgs: 129 active+clean; 81 MiB data, 1.1 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 344 op/s 2024-05-09T09:25:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- 172.21.15.69:0/1397802360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 msgr2=0x7f1184101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 --2- 172.21.15.69:0/1397802360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f1184101cb0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f1180009ca0 tx=0x7f118002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- 172.21.15.69:0/1397802360 shutdown_connections 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 --2- 172.21.15.69:0/1397802360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 0x7f1184109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 --2- 172.21.15.69:0/1397802360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 0x7f1184102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 --2- 172.21.15.69:0/1397802360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f1184101cb0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- 172.21.15.69:0/1397802360 >> 172.21.15.69:0/1397802360 conn(0x7f11840fce20 msgr2=0x7f11840ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- 172.21.15.69:0/1397802360 shutdown_connections 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- 172.21.15.69:0/1397802360 wait complete. 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 Processor -- start 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.029+0000 7f118ad7e640 1 -- start start 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 0x7f1184073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 0x7f1184074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f118410d510 con 0x7f1184101890 2024-05-09T09:25:40.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f118410d390 con 0x7f1184102a80 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f118410d690 con 0x7f1184103440 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59446/0 (socket says 172.21.15.69:59446) 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 -- 172.21.15.69:0/175164189 learned_addr learned my addr 172.21.15.69:0/175164189 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f117bfff640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 0x7f1184073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118931b640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 0x7f1184074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 -- 172.21.15.69:0/175164189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 msgr2=0x7f1184074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 0x7f1184074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 -- 172.21.15.69:0/175164189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 msgr2=0x7f1184073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 0x7f1184073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 -- 172.21.15.69:0/175164189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f11840749f0 con 0x7f1184101890 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1188b1a640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1180009bf0 tx=0x7f1180031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1179ffb640 1 -- 172.21.15.69:0/175164189 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f118002fa20 con 0x7f1184101890 2024-05-09T09:25:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1179ffb640 1 -- 172.21.15.69:0/175164189 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f118002fbc0 con 0x7f1184101890 2024-05-09T09:25:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f11841a1c10 con 0x7f1184101890 2024-05-09T09:25:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.030+0000 7f1179ffb640 1 -- 172.21.15.69:0/175164189 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1180046750 con 0x7f1184101890 2024-05-09T09:25:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.031+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f11841a2160 con 0x7f1184101890 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f1179ffb640 1 -- 172.21.15.69:0/175164189 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f118002fd70 con 0x7f1184101890 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 Processor -- start 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 -- start start 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f1179ffb640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 0x7f116c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124564670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x56512456c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651244c8340 con 0x565124569db0 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565124470290 con 0x565124564c40 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.033+0000 7f118db6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651244c4300 con 0x565124564250 2024-05-09T09:25:40.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f118931b640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 0x7f116c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f117bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124564670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f117bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:44064/0 (socket says 172.21.15.69:44064) 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f117bfff640 1 -- 172.21.15.69:0/1336723700 learned_addr learned my addr 172.21.15.69:0/1336723700 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f118931b640 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x56512456c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1179ffb640 1 -- 172.21.15.69:0/175164189 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f11800f06e0 con 0x7f1184101890 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 -- 172.21.15.69:0/1336723700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 msgr2=0x565124569740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 -- 172.21.15.69:0/1336723700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 msgr2=0x56512456c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x56512456c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 -- 172.21.15.69:0/1336723700 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651244948e0 con 0x565124564250 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f117bfff640 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f118931b640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 0x7f116c0a50f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f117c009fd0 tx=0x7f117c009b80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f1188b1a640 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124564670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11800edeb0 tx=0x7f1180005c30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df446e840e59354f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f116affd640 1 -- 172.21.15.69:0/1336723700 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11800d5dc0 con 0x565124564250 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f116affd640 1 -- 172.21.15.69:0/1336723700 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11800d7070 con 0x565124564250 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.034+0000 7f116affd640 1 -- 172.21.15.69:0/1336723700 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f11800f0420 con 0x565124564250 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- 172.21.15.69:0/1336723700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 msgr2=0x565124564670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124564670 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f11800edeb0 tx=0x7f1180005c30 comp rx=0 tx=0).stop 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- 172.21.15.69:0/1336723700 shutdown_connections 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x56512456c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564c40 0x565124569740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 --2- 172.21.15.69:0/1336723700 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124564670 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- 172.21.15.69:0/1336723700 >> 172.21.15.69:0/1336723700 conn(0x56512455f8c0 msgr2=0x56512455fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- 172.21.15.69:0/1336723700 shutdown_connections 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- 172.21.15.69:0/1336723700 wait complete. 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 -- start start 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.035+0000 7f118db6f740 1 Processor -- start 2024-05-09T09:25:40.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x56512456e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 0x56512456e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651244c4300 con 0x565124569db0 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565124483790 con 0x565124564250 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56512456cb20 con 0x565124564c40 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118db6f740 10 client.0 _ll_register_callbacks cb 0x5651244bea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f1188b1a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x56512456e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59474/0 (socket says 172.21.15.69:59474) 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 -- 172.21.15.69:0/2227945851 learned_addr learned my addr 172.21.15.69:0/2227945851 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f117bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 0x56512456e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 -- 172.21.15.69:0/2227945851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 msgr2=0x56512456e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 0x56512456e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 -- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 msgr2=0x56512456e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x56512456e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 -- 172.21.15.69:0/2227945851 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651241dc210 con 0x565124569db0 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f1188b1a640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x56512456e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f117bfff640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 0x56512456e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.036+0000 7f118931b640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f117c004500 tx=0x7f117c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378f5c5aebb3ff98 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f1163fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f117c024070 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f117c003d80 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11800d7070 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f117c00fe20 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f118db6f740 10 client.16587 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5651242a8620 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.037+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11800d7070 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f117c004a70 con 0x565124569db0 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 10 client.16587 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:40.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 10 client.16587 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 20 client.16587 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565124579cf0 0x56512457c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 4 client.16587 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651244d5a70 con 0x565124579cf0 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f1188b1a640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565124579cf0 0x56512457c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f1188b1a640 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565124579cf0 0x56512457c0e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5b35f666fae6269 server_cookie=e02291e87aab9d1a in_seq=0 out_seq=0 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.038+0000 7f1163fff640 10 client.16587 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f1163fff640 1 -- 172.21.15.69:0/2227945851 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651244d5a70 con 0x565124579cf0 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f1163fff640 10 client.16587 handle_command_reply: tid=0 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 client.16587 shutdown 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 msgr2=0x565124575670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f117c004500 tx=0x7f117c01d000 comp rx=0 tx=0).stop 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 shutdown_connections 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124569db0 0x565124575670 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565124579cf0 0x56512457c0e0 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564c40 0x56512456e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 --2- 172.21.15.69:0/2227945851 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x56512456e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 >> 172.21.15.69:0/2227945851 conn(0x56512455f8c0 msgr2=0x56512455ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f1163fff640 0 client.16587 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 shutdown_connections 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 1 -- 172.21.15.69:0/2227945851 wait complete. 2024-05-09T09:25:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.039+0000 7f118db6f740 20 client.16587 trim_cache size 0 max 16384 2024-05-09T09:25:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.055+0000 7f118db6f740 1 Processor -- start 2024-05-09T09:25:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.055+0000 7f118db6f740 1 -- start start 2024-05-09T09:25:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.055+0000 7f118db6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x5651245819d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.055+0000 7f118db6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.055+0000 7f118db6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 0x565124575630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f118db6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11841aee40 con 0x565124564c40 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f118db6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11841ac710 con 0x565124564250 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f118db6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11841aa560 con 0x565124569db0 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59490/0 (socket says 172.21.15.69:59490) 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 -- 172.21.15.69:0/47006849 learned_addr learned my addr 172.21.15.69:0/47006849 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f118931b640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 0x565124575630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f1188b1a640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x5651245819d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 -- 172.21.15.69:0/47006849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 msgr2=0x565124575630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 0x565124575630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 -- 172.21.15.69:0/47006849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 msgr2=0x5651245819d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x5651245819d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 -- 172.21.15.69:0/47006849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565124559ba0 con 0x565124564c40 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f118931b640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 0x565124575630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f1188b1a640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x5651245819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f117bfff640 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x5651245656b0 tx=0x7f1174005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7dd1a7c2a7adbd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f116a7fc640 1 -- 172.21.15.69:0/47006849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174022070 con 0x565124564c40 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f116a7fc640 1 -- 172.21.15.69:0/47006849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11740120d0 con 0x565124564c40 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.056+0000 7f116a7fc640 1 -- 172.21.15.69:0/47006849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174003850 con 0x565124564c40 2024-05-09T09:25:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- 172.21.15.69:0/47006849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 msgr2=0x565124581f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x5651245656b0 tx=0x7f1174005f90 comp rx=0 tx=0).stop 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- 172.21.15.69:0/47006849 shutdown_connections 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124569db0 0x565124575630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124581f10 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 --2- 172.21.15.69:0/47006849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124564250 0x5651245819d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- 172.21.15.69:0/47006849 >> 172.21.15.69:0/47006849 conn(0x56512455f8c0 msgr2=0x565124565f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- 172.21.15.69:0/47006849 shutdown_connections 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- 172.21.15.69:0/47006849 wait complete. 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 -- start start 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.057+0000 7f118db6f740 1 Processor -- start 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124572a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 0x565124573fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f11841aabd0 con 0x565124564c40 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f11841ac040 con 0x565124569db0 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f11841ab2a0 con 0x565124564250 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118db6f740 10 client.0 _ll_register_callbacks cb 0x7f117c009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59494/0 (socket says 172.21.15.69:59494) 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 -- 172.21.15.69:0/3282241837 learned_addr learned my addr 172.21.15.69:0/3282241837 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118931b640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 0x565124573fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f1188b1a640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124572a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 -- 172.21.15.69:0/3282241837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 msgr2=0x565124572a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124572a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 -- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 msgr2=0x565124573fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 0x565124573fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 -- 172.21.15.69:0/3282241837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651245745b0 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f118931b640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 0x565124573fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f117bfff640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1174020040 tx=0x7f11740124b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a78dcb72fe1b614d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f1163fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1174022070 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.058+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f1174004460 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f11740120d0 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1174037050 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 10 client.16593 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5651245758f0 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f11740120d0 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f1174029640 con 0x565124564c40 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 10 client.16593 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 10 client.16593 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 20 client.16593 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:40.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651245e0990 0x5651245e2db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 4 client.16593 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:25:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651244cfdc0 con 0x5651245e0990 2024-05-09T09:25:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.059+0000 7f1188b1a640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651245e0990 0x5651245e2db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.060+0000 7f1188b1a640 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651245e0990 0x5651245e2db0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d32ea2bfe8d9aea server_cookie=17b51db88893f1f in_seq=0 out_seq=0 2024-05-09T09:25:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.060+0000 7f1163fff640 10 client.16593 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:40.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f1163fff640 1 -- 172.21.15.69:0/3282241837 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651244cfdc0 con 0x5651245e0990 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f1163fff640 10 client.16593 handle_command_reply: tid=0 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 client.16593 shutdown 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 msgr2=0x565124572fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f1174020040 tx=0x7f11740124b0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 shutdown_connections 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565124569db0 0x565124573fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565124564c40 0x565124572fb0 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651245e0990 0x5651245e2db0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 --2- 172.21.15.69:0/3282241837 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x565124564250 0x565124572a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 >> 172.21.15.69:0/3282241837 conn(0x56512455f8c0 msgr2=0x565124565f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f1163fff640 0 client.16593 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 shutdown_connections 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 1 -- 172.21.15.69:0/3282241837 wait complete. 2024-05-09T09:25:40.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.166+0000 7f118db6f740 20 client.16593 trim_cache size 0 max 16384 2024-05-09T09:25:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:25:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:25:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:25:40.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:25:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 msgr2=0x7f116c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 0x7f116c0a50f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f117c009fd0 tx=0x7f117c009b80 comp rx=0 tx=0).stop 2024-05-09T09:25:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 msgr2=0x7f11840755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f1180009bf0 tx=0x7f1180031ee0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 shutdown_connections 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f116c0a2c30 0x7f116c0a50f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1184103440 0x7f1184074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1184102a80 0x7f1184073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 --2- 172.21.15.69:0/175164189 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1184101890 0x7f11840755e0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 >> 172.21.15.69:0/175164189 conn(0x7f11840fce20 msgr2=0x7f1184105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 shutdown_connections 2024-05-09T09:25:40.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:40.167+0000 7f118ad7e640 1 -- 172.21.15.69:0/175164189 wait complete. 2024-05-09T09:25:40.491 INFO:tasks.workunit.client.0.smithi069.stdout: 1 315531 6.03 MB/sec execute 218 sec latency 62.360 ms 2024-05-09T09:25:41.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 316888 6.03 MB/sec execute 219 sec latency 13.668 ms 2024-05-09T09:25:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:41 smithi103 ceph-mon[34421]: pgmap v573: 129 pgs: 129 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-05-09T09:25:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:41 smithi069 ceph-mon[30123]: pgmap v573: 129 pgs: 129 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-05-09T09:25:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:41 smithi163 ceph-mon[34354]: pgmap v573: 129 pgs: 129 active+clean; 81 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.5 MiB/s wr, 279 op/s 2024-05-09T09:25:42.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 319501 6.08 MB/sec execute 220 sec latency 14.658 ms 2024-05-09T09:25:43.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 321010 6.08 MB/sec execute 221 sec latency 17.369 ms 2024-05-09T09:25:43.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:43 smithi163 ceph-mon[34354]: pgmap v574: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 385 op/s 2024-05-09T09:25:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:43 smithi103 ceph-mon[34421]: pgmap v574: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 385 op/s 2024-05-09T09:25:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:43 smithi069 ceph-mon[30123]: pgmap v574: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 3.2 MiB/s wr, 385 op/s 2024-05-09T09:25:44.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 323155 6.11 MB/sec execute 222 sec latency 40.428 ms 2024-05-09T09:25:45.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 324777 6.11 MB/sec execute 223 sec latency 18.297 ms 2024-05-09T09:25:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:45 smithi103 ceph-mon[34421]: pgmap v575: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 306 op/s 2024-05-09T09:25:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:45 smithi069 ceph-mon[30123]: pgmap v575: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 306 op/s 2024-05-09T09:25:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:45 smithi163 ceph-mon[34354]: pgmap v575: 129 pgs: 129 active+clean; 79 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 306 op/s 2024-05-09T09:25:46.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 326817 6.15 MB/sec execute 224 sec latency 9.915 ms 2024-05-09T09:25:47.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 328921 6.15 MB/sec execute 225 sec latency 23.176 ms 2024-05-09T09:25:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:47 smithi103 ceph-mon[34421]: pgmap v576: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 357 op/s 2024-05-09T09:25:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:47 smithi069 ceph-mon[30123]: pgmap v576: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 357 op/s 2024-05-09T09:25:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:47 smithi163 ceph-mon[34354]: pgmap v576: 129 pgs: 129 active+clean; 80 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 357 op/s 2024-05-09T09:25:48.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 330571 6.18 MB/sec execute 226 sec latency 14.337 ms 2024-05-09T09:25:49.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 332808 6.20 MB/sec execute 227 sec latency 54.350 ms 2024-05-09T09:25:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:49 smithi103 ceph-mon[34421]: pgmap v577: 129 pgs: 129 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 369 op/s 2024-05-09T09:25:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:49 smithi069 ceph-mon[30123]: pgmap v577: 129 pgs: 129 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 369 op/s 2024-05-09T09:25:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:49 smithi163 ceph-mon[34354]: pgmap v577: 129 pgs: 129 active+clean; 83 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 369 op/s 2024-05-09T09:25:50.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 334219 6.21 MB/sec execute 228 sec latency 15.044 ms 2024-05-09T09:25:51.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 336983 6.25 MB/sec execute 229 sec latency 12.790 ms 2024-05-09T09:25:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:51 smithi103 ceph-mon[34421]: pgmap v578: 129 pgs: 129 active+clean; 82 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.5 MiB/s wr, 320 op/s 2024-05-09T09:25:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:51 smithi069 ceph-mon[30123]: pgmap v578: 129 pgs: 129 active+clean; 82 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.5 MiB/s wr, 320 op/s 2024-05-09T09:25:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:51 smithi163 ceph-mon[34354]: pgmap v578: 129 pgs: 129 active+clean; 82 MiB data, 1.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1020 KiB/s rd, 2.5 MiB/s wr, 320 op/s 2024-05-09T09:25:52.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 338198 6.24 MB/sec execute 230 sec latency 14.131 ms 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 -- 172.21.15.69:0/4241550803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4101980 msgr2=0x7f7da4101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 --2- 172.21.15.69:0/4241550803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4101980 0x7f7da4101e00 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f7d94009900 tx=0x7f7d9402a410 comp rx=0 tx=0).stop 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 -- 172.21.15.69:0/4241550803 shutdown_connections 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 --2- 172.21.15.69:0/4241550803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4102340 0x7f7da4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 --2- 172.21.15.69:0/4241550803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4101980 0x7f7da4101e00 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 --2- 172.21.15.69:0/4241550803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4100790 0x7f7da4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 -- 172.21.15.69:0/4241550803 >> 172.21.15.69:0/4241550803 conn(0x7f7da40fbd00 msgr2=0x7f7da40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 -- 172.21.15.69:0/4241550803 shutdown_connections 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.259+0000 7f7dab932640 1 -- 172.21.15.69:0/4241550803 wait complete. 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 Processor -- start 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 -- start start 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 0x7f7da4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4102340 0x7f7da419f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7da410c3f0 con 0x7f7da4100790 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7da410c270 con 0x7f7da4102340 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7dab932640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7da410c570 con 0x7f7da4101980 2024-05-09T09:25:53.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52916/0 (socket says 172.21.15.69:52916) 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 -- 172.21.15.69:0/86228004 learned_addr learned my addr 172.21.15.69:0/86228004 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4102340 0x7f7da419f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 0x7f7da4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 -- 172.21.15.69:0/86228004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 msgr2=0x7f7da4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 0x7f7da4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 -- 172.21.15.69:0/86228004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4102340 msgr2=0x7f7da419f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4102340 0x7f7da419f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da96ce640 1 -- 172.21.15.69:0/86228004 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7da419f6f0 con 0x7f7da4100790 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.260+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 0x7f7da4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7da96ce640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7da000e9c0 tx=0x7f7da000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7d9a7fc640 1 -- 172.21.15.69:0/86228004 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da000cd80 con 0x7f7da4100790 2024-05-09T09:25:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7d9a7fc640 1 -- 172.21.15.69:0/86228004 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da00035f0 con 0x7f7da4100790 2024-05-09T09:25:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7d9a7fc640 1 -- 172.21.15.69:0/86228004 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da0010690 con 0x7f7da4100790 2024-05-09T09:25:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7da419f9e0 con 0x7f7da4100790 2024-05-09T09:25:53.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.261+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7da419ff00 con 0x7f7da4100790 2024-05-09T09:25:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7d9a7fc640 1 -- 172.21.15.69:0/86228004 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7da00037a0 con 0x7f7da4100790 2024-05-09T09:25:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 Processor -- start 2024-05-09T09:25:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 -- start start 2024-05-09T09:25:53.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b011e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0123740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5561b0082340 con 0x5561b0123db0 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5561b002a290 con 0x5561b011ec40 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7dae63d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5561b007e300 con 0x5561b011e250 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da96ce640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0123740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52928/0 (socket says 172.21.15.69:52928) 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 -- 172.21.15.69:0/3352315863 learned_addr learned my addr 172.21.15.69:0/3352315863 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b011e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 -- 172.21.15.69:0/3352315863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 msgr2=0x5561b011e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7d9a7fc640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 0x7f7d900a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b011e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 -- 172.21.15.69:0/3352315863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 msgr2=0x5561b0123740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0123740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.264+0000 7f7da8ecd640 1 -- 172.21.15.69:0/3352315863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b004e8e0 con 0x5561b0123db0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7da96ce640 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0123740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 0x7f7d900a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7d9a7fc640 1 -- 172.21.15.69:0/86228004 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f7da00cf850 con 0x7f7da4100790 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7d9402a930 tx=0x7f7d94032d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f50b797fb49945d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7d8f7fe640 1 -- 172.21.15.69:0/3352315863 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9403f070 con 0x5561b0123db0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7d8f7fe640 1 -- 172.21.15.69:0/3352315863 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d940318b0 con 0x5561b0123db0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.265+0000 7f7d8f7fe640 1 -- 172.21.15.69:0/3352315863 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d94035070 con 0x5561b0123db0 2024-05-09T09:25:53.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- 172.21.15.69:0/3352315863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 msgr2=0x5561b0126240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 0x7f7d900a50f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007310 tx=0x7f7d9c004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f7d9402a930 tx=0x7f7d94032d40 comp rx=0 tx=0).stop 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- 172.21.15.69:0/3352315863 shutdown_connections 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b0126240 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0123740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 --2- 172.21.15.69:0/3352315863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b011e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- 172.21.15.69:0/3352315863 >> 172.21.15.69:0/3352315863 conn(0x5561b01198c0 msgr2=0x5561b0119cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- 172.21.15.69:0/3352315863 shutdown_connections 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- 172.21.15.69:0/3352315863 wait complete. 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 -- start start 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.266+0000 7f7dae63d740 1 Processor -- start 2024-05-09T09:25:53.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b0128400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0128940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5561b0126d00 con 0x5561b0123db0 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5561b003d790 con 0x5561b011ec40 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5561b0126e80 con 0x5561b011e250 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 10 client.0 _ll_register_callbacks cb 0x5561b0078a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52942/0 (socket says 172.21.15.69:52942) 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 -- 172.21.15.69:0/918169595 learned_addr learned my addr 172.21.15.69:0/918169595 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:53.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0128940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da96ce640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b0128400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 -- 172.21.15.69:0/918169595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 msgr2=0x5561b0128400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b0128400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 -- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 msgr2=0x5561b0128940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0128940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 -- 172.21.15.69:0/918169595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561afd96210 con 0x5561b0123db0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0128940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c012e50 tx=0x7f7d9c004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4f5ff8930e2010f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c027670 con 0x5561b0123db0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7d940318b0 con 0x5561b0123db0 2024-05-09T09:25:53.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.267+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f7d9c027810 con 0x5561b0123db0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7d9c027a10 con 0x5561b0123db0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 10 client.16605 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5561afe62620 con 0x5561b0123db0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7d940318b0 con 0x5561b0123db0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f7d9c01b440 con 0x5561b0123db0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 10 client.16605 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 10 client.16605 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 20 client.16605 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b0133fa0 0x5561b0136390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 4 client.16605 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5561b008fa70 con 0x5561b0133fa0 2024-05-09T09:25:53.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7da96ce640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b0133fa0 0x5561b0136390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.268+0000 7f7d8cff9640 10 client.16605 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.269+0000 7f7da96ce640 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b0133fa0 0x5561b0136390 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8a74f005dccd1d5 server_cookie=21ead12f76ec2aac in_seq=0 out_seq=0 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7d8cff9640 1 -- 172.21.15.69:0/918169595 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5561b008fa70 con 0x5561b0133fa0 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7d8cff9640 10 client.16605 handle_command_reply: tid=0 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 client.16605 shutdown 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 msgr2=0x5561b012f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c012e50 tx=0x7f7d9c004ab0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 shutdown_connections 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012f980 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b0128940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.278 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b0133fa0 0x5561b0136390 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 --2- 172.21.15.69:0/918169595 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b0128400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7d8cff9640 0 client.16605 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 >> 172.21.15.69:0/918169595 conn(0x5561b01198c0 msgr2=0x5561b0118a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.270+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 shutdown_connections 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.271+0000 7f7dae63d740 1 -- 172.21.15.69:0/918169595 wait complete. 2024-05-09T09:25:53.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.271+0000 7f7dae63d740 20 client.16605 trim_cache size 0 max 16384 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.285+0000 7f7dae63d740 1 Processor -- start 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.285+0000 7f7dae63d740 1 -- start start 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 0x5561b012f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b0123db0 0x5561b013bca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41acc20 con 0x5561b011e250 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41aa4f0 con 0x5561b0123db0 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7dae63d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41a8340 con 0x5561b011ec40 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52950/0 (socket says 172.21.15.69:52950) 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 -- 172.21.15.69:0/2991105863 learned_addr learned my addr 172.21.15.69:0/2991105863 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b0123db0 0x5561b013bca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 0x5561b012f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 -- 172.21.15.69:0/2991105863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 msgr2=0x5561b012f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 0x5561b012f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 -- 172.21.15.69:0/2991105863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b0123db0 msgr2=0x5561b013bca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b0123db0 0x5561b013bca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 -- 172.21.15.69:0/2991105863 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5561b011f020 con 0x5561b011e250 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 0x5561b012f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7da96ce640 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009830 tx=0x7f7da00cfa50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f90831e1ade8e2e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7d8effd640 1 -- 172.21.15.69:0/2991105863 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da009b690 con 0x5561b011e250 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7d8effd640 1 -- 172.21.15.69:0/2991105863 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da009c430 con 0x5561b011e250 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.286+0000 7f7d8effd640 1 -- 172.21.15.69:0/2991105863 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7da009a0c0 con 0x5561b011e250 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- 172.21.15.69:0/2991105863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 msgr2=0x5561b0119ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7f7da0009830 tx=0x7f7da00cfa50 comp rx=0 tx=0).stop 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- 172.21.15.69:0/2991105863 shutdown_connections 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b0123db0 0x5561b013bca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011ec40 0x5561b012f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 --2- 172.21.15.69:0/2991105863 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b011e250 0x5561b0119ca0 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- 172.21.15.69:0/2991105863 >> 172.21.15.69:0/2991105863 conn(0x5561b01198c0 msgr2=0x5561b0123450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- 172.21.15.69:0/2991105863 shutdown_connections 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- 172.21.15.69:0/2991105863 wait complete. 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.287+0000 7f7dae63d740 1 -- start start 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 Processor -- start 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b012cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b012d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41a89b0 con 0x5561b0123db0 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41a9e20 con 0x5561b011ec40 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7da41a9080 con 0x5561b011e250 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7dae63d740 10 client.0 _ll_register_callbacks cb 0x7f7d9c011f00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:25:53.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52964/0 (socket says 172.21.15.69:52964) 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 -- 172.21.15.69:0/236609605 learned_addr learned my addr 172.21.15.69:0/236609605 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da8ecd640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b012d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da96ce640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b012cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 -- 172.21.15.69:0/236609605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 msgr2=0x5561b012cfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b012cfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 -- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 msgr2=0x5561b012d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b012d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.288+0000 7f7da9ecf640 1 -- 172.21.15.69:0/236609605 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5561b012eab0 con 0x5561b0123db0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7da9ecf640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c01b880 tx=0x7f7d9c027de0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d8a720de2638898 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7d9c01db80 con 0x5561b0123db0 2024-05-09T09:25:53.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f7d9c027470 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7da009c430 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f7d9c027670 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 10 client.16611 fetch_fsmap learned FSMap version 37 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5561b019a510 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7da009c430 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f7d9c01ec00 con 0x5561b0123db0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 10 client.16611 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 10 client.16611 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 20 client.16611 populate_metadata read hostname 'smithi069' 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b019ae20 0x5561b019d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 4 client.16611 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.289+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5561b0089dc0 con 0x5561b019ae20 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.290+0000 7f7da96ce640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b019ae20 0x5561b019d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.290+0000 7f7d8cff9640 10 client.16611 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:53.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.290+0000 7f7da96ce640 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b019ae20 0x5561b019d240 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bcda40cac6416a96 server_cookie=27885a1733847b7f in_seq=0 out_seq=0 2024-05-09T09:25:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.434+0000 7f7d8cff9640 1 -- 172.21.15.69:0/236609605 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5561b0089dc0 con 0x5561b019ae20 2024-05-09T09:25:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.434+0000 7f7d8cff9640 10 client.16611 handle_command_reply: tid=0 2024-05-09T09:25:53.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.434+0000 7f7dae63d740 1 client.16611 shutdown 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 msgr2=0x5561b012e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c01b880 tx=0x7f7d9c027de0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 shutdown_connections 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5561b0123db0 0x5561b012e4c0 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5561b011ec40 0x5561b012d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5561b019ae20 0x5561b019d240 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 --2- 172.21.15.69:0/236609605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5561b011e250 0x5561b012cfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7d8cff9640 0 client.16611 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 >> 172.21.15.69:0/236609605 conn(0x5561b01198c0 msgr2=0x5561b011fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 shutdown_connections 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 1 -- 172.21.15.69:0/236609605 wait complete. 2024-05-09T09:25:53.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.435+0000 7f7dae63d740 20 client.16611 trim_cache size 0 max 16384 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 msgr2=0x7f7d900a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 0x7f7d900a50f0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f7d9c007310 tx=0x7f7d9c004a40 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 msgr2=0x7f7da4197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f7da000e9c0 tx=0x7f7da000ee80 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 shutdown_connections 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7d900a2c30 0x7f7d900a50f0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7da4102340 0x7f7da419f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7da4101980 0x7f7da4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 --2- 172.21.15.69:0/86228004 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7da4100790 0x7f7da4197b20 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 >> 172.21.15.69:0/86228004 conn(0x7f7da40fbd00 msgr2=0x7f7da40fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 shutdown_connections 2024-05-09T09:25:53.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:25:53.436+0000 7f7dab932640 1 -- 172.21.15.69:0/86228004 wait complete. 2024-05-09T09:25:53.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 340627 6.28 MB/sec execute 231 sec latency 93.334 ms 2024-05-09T09:25:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:53 smithi103 ceph-mon[34421]: pgmap v579: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:25:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:53 smithi069 ceph-mon[30123]: pgmap v579: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:25:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:53 smithi163 ceph-mon[34354]: pgmap v579: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:25:54.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 341445 6.27 MB/sec execute 232 sec latency 58.488 ms 2024-05-09T09:25:55.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 344223 6.31 MB/sec execute 233 sec latency 35.108 ms 2024-05-09T09:25:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:55 smithi103 ceph-mon[34421]: pgmap v580: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:25:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:55 smithi069 ceph-mon[30123]: pgmap v580: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:25:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:55 smithi163 ceph-mon[34354]: pgmap v580: 129 pgs: 129 active+clean; 87 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:25:56.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 345520 6.31 MB/sec execute 234 sec latency 22.221 ms 2024-05-09T09:25:57.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 347917 6.34 MB/sec execute 235 sec latency 17.463 ms 2024-05-09T09:25:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:57 smithi103 ceph-mon[34421]: pgmap v581: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-05-09T09:25:57.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:57 smithi069 ceph-mon[30123]: pgmap v581: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-05-09T09:25:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:57 smithi163 ceph-mon[34354]: pgmap v581: 129 pgs: 129 active+clean; 83 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 334 op/s 2024-05-09T09:25:58.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 349314 6.34 MB/sec execute 236 sec latency 19.851 ms 2024-05-09T09:25:59.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 351487 6.37 MB/sec execute 237 sec latency 46.344 ms 2024-05-09T09:25:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:25:59 smithi103 ceph-mon[34421]: pgmap v582: 129 pgs: 129 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-05-09T09:25:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:25:59 smithi069 ceph-mon[30123]: pgmap v582: 129 pgs: 129 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-05-09T09:25:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:25:59 smithi163 ceph-mon[34354]: pgmap v582: 129 pgs: 129 active+clean; 86 MiB data, 1.3 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.6 MiB/s wr, 367 op/s 2024-05-09T09:26:00.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 353455 6.37 MB/sec execute 238 sec latency 18.038 ms 2024-05-09T09:26:01.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 355255 6.40 MB/sec execute 239 sec latency 11.938 ms 2024-05-09T09:26:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:01 smithi103 ceph-mon[34421]: pgmap v583: 129 pgs: 129 active+clean; 85 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 316 op/s 2024-05-09T09:26:01.639 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:01 smithi163 ceph-mon[34354]: pgmap v583: 129 pgs: 129 active+clean; 85 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 316 op/s 2024-05-09T09:26:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:01 smithi069 ceph-mon[30123]: pgmap v583: 129 pgs: 129 active+clean; 85 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 316 op/s 2024-05-09T09:26:02.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 357680 6.43 MB/sec execute 240 sec latency 14.705 ms 2024-05-09T09:26:03.492 INFO:tasks.workunit.client.0.smithi069.stdout: 1 359152 6.43 MB/sec execute 241 sec latency 11.971 ms 2024-05-09T09:26:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:03 smithi103 ceph-mon[34421]: pgmap v584: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:26:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:03 smithi069 ceph-mon[30123]: pgmap v584: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:26:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:03 smithi163 ceph-mon[34354]: pgmap v584: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:26:04.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 361455 6.45 MB/sec execute 242 sec latency 60.848 ms 2024-05-09T09:26:05.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:04 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:26:05.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:04 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:26:05.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:04 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:26:05.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 362694 6.45 MB/sec execute 243 sec latency 28.910 ms 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: pgmap v585: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:26:06.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:05 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: pgmap v585: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:26:06.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:05 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:26:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: pgmap v585: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 360 op/s 2024-05-09T09:26:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:26:06.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:05 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:26:06.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 365420 6.50 MB/sec execute 244 sec latency 18.775 ms 2024-05-09T09:26:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- 172.21.15.69:0/3666850543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c101890 msgr2=0x7f158c101cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 --2- 172.21.15.69:0/3666850543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c101890 0x7f158c101cb0 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7f157c009900 tx=0x7f157c02a410 comp rx=0 tx=0).stop 2024-05-09T09:26:06.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- 172.21.15.69:0/3666850543 shutdown_connections 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 --2- 172.21.15.69:0/3666850543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c103440 0x7f158c109cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 --2- 172.21.15.69:0/3666850543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 0x7f158c102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 --2- 172.21.15.69:0/3666850543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c101890 0x7f158c101cb0 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- 172.21.15.69:0/3666850543 >> 172.21.15.69:0/3666850543 conn(0x7f158c0fce20 msgr2=0x7f158c0ff240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- 172.21.15.69:0/3666850543 shutdown_connections 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- 172.21.15.69:0/3666850543 wait complete. 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 Processor -- start 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.527+0000 7f1592314640 1 -- start start 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c101890 0x7f158c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 0x7f158c198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f158c10d510 con 0x7f158c103440 2024-05-09T09:26:06.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f158c10d390 con 0x7f158c102a80 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f1592314640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f158c10d690 con 0x7f158c101890 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38826/0 (socket says 172.21.15.69:38826) 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 -- 172.21.15.69:0/4069185990 learned_addr learned my addr 172.21.15.69:0/4069185990 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f158b7fe640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 0x7f158c198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f158bfff640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c101890 0x7f158c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 -- 172.21.15.69:0/4069185990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c101890 msgr2=0x7f158c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c101890 0x7f158c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 -- 172.21.15.69:0/4069185990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 msgr2=0x7f158c198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 0x7f158c198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 -- 172.21.15.69:0/4069185990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f158c19f710 con 0x7f158c103440 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.528+0000 7f15908b1640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f158400eb10 tx=0x7f158400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.529+0000 7f15897fa640 1 -- 172.21.15.69:0/4069185990 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584004b20 con 0x7f158c103440 2024-05-09T09:26:06.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.529+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f158c19fa00 con 0x7f158c103440 2024-05-09T09:26:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.529+0000 7f15897fa640 1 -- 172.21.15.69:0/4069185990 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1584004cc0 con 0x7f158c103440 2024-05-09T09:26:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.529+0000 7f15897fa640 1 -- 172.21.15.69:0/4069185990 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1584010760 con 0x7f158c103440 2024-05-09T09:26:06.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.529+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f158c19ff50 con 0x7f158c103440 2024-05-09T09:26:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f15897fa640 1 -- 172.21.15.69:0/4069185990 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1584005020 con 0x7f158c103440 2024-05-09T09:26:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 Processor -- start 2024-05-09T09:26:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 -- start start 2024-05-09T09:26:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f060d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0610240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5653f056c340 con 0x5653f0608250 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5653f0514290 con 0x5653f060ddb0 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f1595026740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5653f0568300 con 0x5653f0608c40 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.531+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38832/0 (socket says 172.21.15.69:38832) 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 -- 172.21.15.69:0/3259843992 learned_addr learned my addr 172.21.15.69:0/3259843992 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158bfff640 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f060d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0610240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15897fa640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 0x7f15740a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15908b1640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 0x7f15740a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 -- 172.21.15.69:0/3259843992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 msgr2=0x5653f060d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f060d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 -- 172.21.15.69:0/3259843992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 msgr2=0x5653f0610240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0610240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 -- 172.21.15.69:0/3259843992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653f05388e0 con 0x5653f0608250 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15897fa640 1 -- 172.21.15.69:0/4069185990 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f15840cfa60 con 0x7f158c103440 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1580013840 tx=0x7f1580005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1194a6b62647a64b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15727fc640 1 -- 172.21.15.69:0/3259843992 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580013170 con 0x5653f0608250 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15908b1640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 0x7f15740a50f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1584098910 tx=0x7f15840c6ad0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15727fc640 1 -- 172.21.15.69:0/3259843992 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1580005500 con 0x5653f0608250 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.532+0000 7f15727fc640 1 -- 172.21.15.69:0/3259843992 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580004d70 con 0x5653f0608250 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- 172.21.15.69:0/3259843992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 msgr2=0x5653f0608670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7f1580013840 tx=0x7f1580005f90 comp rx=0 tx=0).stop 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- 172.21.15.69:0/3259843992 shutdown_connections 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0610240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f060d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 --2- 172.21.15.69:0/3259843992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0608670 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- 172.21.15.69:0/3259843992 >> 172.21.15.69:0/3259843992 conn(0x5653f06038c0 msgr2=0x5653f0603cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- 172.21.15.69:0/3259843992 shutdown_connections 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- 172.21.15.69:0/3259843992 wait complete. 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.533+0000 7f1595026740 1 -- start start 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 Processor -- start 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608250 0x5653f06123d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 0x5653f0619950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5653f0610d00 con 0x5653f060ddb0 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5653f0527790 con 0x5653f0608c40 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5653f0610e80 con 0x5653f0608250 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f1595026740 10 client.0 _ll_register_callbacks cb 0x5653f0562a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:06.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 0x5653f0619950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54288/0 (socket says 172.21.15.69:54288) 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 -- 172.21.15.69:0/2461835435 learned_addr learned my addr 172.21.15.69:0/2461835435 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608250 0x5653f06123d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 -- 172.21.15.69:0/2461835435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608250 msgr2=0x5653f06123d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608250 0x5653f06123d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 -- 172.21.15.69:0/2461835435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 msgr2=0x5653f0619950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 0x5653f0619950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 -- 172.21.15.69:0/2461835435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653f0280210 con 0x5653f0608c40 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 0x5653f0619950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f158bfff640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f157c000c00 tx=0x7f157c005e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=85257b9fab1aaeb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f156b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.534+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c035040 con 0x5653f0608c40 2024-05-09T09:26:06.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1580005500 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f157c0319b0 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f157c04f050 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 10 client.25940 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5653f034c620 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1580005500 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f157c004780 con 0x5653f0608c40 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 10 client.25940 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 10 client.25940 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 20 client.25940 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f061deb0 0x5653f06202c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 4 client.25940 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5653f0579a70 con 0x5653f061deb0 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.535+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f061deb0 0x5653f06202c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.536+0000 7f156b7fe640 10 client.25940 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.536+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f061deb0 0x5653f06202c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=feafe9f0a393ca0c server_cookie=e4237c361ad545c2 in_seq=0 out_seq=0 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f156b7fe640 1 -- 172.21.15.69:0/2461835435 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5653f0579a70 con 0x5653f061deb0 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f156b7fe640 10 client.25940 handle_command_reply: tid=0 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 client.25940 shutdown 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 msgr2=0x5653f0612910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f157c000c00 tx=0x7f157c005e70 comp rx=0 tx=0).stop 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 shutdown_connections 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f060ddb0 0x5653f0619950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0612910 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f061deb0 0x5653f06202c0 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 --2- 172.21.15.69:0/2461835435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608250 0x5653f06123d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f156b7fe640 0 client.25940 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 >> 172.21.15.69:0/2461835435 conn(0x5653f06038c0 msgr2=0x5653f0602a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 shutdown_connections 2024-05-09T09:26:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 1 -- 172.21.15.69:0/2461835435 wait complete. 2024-05-09T09:26:06.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.537+0000 7f1595026740 20 client.25940 trim_cache size 0 max 16384 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 Processor -- start 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 -- start start 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0619910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f060ddb0 0x5653f0609e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1acc50 con 0x5653f0608250 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1aa520 con 0x5653f0608c40 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f1595026740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1a8370 con 0x5653f060ddb0 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38848/0 (socket says 172.21.15.69:38848) 2024-05-09T09:26:06.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 -- 172.21.15.69:0/3600959273 learned_addr learned my addr 172.21.15.69:0/3600959273 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158bfff640 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0619910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f060ddb0 0x5653f0609e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 -- 172.21.15.69:0/3600959273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f060ddb0 msgr2=0x5653f0609e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f060ddb0 0x5653f0609e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 -- 172.21.15.69:0/3600959273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 msgr2=0x5653f0619910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0619910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 -- 172.21.15.69:0/3600959273 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5653f0620940 con 0x5653f0608250 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.552+0000 7f158b7fe640 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1580013810 tx=0x7f1580009950 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a72a38daed4f38f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1571ffb640 1 -- 172.21.15.69:0/3600959273 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15800055d0 con 0x5653f0608250 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1571ffb640 1 -- 172.21.15.69:0/3600959273 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1580031830 con 0x5653f0608250 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1571ffb640 1 -- 172.21.15.69:0/3600959273 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1580004a70 con 0x5653f0608250 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- 172.21.15.69:0/3600959273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 msgr2=0x5653f060d450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7f1580013810 tx=0x7f1580009950 comp rx=0 tx=0).stop 2024-05-09T09:26:06.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- 172.21.15.69:0/3600959273 shutdown_connections 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f060ddb0 0x5653f0609e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f0608c40 0x5653f0619910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 --2- 172.21.15.69:0/3600959273 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f060d450 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- 172.21.15.69:0/3600959273 >> 172.21.15.69:0/3600959273 conn(0x5653f06038c0 msgr2=0x5653f0603ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- 172.21.15.69:0/3600959273 shutdown_connections 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- 172.21.15.69:0/3600959273 wait complete. 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.553+0000 7f1595026740 1 -- start start 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 Processor -- start 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f0616c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0682430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1a89e0 con 0x5653f0608250 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1a9e50 con 0x5653f060ddb0 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f158c1a90b0 con 0x5653f0608c40 2024-05-09T09:26:06.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f1595026740 10 client.0 _ll_register_callbacks cb 0x7f157c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38858/0 (socket says 172.21.15.69:38858) 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 -- 172.21.15.69:0/2952143783 learned_addr learned my addr 172.21.15.69:0/2952143783 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158bfff640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f0616c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0682430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 -- 172.21.15.69:0/2952143783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 msgr2=0x5653f0616c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f0616c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 -- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 msgr2=0x5653f0682430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0682430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.554+0000 7f158b7fe640 1 -- 172.21.15.69:0/2952143783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5653f06172e0 con 0x5653f0608250 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f158b7fe640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1580035040 tx=0x7f1580005b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=57b8c971ea80f188 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:06.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f15800319e0 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f1580031b80 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1580031830 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f1580042050 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 10 client.16629 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5653f0683a80 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1580031830 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f1580022450 con 0x5653f0608250 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 10 client.16629 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 10 client.16629 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 20 client.16629 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f06842d0 0x5653f06866f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 4 client.16629 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5653f0573dc0 con 0x5653f06842d0 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.555+0000 7f158bfff640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f06842d0 0x5653f06866f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:06.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.556+0000 7f156b7fe640 10 client.16629 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:06.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.556+0000 7f158bfff640 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f06842d0 0x5653f06866f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b2ad6e07e572aac8 server_cookie=46c2955d06af1310 in_seq=0 out_seq=0 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.659+0000 7f156b7fe640 1 -- 172.21.15.69:0/2952143783 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5653f0573dc0 con 0x5653f06842d0 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.659+0000 7f156b7fe640 10 client.16629 handle_command_reply: tid=0 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.659+0000 7f1595026740 1 client.16629 shutdown 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 msgr2=0x5653f0616730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7f1580035040 tx=0x7f1580005b90 comp rx=0 tx=0).stop 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 shutdown_connections 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5653f060ddb0 0x5653f0682430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5653f06842d0 0x5653f06866f0 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5653f0608c40 0x5653f0616c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 --2- 172.21.15.69:0/2952143783 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5653f0608250 0x5653f0616730 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f156b7fe640 0 client.16629 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 >> 172.21.15.69:0/2952143783 conn(0x5653f06038c0 msgr2=0x5653f0609e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 shutdown_connections 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 1 -- 172.21.15.69:0/2952143783 wait complete. 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1595026740 20 client.16629 trim_cache size 0 max 16384 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:26:06.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:26:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 msgr2=0x7f15740a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 0x7f15740a50f0 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f1584098910 tx=0x7f15840c6ad0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 msgr2=0x7f158c19f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.660+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7f158400eb10 tx=0x7f158400c2d0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 shutdown_connections 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f15740a2c30 0x7f15740a50f0 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f158c103440 0x7f158c19f0a0 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f158c102a80 0x7f158c198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 --2- 172.21.15.69:0/4069185990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f158c101890 0x7f158c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 >> 172.21.15.69:0/4069185990 conn(0x7f158c0fce20 msgr2=0x7f158c105d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 shutdown_connections 2024-05-09T09:26:06.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:06.661+0000 7f1592314640 1 -- 172.21.15.69:0/4069185990 wait complete. 2024-05-09T09:26:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:06 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:26:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:06 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:26:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:06 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:26:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:06 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:26:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:06 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:26:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:06 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:26:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:06 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:26:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:06 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:26:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:06 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:26:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:06 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:26:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:06 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:26:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:06 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:26:07.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 366448 6.48 MB/sec execute 245 sec latency 97.534 ms 2024-05-09T09:26:08.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:07 smithi069 ceph-mon[30123]: pgmap v586: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-05-09T09:26:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:07 smithi163 ceph-mon[34354]: pgmap v586: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-05-09T09:26:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:07 smithi103 ceph-mon[34421]: pgmap v586: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.0 MiB/s wr, 365 op/s 2024-05-09T09:26:08.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 369106 6.53 MB/sec execute 246 sec latency 13.414 ms 2024-05-09T09:26:09.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 370191 6.51 MB/sec execute 247 sec latency 89.919 ms 2024-05-09T09:26:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:09 smithi103 ceph-mon[34421]: pgmap v587: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 381 op/s 2024-05-09T09:26:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:09 smithi069 ceph-mon[30123]: pgmap v587: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 381 op/s 2024-05-09T09:26:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:09 smithi163 ceph-mon[34354]: pgmap v587: 129 pgs: 129 active+clean; 87 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.2 MiB/s wr, 381 op/s 2024-05-09T09:26:10.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 372803 6.55 MB/sec execute 248 sec latency 20.111 ms 2024-05-09T09:26:11.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 374899 6.56 MB/sec execute 249 sec latency 17.916 ms 2024-05-09T09:26:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:11 smithi103 ceph-mon[34421]: pgmap v588: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-09T09:26:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:11 smithi069 ceph-mon[30123]: pgmap v588: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-09T09:26:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:11 smithi163 ceph-mon[34354]: pgmap v588: 129 pgs: 129 active+clean; 88 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 975 KiB/s rd, 2.6 MiB/s wr, 300 op/s 2024-05-09T09:26:12.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 376729 6.58 MB/sec execute 250 sec latency 12.612 ms 2024-05-09T09:26:13.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 379551 6.62 MB/sec execute 251 sec latency 12.448 ms 2024-05-09T09:26:13.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:13 smithi163 ceph-mon[34354]: pgmap v589: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-05-09T09:26:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:13 smithi103 ceph-mon[34421]: pgmap v589: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-05-09T09:26:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:13 smithi069 ceph-mon[30123]: pgmap v589: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.3 MiB/s wr, 382 op/s 2024-05-09T09:26:14.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 380543 6.60 MB/sec execute 252 sec latency 64.507 ms 2024-05-09T09:26:15.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 383345 6.65 MB/sec execute 253 sec latency 30.082 ms 2024-05-09T09:26:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:15 smithi103 ceph-mon[34421]: pgmap v590: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 353 op/s 2024-05-09T09:26:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:15 smithi069 ceph-mon[30123]: pgmap v590: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 353 op/s 2024-05-09T09:26:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:15 smithi163 ceph-mon[34354]: pgmap v590: 129 pgs: 129 active+clean; 91 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.9 MiB/s wr, 353 op/s 2024-05-09T09:26:16.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 384790 6.64 MB/sec execute 254 sec latency 16.809 ms 2024-05-09T09:26:17.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 387096 6.67 MB/sec execute 255 sec latency 13.668 ms 2024-05-09T09:26:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:17 smithi103 ceph-mon[34421]: pgmap v591: 129 pgs: 129 active+clean; 92 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-05-09T09:26:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:17 smithi069 ceph-mon[30123]: pgmap v591: 129 pgs: 129 active+clean; 92 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-05-09T09:26:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:17 smithi163 ceph-mon[34354]: pgmap v591: 129 pgs: 129 active+clean; 92 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 352 op/s 2024-05-09T09:26:18.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 389234 6.68 MB/sec execute 256 sec latency 23.051 ms 2024-05-09T09:26:19.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 390881 6.70 MB/sec execute 257 sec latency 75.301 ms 2024-05-09T09:26:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:19 smithi103 ceph-mon[34421]: pgmap v592: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 332 op/s 2024-05-09T09:26:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:19 smithi069 ceph-mon[30123]: pgmap v592: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 332 op/s 2024-05-09T09:26:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 -- 172.21.15.69:0/3294041616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 msgr2=0x7f9b58108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.774 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/3294041616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b58108bd0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7f9b50009900 tx=0x7f9b5002a410 comp rx=0 tx=0).stop 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 -- 172.21.15.69:0/3294041616 shutdown_connections 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/3294041616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b58108bd0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/3294041616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 0x7f9b58101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/3294041616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 -- 172.21.15.69:0/3294041616 >> 172.21.15.69:0/3294041616 conn(0x7f9b580fbd00 msgr2=0x7f9b580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 -- 172.21.15.69:0/3294041616 shutdown_connections 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.772+0000 7f9b5fda5640 1 -- 172.21.15.69:0/3294041616 wait complete. 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 Processor -- start 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 -- start start 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 0x7f9b5819c7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b5819ee10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.775 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5810c440 con 0x7f9b58102340 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5810c2c0 con 0x7f9b58100790 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5fda5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b5810c5c0 con 0x7f9b58101980 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5e342640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b5819ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60602/0 (socket says 172.21.15.69:60602) 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 -- 172.21.15.69:0/1200672348 learned_addr learned my addr 172.21.15.69:0/1200672348 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5d340640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 0x7f9b5819c7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 -- 172.21.15.69:0/1200672348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 msgr2=0x7f9b5819c7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 0x7f9b5819c7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 -- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 msgr2=0x7f9b5819ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b5819ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5db41640 1 -- 172.21.15.69:0/1200672348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9b5819f720 con 0x7f9b58100790 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.773+0000 7f9b5e342640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b5819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:19.776 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b5db41640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009d20 tx=0x7f9b5400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b4effd640 1 -- 172.21.15.69:0/1200672348 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54010630 con 0x7f9b58100790 2024-05-09T09:26:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9b5819f9b0 con 0x7f9b58100790 2024-05-09T09:26:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b4effd640 1 -- 172.21.15.69:0/1200672348 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b54010cb0 con 0x7f9b58100790 2024-05-09T09:26:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b4effd640 1 -- 172.21.15.69:0/1200672348 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54018de0 con 0x7f9b58100790 2024-05-09T09:26:19.777 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.774+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9b5819feb0 con 0x7f9b58100790 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.776+0000 7f9b4effd640 1 -- 172.21.15.69:0/1200672348 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9b5401f020 con 0x7f9b58100790 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.776+0000 7f9b62b8d740 1 Processor -- start 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 -- start start 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c85a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 0x55dde5c8ab50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dde5be5340 con 0x55dde5c8b1c0 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dde5b8d290 con 0x55dde5c86050 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dde5be1300 con 0x55dde5c85660 2024-05-09T09:26:19.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 0x55dde5c8ab50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b4effd640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 0x7f9b440a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48474/0 (socket says 172.21.15.69:48474) 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 -- 172.21.15.69:0/4097468641 learned_addr learned my addr 172.21.15.69:0/4097468641 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5e342640 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c85a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 -- 172.21.15.69:0/4097468641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 msgr2=0x55dde5c85a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c85a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 -- 172.21.15.69:0/4097468641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 msgr2=0x55dde5c8ab50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.777+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 0x55dde5c8ab50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b5e342640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 0x7f9b440a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b5d340640 1 -- 172.21.15.69:0/4097468641 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde5bb18e0 con 0x55dde5c8b1c0 2024-05-09T09:26:19.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b5db41640 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 0x55dde5c8ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b4effd640 1 -- 172.21.15.69:0/1200672348 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f9b540cf840 con 0x7f9b58100790 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009c90 tx=0x7f9b48011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65d0f3975b11b4d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b43fff640 1 -- 172.21.15.69:0/4097468641 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4801e070 con 0x55dde5c8b1c0 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b43fff640 1 -- 172.21.15.69:0/4097468641 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b48019b10 con 0x55dde5c8b1c0 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.778+0000 7f9b43fff640 1 -- 172.21.15.69:0/4097468641 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4800eac0 con 0x55dde5c8b1c0 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4097468641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 msgr2=0x55dde5c8d650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b5e342640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 0x7f9b440a50f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b50004770 tx=0x7f9b5002a410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7f9b48009c90 tx=0x7f9b48011810 comp rx=0 tx=0).stop 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4097468641 shutdown_connections 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c8d650 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c86050 0x55dde5c8ab50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4097468641 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c85a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4097468641 >> 172.21.15.69:0/4097468641 conn(0x55dde5c7c8c0 msgr2=0x55dde5c7ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4097468641 shutdown_connections 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4097468641 wait complete. 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 -- start start 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.779+0000 7f9b62b8d740 1 Processor -- start 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c86050 0x55dde5c8fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c96d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dde5c8e0e0 con 0x55dde5c8b1c0 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dde5ba0790 con 0x55dde5c85660 2024-05-09T09:26:19.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dde5c8e260 con 0x55dde5c86050 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 10 client.0 _ll_register_callbacks cb 0x55dde5bdba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60612/0 (socket says 172.21.15.69:60612) 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 -- 172.21.15.69:0/2063877373 learned_addr learned my addr 172.21.15.69:0/2063877373 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5e342640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c96d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5d340640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c86050 0x55dde5c8fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 -- 172.21.15.69:0/2063877373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c86050 msgr2=0x55dde5c8fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c86050 0x55dde5c8fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 -- 172.21.15.69:0/2063877373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 msgr2=0x55dde5c96d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c96d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 -- 172.21.15.69:0/2063877373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde58f9210 con 0x55dde5c85660 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400cba0 tx=0x7f9b54009900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b343d29786637a4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b417fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:26:19.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54010e60 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f9b54098390 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.780+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b48019b10 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9b5409b680 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 10 client.25958 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dde59c5620 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b48019b10 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f9b54097a70 con 0x55dde5c85660 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 10 client.25958 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 10 client.25958 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 20 client.25958 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5c9b2c0 0x55dde5c9d710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 4 client.25958 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dde5bf2a70 con 0x55dde5c9b2c0 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.781+0000 7f9b5d340640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5c9b2c0 0x55dde5c9d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.782+0000 7f9b5d340640 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5c9b2c0 0x55dde5c9d710 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c6bfbde991b6146 server_cookie=13aceb4d2c64c117 in_seq=0 out_seq=0 2024-05-09T09:26:19.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.782+0000 7f9b417fa640 10 client.25958 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.783+0000 7f9b417fa640 1 -- 172.21.15.69:0/2063877373 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dde5bf2a70 con 0x55dde5c9b2c0 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.783+0000 7f9b417fa640 10 client.25958 handle_command_reply: tid=0 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.783+0000 7f9b62b8d740 1 client.25958 shutdown 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.783+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 msgr2=0x55dde5c8f850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.783+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f9b5400cba0 tx=0x7f9b54009900 comp rx=0 tx=0).stop 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 shutdown_connections 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c8b1c0 0x55dde5c96d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5c9b2c0 0x55dde5c9d710 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b417fa640 0 client.25958 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c86050 0x55dde5c8fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2063877373 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c85660 0x55dde5c8f850 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 >> 172.21.15.69:0/2063877373 conn(0x55dde5c7c8c0 msgr2=0x55dde5c7ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 shutdown_connections 2024-05-09T09:26:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2063877373 wait complete. 2024-05-09T09:26:19.790 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.784+0000 7f9b62b8d740 20 client.25958 trim_cache size 0 max 16384 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.799+0000 7f9b62b8d740 1 Processor -- start 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.799+0000 7f9b62b8d740 1 -- start start 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c8a8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c96d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c86740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581aaac0 con 0x55dde5c86050 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581a8390 con 0x55dde5c8b1c0 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581a61e0 con 0x55dde5c85660 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c96d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c8a8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c96d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48490/0 (socket says 172.21.15.69:48490) 2024-05-09T09:26:19.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5d340640 1 -- 172.21.15.69:0/2281967543 learned_addr learned my addr 172.21.15.69:0/2281967543 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5e342640 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c86740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 -- 172.21.15.69:0/2281967543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 msgr2=0x55dde5c86740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c86740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 -- 172.21.15.69:0/2281967543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 msgr2=0x55dde5c96d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c96d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 -- 172.21.15.69:0/2281967543 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dde5c7af00 con 0x55dde5c85660 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b5db41640 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c8a8a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b540188c0 tx=0x7f9b5409a230 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c65593c41f5d5e96 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.800+0000 7f9b437fe640 1 -- 172.21.15.69:0/2281967543 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b54010e60 con 0x55dde5c85660 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b437fe640 1 -- 172.21.15.69:0/2281967543 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b5409c420 con 0x55dde5c85660 2024-05-09T09:26:19.802 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b437fe640 1 -- 172.21.15.69:0/2281967543 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b540a00d0 con 0x55dde5c85660 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2281967543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 msgr2=0x55dde5c8a8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c8a8a0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9b540188c0 tx=0x7f9b5409a230 comp rx=0 tx=0).stop 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2281967543 shutdown_connections 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c86740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c96d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/2281967543 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c8a8a0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2281967543 >> 172.21.15.69:0/2281967543 conn(0x55dde5c7c8c0 msgr2=0x55dde5c7cca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2281967543 shutdown_connections 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- 172.21.15.69:0/2281967543 wait complete. 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.801+0000 7f9b62b8d740 1 -- start start 2024-05-09T09:26:19.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 Processor -- start 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c94380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c95710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581a6850 con 0x55dde5c86050 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581a7cc0 con 0x55dde5c8b1c0 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9b581a6f20 con 0x55dde5c85660 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b62b8d740 10 client.0 _ll_register_callbacks cb 0x7f9b54004a40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48496/0 (socket says 172.21.15.69:48496) 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 -- 172.21.15.69:0/4270590852 learned_addr learned my addr 172.21.15.69:0/4270590852 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5e342640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c95710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5db41640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c94380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 -- 172.21.15.69:0/4270590852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 msgr2=0x55dde5c94380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.807 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c94380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 -- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 msgr2=0x55dde5c95710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c95710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5d340640 1 -- 172.21.15.69:0/4270590852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dde5c95d00 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.802+0000 7f9b5e342640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c95710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b5d340640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x55dde5c7d830 tx=0x7f9b48009be0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d959fcf9e0a0f50 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9b4801e040 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f9b48005240 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9b5409c420 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f9b48039050 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 10 client.16647 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55dde5c970b0 con 0x55dde5c86050 2024-05-09T09:26:19.808 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9b5409c420 con 0x55dde5c86050 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f9b4802e020 con 0x55dde5c86050 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 10 client.16647 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 10 client.16647 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 20 client.16647 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5d01e60 0x55dde5d04280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 4 client.16647 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dde5becdc0 con 0x55dde5d01e60 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.803+0000 7f9b5db41640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5d01e60 0x55dde5d04280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.804+0000 7f9b417fa640 10 client.16647 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:19.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.804+0000 7f9b5db41640 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5d01e60 0x55dde5d04280 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca8c1649bbe96dff server_cookie=2eeff2f45ab00b9f in_seq=0 out_seq=0 2024-05-09T09:26:19.809 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:19 smithi163 ceph-mon[34354]: pgmap v592: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 332 op/s 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b417fa640 1 -- 172.21.15.69:0/4270590852 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dde5becdc0 con 0x55dde5d01e60 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b417fa640 10 client.16647 handle_command_reply: tid=0 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 client.16647 shutdown 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 msgr2=0x55dde5c948c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x55dde5c7d830 tx=0x7f9b48009be0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 shutdown_connections 2024-05-09T09:26:19.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dde5c8b1c0 0x55dde5c95710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dde5c86050 0x55dde5c948c0 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dde5d01e60 0x55dde5d04280 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 --2- 172.21.15.69:0/4270590852 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dde5c85660 0x55dde5c94380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b417fa640 0 client.16647 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 >> 172.21.15.69:0/4270590852 conn(0x55dde5c7c8c0 msgr2=0x55dde5c86740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 shutdown_connections 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 1 -- 172.21.15.69:0/4270590852 wait complete. 2024-05-09T09:26:19.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.927+0000 7f9b62b8d740 20 client.16647 trim_cache size 0 max 16384 2024-05-09T09:26:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:26:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:26:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:26:19.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 msgr2=0x7f9b440a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 0x7f9b440a50f0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f9b50004770 tx=0x7f9b5002a410 comp rx=0 tx=0).stop 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 msgr2=0x7f9b58069a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f9b54009d20 tx=0x7f9b5400c6a0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 shutdown_connections 2024-05-09T09:26:19.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9b440a2c30 0x7f9b440a50f0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9b58102340 0x7f9b5819ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9b58101980 0x7f9b5819c7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 --2- 172.21.15.69:0/1200672348 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9b58100790 0x7f9b58069a50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 >> 172.21.15.69:0/1200672348 conn(0x7f9b580fbd00 msgr2=0x7f9b58104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 shutdown_connections 2024-05-09T09:26:19.932 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:19.928+0000 7f9b5fda5640 1 -- 172.21.15.69:0/1200672348 wait complete. 2024-05-09T09:26:20.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 393001 6.71 MB/sec execute 258 sec latency 49.282 ms 2024-05-09T09:26:21.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 394567 6.72 MB/sec execute 259 sec latency 14.715 ms 2024-05-09T09:26:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:21 smithi103 ceph-mon[34421]: pgmap v593: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 273 op/s 2024-05-09T09:26:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:21 smithi069 ceph-mon[30123]: pgmap v593: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 273 op/s 2024-05-09T09:26:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:21 smithi163 ceph-mon[34354]: pgmap v593: 129 pgs: 129 active+clean; 94 MiB data, 1.4 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.6 MiB/s wr, 273 op/s 2024-05-09T09:26:22.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 397151 6.75 MB/sec execute 260 sec latency 11.585 ms 2024-05-09T09:26:23.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 398451 6.75 MB/sec execute 261 sec latency 22.228 ms 2024-05-09T09:26:23.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:23 smithi103 ceph-mon[34421]: pgmap v594: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-05-09T09:26:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:23 smithi069 ceph-mon[30123]: pgmap v594: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-05-09T09:26:23.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:23 smithi163 ceph-mon[34354]: pgmap v594: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 368 op/s 2024-05-09T09:26:24.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 401078 6.79 MB/sec execute 262 sec latency 69.290 ms 2024-05-09T09:26:25.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 402243 6.78 MB/sec execute 263 sec latency 58.604 ms 2024-05-09T09:26:25.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:25 smithi103 ceph-mon[34421]: pgmap v595: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2024-05-09T09:26:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:25 smithi069 ceph-mon[30123]: pgmap v595: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2024-05-09T09:26:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:25 smithi163 ceph-mon[34354]: pgmap v595: 129 pgs: 129 active+clean; 92 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.5 MiB/s wr, 348 op/s 2024-05-09T09:26:26.493 INFO:tasks.workunit.client.0.smithi069.stdout: 1 404712 6.81 MB/sec execute 264 sec latency 16.748 ms 2024-05-09T09:26:27.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 406246 6.80 MB/sec execute 265 sec latency 16.421 ms 2024-05-09T09:26:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:27 smithi103 ceph-mon[34421]: pgmap v596: 129 pgs: 129 active+clean; 94 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-05-09T09:26:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:27 smithi069 ceph-mon[30123]: pgmap v596: 129 pgs: 129 active+clean; 94 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-05-09T09:26:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:27 smithi163 ceph-mon[34354]: pgmap v596: 129 pgs: 129 active+clean; 94 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 359 op/s 2024-05-09T09:26:28.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 408402 6.83 MB/sec execute 266 sec latency 18.407 ms 2024-05-09T09:26:29.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 410730 6.84 MB/sec execute 267 sec latency 15.303 ms 2024-05-09T09:26:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:29 smithi103 ceph-mon[34421]: pgmap v597: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-05-09T09:26:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:29 smithi069 ceph-mon[30123]: pgmap v597: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-05-09T09:26:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:29 smithi163 ceph-mon[34354]: pgmap v597: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.8 MiB/s wr, 378 op/s 2024-05-09T09:26:30.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 412084 6.85 MB/sec execute 268 sec latency 48.096 ms 2024-05-09T09:26:31.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 414474 6.87 MB/sec execute 269 sec latency 22.874 ms 2024-05-09T09:26:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:31 smithi103 ceph-mon[34421]: pgmap v598: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 316 op/s 2024-05-09T09:26:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:31 smithi069 ceph-mon[30123]: pgmap v598: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 316 op/s 2024-05-09T09:26:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:31 smithi163 ceph-mon[34354]: pgmap v598: 129 pgs: 129 active+clean; 95 MiB data, 1.5 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.3 MiB/s wr, 316 op/s 2024-05-09T09:26:32.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 415795 6.87 MB/sec execute 270 sec latency 15.786 ms 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 -- 172.21.15.69:0/3914822638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101019b0 msgr2=0x7f5b10101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 --2- 172.21.15.69:0/3914822638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101019b0 0x7f5b10101e30 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7f5b00009900 tx=0x7f5b0002a410 comp rx=0 tx=0).stop 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 -- 172.21.15.69:0/3914822638 shutdown_connections 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 --2- 172.21.15.69:0/3914822638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b10108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 --2- 172.21.15.69:0/3914822638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101019b0 0x7f5b10101e30 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 --2- 172.21.15.69:0/3914822638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101007c0 0x7f5b10100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 -- 172.21.15.69:0/3914822638 >> 172.21.15.69:0/3914822638 conn(0x7f5b100fbd30 msgr2=0x7f5b100fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 -- 172.21.15.69:0/3914822638 shutdown_connections 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.019+0000 7f5b174a5640 1 -- 172.21.15.69:0/3914822638 wait complete. 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 Processor -- start 2024-05-09T09:26:33.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 -- start start 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 0x7f5b10198080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b1019f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1010c440 con 0x7f5b101007c0 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1010c2c0 con 0x7f5b10102370 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b174a5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5b1010c5c0 con 0x7f5b101019b0 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b14a40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 0x7f5b10198080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52862/0 (socket says 172.21.15.69:52862) 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b15241640 1 -- 172.21.15.69:0/882060412 learned_addr learned my addr 172.21.15.69:0/882060412 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.020+0000 7f5b15a42640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b1019f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 -- 172.21.15.69:0/882060412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 msgr2=0x7f5b10198080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 0x7f5b10198080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 -- 172.21.15.69:0/882060412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 msgr2=0x7f5b1019f0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b1019f0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 -- 172.21.15.69:0/882060412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5b10102c50 con 0x7f5b101007c0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15a42640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b1019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b14a40640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 0x7f5b10198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b15241640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9c0 tx=0x7f5b0c00edc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b067fc640 1 -- 172.21.15.69:0/882060412 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c00ccc0 con 0x7f5b101007c0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5b10102f40 con 0x7f5b101007c0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b067fc640 1 -- 172.21.15.69:0/882060412 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0c0035f0 con 0x7f5b101007c0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b067fc640 1 -- 172.21.15.69:0/882060412 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c00ce60 con 0x7f5b101007c0 2024-05-09T09:26:33.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.021+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5b10103460 con 0x7f5b101007c0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.024+0000 7f5b067fc640 1 -- 172.21.15.69:0/882060412 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5b0c0037a0 con 0x7f5b101007c0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 Processor -- start 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 -- start start 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 0x562f15fc0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fc5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562f15f24340 con 0x562f15fc5db0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562f15ecc290 con 0x562f15fc0250 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562f15f20300 con 0x562f15fc0c40 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b15241640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fc5740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b15a42640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 0x562f15fc0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52864/0 (socket says 172.21.15.69:52864) 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 -- 172.21.15.69:0/2402126572 learned_addr learned my addr 172.21.15.69:0/2402126572 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b067fc640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 0x7f5afc0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b15a42640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 0x7f5afc0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 -- 172.21.15.69:0/2402126572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 msgr2=0x562f15fc5740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fc5740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 -- 172.21.15.69:0/2402126572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 msgr2=0x562f15fc0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 0x562f15fc0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b14a40640 1 -- 172.21.15.69:0/2402126572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f15ef08e0 con 0x562f15fc5db0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.025+0000 7f5b15a42640 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 0x562f15fc0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5b067fc640 1 -- 172.21.15.69:0/882060412 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f5b0c0cf720 con 0x7f5b101007c0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5b14a40640 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002a930 tx=0x7f5b00009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8498f9041ba75ab6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5b15a42640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 0x7f5afc0a50f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x562f15fc14a0 tx=0x7f5b08004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5afb7fe640 1 -- 172.21.15.69:0/2402126572 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00040070 con 0x562f15fc5db0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5afb7fe640 1 -- 172.21.15.69:0/2402126572 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b000319b0 con 0x562f15fc5db0 2024-05-09T09:26:33.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.026+0000 7f5afb7fe640 1 -- 172.21.15.69:0/2402126572 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b00035040 con 0x562f15fc5db0 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/2402126572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 msgr2=0x562f15fc8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f5b0002a930 tx=0x7f5b00009680 comp rx=0 tx=0).stop 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/2402126572 shutdown_connections 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc5db0 0x562f15fc8240 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fc5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/2402126572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0250 0x562f15fc0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/2402126572 >> 172.21.15.69:0/2402126572 conn(0x562f15fbb8c0 msgr2=0x562f15fbbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/2402126572 shutdown_connections 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/2402126572 wait complete. 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- start start 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 Processor -- start 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fca970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd19b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562f15fc8d00 con 0x562f15fc0250 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562f15edf790 con 0x562f15fc0c40 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.027+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562f15fc8e80 con 0x562f15fc5db0 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b1a1bc740 10 client.0 _ll_register_callbacks cb 0x562f15f1aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b14a40640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fca970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15a42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd19b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52866/0 (socket says 172.21.15.69:52866) 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 -- 172.21.15.69:0/214906052 learned_addr learned my addr 172.21.15.69:0/214906052 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 -- 172.21.15.69:0/214906052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 msgr2=0x562f15fd19b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd19b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 -- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 msgr2=0x562f15fca970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fca970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 -- 172.21.15.69:0/214906052 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f15c38210 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b15241640 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c004630 tx=0x7f5b0c09b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f636ee73c7ea1839 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5af8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c09bc00 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f5b0c0cfbd0 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b000319b0 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5b0c09ffb0 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b1a1bc740 10 client.16659 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f15d04620 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.028+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b000319b0 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f5b0c09b0e0 con 0x562f15fc0250 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 10 client.16659 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 10 client.16659 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 20 client.16659 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:33.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f15fd5fd0 0x562f15fd8420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 4 client.16659 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562f15f31a70 con 0x562f15fd5fd0 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b14a40640 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f15fd5fd0 0x562f15fd8420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5b14a40640 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f15fd5fd0 0x562f15fd8420 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0c88120140b655d server_cookie=2197b4c057592541 in_seq=0 out_seq=0 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.029+0000 7f5af8ff9640 10 client.16659 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5af8ff9640 1 -- 172.21.15.69:0/214906052 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562f15f31a70 con 0x562f15fd5fd0 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5af8ff9640 10 client.16659 handle_command_reply: tid=0 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 client.16659 shutdown 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 msgr2=0x562f15fca430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c004630 tx=0x7f5b0c09b800 comp rx=0 tx=0).stop 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 shutdown_connections 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f15fd5fd0 0x562f15fd8420 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd19b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fca970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/214906052 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fca430 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 >> 172.21.15.69:0/214906052 conn(0x562f15fbb8c0 msgr2=0x562f15fbaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5af8ff9640 0 client.16659 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 shutdown_connections 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/214906052 wait complete. 2024-05-09T09:26:33.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.030+0000 7f5b1a1bc740 20 client.16659 trim_cache size 0 max 16384 2024-05-09T09:26:33.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 Processor -- start 2024-05-09T09:26:33.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 -- start start 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fde270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd1970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101aed90 con 0x562f15fc0250 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ac660 con 0x562f15fc0c40 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101aa4b0 con 0x562f15fc5db0 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52870/0 (socket says 172.21.15.69:52870) 2024-05-09T09:26:33.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15241640 1 -- 172.21.15.69:0/3072174138 learned_addr learned my addr 172.21.15.69:0/3072174138 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b14a40640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fde270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15a42640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15241640 1 -- 172.21.15.69:0/3072174138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 msgr2=0x562f15fd1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.045+0000 7f5b15241640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b15241640 1 -- 172.21.15.69:0/3072174138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 msgr2=0x562f15fde270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b15241640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fde270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b15241640 1 -- 172.21.15.69:0/3072174138 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562f15fc1020 con 0x562f15fc0250 2024-05-09T09:26:33.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b15a42640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b15241640 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c09bb90 tx=0x7f5b0c003f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b7dbc6a89c6058 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5afaffd640 1 -- 172.21.15.69:0/3072174138 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c0b2030 con 0x562f15fc0250 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5afaffd640 1 -- 172.21.15.69:0/3072174138 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0c0d6040 con 0x562f15fc0250 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5afaffd640 1 -- 172.21.15.69:0/3072174138 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c09b050 con 0x562f15fc0250 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/3072174138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 msgr2=0x562f15fddd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c09bb90 tx=0x7f5b0c003f10 comp rx=0 tx=0).stop 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/3072174138 shutdown_connections 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc5db0 0x562f15fd1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc0c40 0x562f15fde270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/3072174138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fddd30 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/3072174138 >> 172.21.15.69:0/3072174138 conn(0x562f15fbb8c0 msgr2=0x562f15fbbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.046+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/3072174138 shutdown_connections 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/3072174138 wait complete. 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 -- start start 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 Processor -- start 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fce9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc5db0 0x562f1603a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101aab20 con 0x562f15fc0250 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101abf90 con 0x562f15fc5db0 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5b101ab1f0 con 0x562f15fc0c40 2024-05-09T09:26:33.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b1a1bc740 10 client.0 _ll_register_callbacks cb 0x7f5b0c097f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b15241640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52880/0 (socket says 172.21.15.69:52880) 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b15241640 1 -- 172.21.15.69:0/152642986 learned_addr learned my addr 172.21.15.69:0/152642986 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b15a42640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc5db0 0x562f1603a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.047+0000 7f5b14a40640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fce9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 -- 172.21.15.69:0/152642986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 msgr2=0x562f15fce9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fce9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 -- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc5db0 msgr2=0x562f1603a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc5db0 0x562f1603a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 -- 172.21.15.69:0/152642986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562f15fd1a80 con 0x562f15fc0250 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b15241640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0985d0 tx=0x7f5b0c09b800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=24459483971ce104 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:33.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5b0c074020 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f5b0c0cfd90 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5b0c0d6040 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f5b0c0b2070 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 10 client.16665 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x562f1603b970 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5b0c0d6040 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f5b0c0ba4a0 con 0x562f15fc0250 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 10 client.16665 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 10 client.16665 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 20 client.16665 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f1603c070 0x562f1603e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 4 client.16665 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.048+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562f15f2bdc0 con 0x562f1603c070 2024-05-09T09:26:33.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.049+0000 7f5b14a40640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f1603c070 0x562f1603e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.049+0000 7f5af8ff9640 10 client.16665 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:33.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.049+0000 7f5b14a40640 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f1603c070 0x562f1603e460 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=be93972e1d094f9e server_cookie=d662df6490e86cf2 in_seq=0 out_seq=0 2024-05-09T09:26:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5af8ff9640 1 -- 172.21.15.69:0/152642986 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562f15f2bdc0 con 0x562f1603c070 2024-05-09T09:26:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5af8ff9640 10 client.16665 handle_command_reply: tid=0 2024-05-09T09:26:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 client.16665 shutdown 2024-05-09T09:26:33.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 msgr2=0x562f15fce4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c0985d0 tx=0x7f5b0c09b800 comp rx=0 tx=0).stop 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 shutdown_connections 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562f15fc5db0 0x562f1603a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562f1603c070 0x562f1603e460 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562f15fc0c40 0x562f15fce9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 --2- 172.21.15.69:0/152642986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562f15fc0250 0x562f15fce4a0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5af8ff9640 0 client.16665 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.160+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 >> 172.21.15.69:0/152642986 conn(0x562f15fbb8c0 msgr2=0x562f15fd2040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 shutdown_connections 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b1a1bc740 1 -- 172.21.15.69:0/152642986 wait complete. 2024-05-09T09:26:33.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b1a1bc740 20 client.16665 trim_cache size 0 max 16384 2024-05-09T09:26:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:26:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:26:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:26:33.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 msgr2=0x7f5afc0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 0x7f5afc0a50f0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x562f15fc14a0 tx=0x7f5b08004ab0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 msgr2=0x7f5b10197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f5b0c00e9c0 tx=0x7f5b0c00edc0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 shutdown_connections 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5afc0a2c30 0x7f5afc0a50f0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5b10102370 0x7f5b1019f0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5b101019b0 0x7f5b10198080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 --2- 172.21.15.69:0/882060412 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5b101007c0 0x7f5b10197b40 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.161+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 >> 172.21.15.69:0/882060412 conn(0x7f5b100fbd30 msgr2=0x7f5b100fd760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.162+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 shutdown_connections 2024-05-09T09:26:33.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:33.162+0000 7f5b174a5640 1 -- 172.21.15.69:0/882060412 wait complete. 2024-05-09T09:26:33.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 418053 6.89 MB/sec execute 271 sec latency 40.949 ms 2024-05-09T09:26:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:33 smithi103 ceph-mon[34421]: pgmap v599: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-09T09:26:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:33 smithi069 ceph-mon[30123]: pgmap v599: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-09T09:26:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:33 smithi163 ceph-mon[34354]: pgmap v599: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.5 MiB/s rd, 2.8 MiB/s wr, 365 op/s 2024-05-09T09:26:34.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 419374 6.89 MB/sec execute 272 sec latency 56.818 ms 2024-05-09T09:26:35.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 421604 6.91 MB/sec execute 273 sec latency 56.143 ms 2024-05-09T09:26:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:35 smithi103 ceph-mon[34421]: pgmap v600: 129 pgs: 129 active+clean; 92 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 326 op/s 2024-05-09T09:26:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:35 smithi069 ceph-mon[30123]: pgmap v600: 129 pgs: 129 active+clean; 92 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 326 op/s 2024-05-09T09:26:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:35 smithi163 ceph-mon[34354]: pgmap v600: 129 pgs: 129 active+clean; 92 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 326 op/s 2024-05-09T09:26:36.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 422916 6.91 MB/sec execute 274 sec latency 17.473 ms 2024-05-09T09:26:37.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 425341 6.93 MB/sec execute 275 sec latency 11.195 ms 2024-05-09T09:26:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:37 smithi103 ceph-mon[34421]: pgmap v601: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-05-09T09:26:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:37 smithi069 ceph-mon[30123]: pgmap v601: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-05-09T09:26:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:37 smithi163 ceph-mon[34354]: pgmap v601: 129 pgs: 129 active+clean; 93 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 3.0 MiB/s wr, 334 op/s 2024-05-09T09:26:38.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 426576 6.93 MB/sec execute 276 sec latency 22.917 ms 2024-05-09T09:26:39.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 429308 6.97 MB/sec execute 277 sec latency 17.628 ms 2024-05-09T09:26:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:39 smithi103 ceph-mon[34421]: pgmap v602: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-05-09T09:26:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:39 smithi069 ceph-mon[30123]: pgmap v602: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-05-09T09:26:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:39 smithi163 ceph-mon[34354]: pgmap v602: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 352 op/s 2024-05-09T09:26:40.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 430202 6.95 MB/sec execute 278 sec latency 39.903 ms 2024-05-09T09:26:41.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 432887 6.99 MB/sec execute 279 sec latency 17.422 ms 2024-05-09T09:26:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:41 smithi103 ceph-mon[34421]: pgmap v603: 129 pgs: 129 active+clean; 94 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-05-09T09:26:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:41 smithi069 ceph-mon[30123]: pgmap v603: 129 pgs: 129 active+clean; 94 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-05-09T09:26:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:41 smithi163 ceph-mon[34354]: pgmap v603: 129 pgs: 129 active+clean; 94 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 993 KiB/s rd, 2.4 MiB/s wr, 300 op/s 2024-05-09T09:26:42.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 433907 6.97 MB/sec execute 280 sec latency 15.728 ms 2024-05-09T09:26:43.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 436469 7.00 MB/sec execute 281 sec latency 20.289 ms 2024-05-09T09:26:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:43 smithi103 ceph-mon[34421]: pgmap v604: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-05-09T09:26:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:43 smithi069 ceph-mon[30123]: pgmap v604: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-05-09T09:26:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:43 smithi163 ceph-mon[34354]: pgmap v604: 129 pgs: 129 active+clean; 95 MiB data, 1.6 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.0 MiB/s wr, 348 op/s 2024-05-09T09:26:44.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 437951 7.00 MB/sec execute 282 sec latency 27.207 ms 2024-05-09T09:26:45.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 440219 7.03 MB/sec execute 283 sec latency 65.770 ms 2024-05-09T09:26:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:45 smithi103 ceph-mon[34421]: pgmap v605: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:26:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:45 smithi069 ceph-mon[30123]: pgmap v605: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:26:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:45 smithi163 ceph-mon[34354]: pgmap v605: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:26:46.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 -- 172.21.15.69:0/3180503511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081019b0 msgr2=0x7f5008101e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 --2- 172.21.15.69:0/3180503511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081019b0 0x7f5008101e30 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009900 tx=0x7f4ff002a410 comp rx=0 tx=0).stop 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 -- 172.21.15.69:0/3180503511 shutdown_connections 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 --2- 172.21.15.69:0/3180503511 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5008102370 0x7f5008108c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 --2- 172.21.15.69:0/3180503511 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081019b0 0x7f5008101e30 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 --2- 172.21.15.69:0/3180503511 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f50081007c0 0x7f5008100be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 -- 172.21.15.69:0/3180503511 >> 172.21.15.69:0/3180503511 conn(0x7f50080fbd30 msgr2=0x7f50080fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 -- 172.21.15.69:0/3180503511 shutdown_connections 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.251+0000 7f500ce31640 1 -- 172.21.15.69:0/3180503511 wait complete. 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 Processor -- start 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 -- start start 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50081019b0 0x7f50081980e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5008102370 0x7f500819f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f500810c440 con 0x7f50081007c0 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f500810c2c0 con 0x7f50081019b0 2024-05-09T09:26:46.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500ce31640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f500810c5c0 con 0x7f5008102370 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37794/0 (socket says 172.21.15.69:37794) 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 -- 172.21.15.69:0/3946021762 learned_addr learned my addr 172.21.15.69:0/3946021762 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f5005d9b640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50081019b0 0x7f50081980e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f5006d9d640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5008102370 0x7f500819f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 -- 172.21.15.69:0/3946021762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5008102370 msgr2=0x7f500819f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5008102370 0x7f500819f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 -- 172.21.15.69:0/3946021762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50081019b0 msgr2=0x7f50081980e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50081019b0 0x7f50081980e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 -- 172.21.15.69:0/3946021762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f500819f770 con 0x7f50081007c0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.252+0000 7f500659c640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800e9c0 tx=0x7f4ff800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.253+0000 7f4fff7fe640 1 -- 172.21.15.69:0/3946021762 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff800cd80 con 0x7f50081007c0 2024-05-09T09:26:46.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.253+0000 7f4fff7fe640 1 -- 172.21.15.69:0/3946021762 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff80035f0 con 0x7f50081007c0 2024-05-09T09:26:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.253+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f500819fa60 con 0x7f50081007c0 2024-05-09T09:26:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.253+0000 7f4fff7fe640 1 -- 172.21.15.69:0/3946021762 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff8010690 con 0x7f50081007c0 2024-05-09T09:26:46.255 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.253+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f500819ff80 con 0x7f50081007c0 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.255+0000 7f4fff7fe640 1 -- 172.21.15.69:0/3946021762 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4ff80037a0 con 0x7f50081007c0 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.255+0000 7f500fc1c740 1 Processor -- start 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 -- start start 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567e1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567e6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567e9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556d56741340 con 0x556d567e2050 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d566e9290 con 0x556d567e71c0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500fc1c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556d5673d300 con 0x556d567e1660 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f4fff7fe640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 0x7f4ff40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567e6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567e9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567e6b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37798/0 (socket says 172.21.15.69:37798) 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f500659c640 1 -- 172.21.15.69:0/2824920453 learned_addr learned my addr 172.21.15.69:0/2824920453 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5006d9d640 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567e1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5006d9d640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 0x7f4ff40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f4fff7fe640 1 -- 172.21.15.69:0/3946021762 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f4ff80cf6a0 con 0x7f50081007c0 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 -- 172.21.15.69:0/2824920453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 msgr2=0x556d567e1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567e1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 -- 172.21.15.69:0/2824920453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 msgr2=0x556d567e6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567e6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 -- 172.21.15.69:0/2824920453 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d5670d8e0 con 0x556d567e71c0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5006d9d640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 0x7f4ff40a5170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5000009fd0 tx=0x7f50000096a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.256+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567e9650 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0031440 tx=0x7f4ff0032c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49cb6d9f6de7a0e6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f4ffcff9640 1 -- 172.21.15.69:0/2824920453 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0040070 con 0x556d567e71c0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f4ffcff9640 1 -- 172.21.15.69:0/2824920453 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff0032e20 con 0x556d567e71c0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f4ffcff9640 1 -- 172.21.15.69:0/2824920453 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0035040 con 0x556d567e71c0 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 -- 172.21.15.69:0/2824920453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 msgr2=0x556d567e9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567e9650 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0031440 tx=0x7f4ff0032c30 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 -- 172.21.15.69:0/2824920453 shutdown_connections 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567e9650 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567e6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2824920453 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567e1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 -- 172.21.15.69:0/2824920453 >> 172.21.15.69:0/2824920453 conn(0x556d567d88c0 msgr2=0x556d567d8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.257+0000 7f500fc1c740 1 -- 172.21.15.69:0/2824920453 shutdown_connections 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 -- 172.21.15.69:0/2824920453 wait complete. 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 -- start start 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 Processor -- start 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567eb790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567f2d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556d567ea080 con 0x556d567e2050 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d566fc790 con 0x556d567e71c0 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556d567ea200 con 0x556d567e1660 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500fc1c740 10 client.0 _ll_register_callbacks cb 0x556d56737a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37812/0 (socket says 172.21.15.69:37812) 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 -- 172.21.15.69:0/2141342603 learned_addr learned my addr 172.21.15.69:0/2141342603 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5006d9d640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567f2d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f500659c640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567eb790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 -- 172.21.15.69:0/2141342603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 msgr2=0x556d567eb790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567eb790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 -- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 msgr2=0x556d567f2d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567f2d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.258+0000 7f5005d9b640 1 -- 172.21.15.69:0/2141342603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d56455210 con 0x556d567e2050 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f5005d9b640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4ff002a930 tx=0x7f4ff00056b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ee3aa0cadd3e222e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0035040 con 0x556d567e2050 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f4ff0004040 con 0x556d567e2050 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ff0032e20 con 0x556d567e2050 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4ff0057050 con 0x556d567e2050 2024-05-09T09:26:46.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 10 client.16677 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556d56521620 con 0x556d567e2050 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff0032e20 con 0x556d567e2050 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f4ff0004810 con 0x556d567e2050 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 10 client.16677 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 10 client.16677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 20 client.16677 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d567f6e70 0x556d567f92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 4 client.16677 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d5674ea70 con 0x556d567f6e70 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.259+0000 7f500659c640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d567f6e70 0x556d567f92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.260+0000 7f500659c640 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d567f6e70 0x556d567f92b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b942244a65b42d4 server_cookie=ce528cd2e1f0121e in_seq=0 out_seq=0 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.260+0000 7f4fedffb640 10 client.16677 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:46.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.260+0000 7f4fedffb640 1 -- 172.21.15.69:0/2141342603 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d5674ea70 con 0x556d567f6e70 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.260+0000 7f4fedffb640 10 client.16677 handle_command_reply: tid=0 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.260+0000 7f500fc1c740 1 client.16677 shutdown 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 msgr2=0x556d567ebcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7f4ff002a930 tx=0x7f4ff00056b0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 shutdown_connections 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e71c0 0x556d567f2d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567ebcd0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d567f6e70 0x556d567f92b0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 --2- 172.21.15.69:0/2141342603 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e1660 0x556d567eb790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 >> 172.21.15.69:0/2141342603 conn(0x556d567d88c0 msgr2=0x556d567d7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f4fedffb640 0 client.16677 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 shutdown_connections 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 1 -- 172.21.15.69:0/2141342603 wait complete. 2024-05-09T09:26:46.267 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.261+0000 7f500fc1c740 20 client.16677 trim_cache size 0 max 16384 2024-05-09T09:26:46.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.275+0000 7f500fc1c740 1 Processor -- start 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 -- start start 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e2050 0x556d567ff110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f2cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50081aa9f0 con 0x556d567e1660 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50081a82c0 con 0x556d567e2050 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500fc1c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f50081a6110 con 0x556d567e71c0 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37824/0 (socket says 172.21.15.69:37824) 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 -- 172.21.15.69:0/3867037970 learned_addr learned my addr 172.21.15.69:0/3867037970 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f5005d9b640 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e2050 0x556d567ff110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f5006d9d640 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f2cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 -- 172.21.15.69:0/3867037970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 msgr2=0x556d567f2cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f2cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 -- 172.21.15.69:0/3867037970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e2050 msgr2=0x556d567ff110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e2050 0x556d567ff110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 -- 172.21.15.69:0/3867037970 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d567d9630 con 0x556d567e1660 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f500659c640 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4ff809e010 tx=0x7f4ff80cccf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd108e1cbf850c92 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f4feffff640 1 -- 172.21.15.69:0/3867037970 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff809b950 con 0x556d567e1660 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f4feffff640 1 -- 172.21.15.69:0/3867037970 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff80b3070 con 0x556d567e1660 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.276+0000 7f4feffff640 1 -- 172.21.15.69:0/3867037970 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff809bb80 con 0x556d567e1660 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- 172.21.15.69:0/3867037970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 msgr2=0x556d567febd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7f4ff809e010 tx=0x7f4ff80cccf0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- 172.21.15.69:0/3867037970 shutdown_connections 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f2cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e2050 0x556d567ff110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 --2- 172.21.15.69:0/3867037970 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e1660 0x556d567febd0 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- 172.21.15.69:0/3867037970 >> 172.21.15.69:0/3867037970 conn(0x556d567d88c0 msgr2=0x556d567e32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- 172.21.15.69:0/3867037970 shutdown_connections 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- 172.21.15.69:0/3867037970 wait complete. 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 -- start start 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.277+0000 7f500fc1c740 1 Processor -- start 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e1660 0x556d567efe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f50081a6780 con 0x556d567e2050 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f50081a7bf0 con 0x556d567e1660 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f50081a6e50 con 0x556d567e71c0 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500fc1c740 10 client.0 _ll_register_callbacks cb 0x7f4ff8003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37828/0 (socket says 172.21.15.69:37828) 2024-05-09T09:26:46.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f500659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e1660 0x556d567efe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 -- 172.21.15.69:0/4285220454 learned_addr learned my addr 172.21.15.69:0/4285220454 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5006d9d640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 -- 172.21.15.69:0/4285220454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 msgr2=0x556d567f13c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f13c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 -- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e1660 msgr2=0x556d567efe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e1660 0x556d567efe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 -- 172.21.15.69:0/4285220454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d567ea850 con 0x556d567e2050 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f5005d9b640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009900 tx=0x7f4ff0005a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9812c76c117c50c0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f4fedffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.278+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4ff0035070 con 0x556d567e2050 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f4ff0031bd0 con 0x556d567e2050 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4ff80b3070 con 0x556d567e2050 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f4ff0069050 con 0x556d567e2050 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 10 client.16683 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:46.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x556d567ea330 con 0x556d567e2050 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4ff80b3070 con 0x556d567e2050 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f4ff003c5e0 con 0x556d567e2050 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 10 client.16683 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 20 client.16683 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d5685d490 0x556d5685f880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 4 client.16683 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d56748dc0 con 0x556d5685d490 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.279+0000 7f500659c640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d5685d490 0x556d5685f880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.280+0000 7f4fedffb640 10 client.16683 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:46.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.280+0000 7f500659c640 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d5685d490 0x556d5685f880 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b99c1d968a61c66 server_cookie=24e87d5c235af9c0 in_seq=0 out_seq=0 2024-05-09T09:26:46.374 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.374+0000 7f4fedffb640 1 -- 172.21.15.69:0/4285220454 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d56748dc0 con 0x556d5685d490 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.374+0000 7f4fedffb640 10 client.16683 handle_command_reply: tid=0 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.374+0000 7f500fc1c740 1 client.16683 shutdown 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 msgr2=0x556d567f03b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f4ff0009900 tx=0x7f4ff0005a60 comp rx=0 tx=0).stop 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 shutdown_connections 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d5685d490 0x556d5685f880 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d567e71c0 0x556d567f13c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d567e2050 0x556d567f03b0 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f4fedffb640 0 client.16683 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 --2- 172.21.15.69:0/4285220454 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d567e1660 0x556d567efe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 >> 172.21.15.69:0/4285220454 conn(0x556d567d88c0 msgr2=0x556d567db840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 shutdown_connections 2024-05-09T09:26:46.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 1 -- 172.21.15.69:0/4285220454 wait complete. 2024-05-09T09:26:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.375+0000 7f500fc1c740 20 client.16683 trim_cache size 0 max 16384 2024-05-09T09:26:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:26:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:26:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:26:46.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 msgr2=0x7f4ff40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 0x7f4ff40a5170 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f5000009fd0 tx=0x7f50000096a0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 msgr2=0x7f5008197ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7f4ff800e9c0 tx=0x7f4ff800ee80 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 shutdown_connections 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4ff40a2cb0 0x7f4ff40a5170 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5008102370 0x7f500819f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f50081019b0 0x7f50081980e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 --2- 172.21.15.69:0/3946021762 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f50081007c0 0x7f5008197ba0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 >> 172.21.15.69:0/3946021762 conn(0x7f50080fbd30 msgr2=0x7f50080fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:46.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 shutdown_connections 2024-05-09T09:26:46.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:46.376+0000 7f500ce31640 1 -- 172.21.15.69:0/3946021762 wait complete. 2024-05-09T09:26:46.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 441996 7.02 MB/sec execute 284 sec latency 92.283 ms 2024-05-09T09:26:47.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 443944 7.04 MB/sec execute 285 sec latency 12.174 ms 2024-05-09T09:26:47.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:47 smithi103 ceph-mon[34421]: pgmap v606: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-05-09T09:26:48.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:47 smithi069 ceph-mon[30123]: pgmap v606: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-05-09T09:26:48.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:47 smithi163 ceph-mon[34354]: pgmap v606: 129 pgs: 129 active+clean; 97 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.7 MiB/s wr, 321 op/s 2024-05-09T09:26:48.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 446281 7.06 MB/sec execute 286 sec latency 16.820 ms 2024-05-09T09:26:49.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 447688 7.06 MB/sec execute 287 sec latency 56.897 ms 2024-05-09T09:26:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:49 smithi103 ceph-mon[34421]: pgmap v607: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 353 op/s 2024-05-09T09:26:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:49 smithi069 ceph-mon[30123]: pgmap v607: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 353 op/s 2024-05-09T09:26:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:49 smithi163 ceph-mon[34354]: pgmap v607: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 2.7 MiB/s wr, 353 op/s 2024-05-09T09:26:50.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 450134 7.08 MB/sec execute 288 sec latency 38.379 ms 2024-05-09T09:26:51.494 INFO:tasks.workunit.client.0.smithi069.stdout: 1 451475 7.08 MB/sec execute 289 sec latency 19.575 ms 2024-05-09T09:26:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:51 smithi103 ceph-mon[34421]: pgmap v608: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 302 op/s 2024-05-09T09:26:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:51 smithi069 ceph-mon[30123]: pgmap v608: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 302 op/s 2024-05-09T09:26:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:51 smithi163 ceph-mon[34354]: pgmap v608: 129 pgs: 129 active+clean; 96 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.4 MiB/s wr, 302 op/s 2024-05-09T09:26:52.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 454288 7.12 MB/sec execute 290 sec latency 12.802 ms 2024-05-09T09:26:53.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 455691 7.11 MB/sec execute 291 sec latency 22.126 ms 2024-05-09T09:26:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:53 smithi103 ceph-mon[34421]: pgmap v609: 129 pgs: 129 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-05-09T09:26:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:53 smithi069 ceph-mon[30123]: pgmap v609: 129 pgs: 129 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-05-09T09:26:54.522 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:53 smithi163 ceph-mon[34354]: pgmap v609: 129 pgs: 129 active+clean; 98 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 3.1 MiB/s wr, 358 op/s 2024-05-09T09:26:54.522 INFO:tasks.workunit.client.0.smithi069.stdout: 1 457974 7.14 MB/sec execute 292 sec latency 10.266 ms 2024-05-09T09:26:55.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 458797 7.17 MB/sec execute 293 sec latency 51.207 ms 2024-05-09T09:26:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:55 smithi103 ceph-mon[34421]: pgmap v610: 129 pgs: 129 active+clean; 100 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-05-09T09:26:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:55 smithi069 ceph-mon[30123]: pgmap v610: 129 pgs: 129 active+clean; 100 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-05-09T09:26:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:55 smithi163 ceph-mon[34354]: pgmap v610: 129 pgs: 129 active+clean; 100 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 2.8 MiB/s wr, 356 op/s 2024-05-09T09:26:56.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 461541 7.21 MB/sec execute 294 sec latency 22.767 ms 2024-05-09T09:26:57.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 463301 7.22 MB/sec execute 295 sec latency 12.513 ms 2024-05-09T09:26:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:57 smithi103 ceph-mon[34421]: pgmap v611: 129 pgs: 129 active+clean; 94 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:26:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:57 smithi069 ceph-mon[30123]: pgmap v611: 129 pgs: 129 active+clean; 94 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:26:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:57 smithi163 ceph-mon[34354]: pgmap v611: 129 pgs: 129 active+clean; 94 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:26:58.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 465836 7.24 MB/sec execute 296 sec latency 10.430 ms 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 -- 172.21.15.69:0/4220143389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc102380 msgr2=0x7f01fc108c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 --2- 172.21.15.69:0/4220143389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc102380 0x7f01fc108c10 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f01f4009ca0 tx=0x7f01f402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 -- 172.21.15.69:0/4220143389 shutdown_connections 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 --2- 172.21.15.69:0/4220143389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc102380 0x7f01fc108c10 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 --2- 172.21.15.69:0/4220143389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc1019c0 0x7f01fc101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 --2- 172.21.15.69:0/4220143389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1007d0 0x7f01fc100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.467+0000 7f0203b9e640 1 -- 172.21.15.69:0/4220143389 >> 172.21.15.69:0/4220143389 conn(0x7f01fc0fbd40 msgr2=0x7f01fc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.468+0000 7f0203b9e640 1 -- 172.21.15.69:0/4220143389 shutdown_connections 2024-05-09T09:26:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.468+0000 7f0203b9e640 1 -- 172.21.15.69:0/4220143389 wait complete. 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.468+0000 7f0203b9e640 1 Processor -- start 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.468+0000 7f0203b9e640 1 -- start start 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1019c0 0x7f01fc102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc102380 0x7f01fc1032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10c4a0 con 0x7f01fc1007d0 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10c320 con 0x7f01fc1019c0 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0203b9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc10c620 con 0x7f01fc102380 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42738/0 (socket says 172.21.15.69:42738) 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 -- 172.21.15.69:0/2386689157 learned_addr learned my addr 172.21.15.69:0/2386689157 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f0201139640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1019c0 0x7f01fc102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020213b640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc102380 0x7f01fc1032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 -- 172.21.15.69:0/2386689157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc102380 msgr2=0x7f01fc1032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc102380 0x7f01fc1032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 -- 172.21.15.69:0/2386689157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1019c0 msgr2=0x7f01fc102da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1019c0 0x7f01fc102da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 -- 172.21.15.69:0/2386689157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f01fc103b70 con 0x7f01fc1007d0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f020193a640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f01f800e9c0 tx=0x7f01f800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f01f2ffd640 1 -- 172.21.15.69:0/2386689157 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f800cd80 con 0x7f01fc1007d0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.469+0000 7f01f2ffd640 1 -- 172.21.15.69:0/2386689157 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f80035f0 con 0x7f01fc1007d0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.470+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f01fc1a1bc0 con 0x7f01fc1007d0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.470+0000 7f01f2ffd640 1 -- 172.21.15.69:0/2386689157 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8010690 con 0x7f01fc1007d0 2024-05-09T09:26:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.470+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f01fc1a2110 con 0x7f01fc1007d0 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.472+0000 7f01f2ffd640 1 -- 172.21.15.69:0/2386689157 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f01f80037a0 con 0x7f01fc1007d0 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 Processor -- start 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 -- start start 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e314c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e3151740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e7e30b0340 con 0x55e7e314cc40 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7e3058290 con 0x55e7e3151db0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0206986740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7e30ac300 con 0x55e7e314c250 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f01f2ffd640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 0x7f01e80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e3151740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0201139640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f020213b640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 0x7f01e80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0201139640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:33682/0 (socket says 172.21.15.69:33682) 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f0201139640 1 -- 172.21.15.69:0/1096459425 learned_addr learned my addr 172.21.15.69:0/1096459425 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f020213b640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e314c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.473+0000 7f01f2ffd640 1 -- 172.21.15.69:0/2386689157 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f01f80cf890 con 0x7f01fc1007d0 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 -- 172.21.15.69:0/1096459425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 msgr2=0x55e7e314c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e314c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 -- 172.21.15.69:0/1096459425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 msgr2=0x55e7e3151740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e3151740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 -- 172.21.15.69:0/1096459425 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e307c8e0 con 0x55e7e3151db0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f020213b640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e314c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f020193a640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e3151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f0201139640 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009ca0 tx=0x7f01ec011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf395d1c213d5114 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f020213b640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 0x7f01e80a50f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005e30 tx=0x7f01f4005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f01e7fff640 1 -- 172.21.15.69:0/1096459425 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec01e070 con 0x55e7e3151db0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f01e7fff640 1 -- 172.21.15.69:0/1096459425 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01ec009e50 con 0x55e7e3151db0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.474+0000 7f01e7fff640 1 -- 172.21.15.69:0/1096459425 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec00e9e0 con 0x55e7e3151db0 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- 172.21.15.69:0/1096459425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 msgr2=0x55e7e3154240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009ca0 tx=0x7f01ec011810 comp rx=0 tx=0).stop 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- 172.21.15.69:0/1096459425 shutdown_connections 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e3154240 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e3151740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 --2- 172.21.15.69:0/1096459425 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e314c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- 172.21.15.69:0/1096459425 >> 172.21.15.69:0/1096459425 conn(0x55e7e31478c0 msgr2=0x55e7e3147cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- 172.21.15.69:0/1096459425 shutdown_connections 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- 172.21.15.69:0/1096459425 wait complete. 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 -- start start 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.475+0000 7f0206986740 1 Processor -- start 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 0x55e7e3156920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e7e3154d60 con 0x55e7e314c250 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7e306b790 con 0x55e7e3151db0 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7e3154ee0 con 0x55e7e314cc40 2024-05-09T09:26:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 10 client.0 _ll_register_callbacks cb 0x55e7e30a6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42748/0 (socket says 172.21.15.69:42748) 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020193a640 1 -- 172.21.15.69:0/2889010933 learned_addr learned my addr 172.21.15.69:0/2889010933 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0201139640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 0x55e7e3156920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 -- 172.21.15.69:0/2889010933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 msgr2=0x55e7e3156920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 0x55e7e3156920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 -- 172.21.15.69:0/2889010933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 msgr2=0x55e7e31563e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 -- 172.21.15.69:0/2889010933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e2dc4210 con 0x55e7e3151db0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020193a640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0201139640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 0x55e7e3156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f020213b640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315d960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01f402ade0 tx=0x7f01f402a8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18c6a92cd200916 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f01e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f4047040 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01ec009e50 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.476+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f01f4004240 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01f405a050 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 10 client.26012 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7e2e90620 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01ec009e50 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f01f40305c0 con 0x55e7e3151db0 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 10 client.26012 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 10 client.26012 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 20 client.26012 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e3161ea0 0x55e7e3164360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 4 client.26012 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:26:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7e30bda70 con 0x55e7e3161ea0 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f020193a640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e3161ea0 0x55e7e3164360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f01e57fa640 10 client.26012 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.477+0000 7f020193a640 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e3161ea0 0x55e7e3164360 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=13e4728f26a8a064 server_cookie=16ab141498baa454 in_seq=0 out_seq=0 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f01e57fa640 1 -- 172.21.15.69:0/2889010933 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7e30bda70 con 0x55e7e3161ea0 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f01e57fa640 10 client.26012 handle_command_reply: tid=0 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 client.26012 shutdown 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 msgr2=0x55e7e315d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315d960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f01f402ade0 tx=0x7f01f402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 shutdown_connections 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315d960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e3161ea0 0x55e7e3164360 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314cc40 0x55e7e3156920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 --2- 172.21.15.69:0/2889010933 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e31563e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f01e57fa640 0 client.26012 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 >> 172.21.15.69:0/2889010933 conn(0x55e7e31478c0 msgr2=0x55e7e3146a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 shutdown_connections 2024-05-09T09:26:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.478+0000 7f0206986740 1 -- 172.21.15.69:0/2889010933 wait complete. 2024-05-09T09:26:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.479+0000 7f0206986740 20 client.26012 trim_cache size 0 max 16384 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 Processor -- start 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 -- start start 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e314cc40 0x55e7e316a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e3151db0 0x55e7e315d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1aedc0 con 0x55e7e314c250 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1ac690 con 0x55e7e314cc40 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0206986740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1aa4e0 con 0x55e7e3151db0 2024-05-09T09:26:59.495 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42758/0 (socket says 172.21.15.69:42758) 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 -- 172.21.15.69:0/2836410616 learned_addr learned my addr 172.21.15.69:0/2836410616 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f0201139640 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e314cc40 0x55e7e316a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020213b640 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e3151db0 0x55e7e315d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 -- 172.21.15.69:0/2836410616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e3151db0 msgr2=0x55e7e315d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e3151db0 0x55e7e315d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 -- 172.21.15.69:0/2836410616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e314cc40 msgr2=0x55e7e316a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e314cc40 0x55e7e316a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 -- 172.21.15.69:0/2836410616 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7e3141c40 con 0x55e7e314c250 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f020193a640 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f01f8012020 tx=0x7f01f806dd80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=818f4b17c6db0bbb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f01e77fe640 1 -- 172.21.15.69:0/2836410616 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f8066e50 con 0x55e7e314c250 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f01e77fe640 1 -- 172.21.15.69:0/2836410616 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f809bd70 con 0x55e7e314c250 2024-05-09T09:26:59.496 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.494+0000 7f01e77fe640 1 -- 172.21.15.69:0/2836410616 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01f809b1c0 con 0x55e7e314c250 2024-05-09T09:26:59.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 468005 7.24 MB/sec execute 297 sec latency 15.720 ms 2024-05-09T09:26:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- 172.21.15.69:0/2836410616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 msgr2=0x55e7e3169c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.500 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f01f8012020 tx=0x7f01f806dd80 comp rx=0 tx=0).stop 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- 172.21.15.69:0/2836410616 shutdown_connections 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e3151db0 0x55e7e315d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e314cc40 0x55e7e316a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 --2- 172.21.15.69:0/2836410616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314c250 0x55e7e3169c70 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- 172.21.15.69:0/2836410616 >> 172.21.15.69:0/2836410616 conn(0x55e7e31478c0 msgr2=0x55e7e31563e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- 172.21.15.69:0/2836410616 shutdown_connections 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- 172.21.15.69:0/2836410616 wait complete. 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.495+0000 7f0206986740 1 -- start start 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 Processor -- start 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e315b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315c9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1aab50 con 0x55e7e314cc40 2024-05-09T09:26:59.501 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1abfc0 con 0x55e7e3151db0 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f01fc1ab220 con 0x55e7e314c250 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0206986740 10 client.0 _ll_register_callbacks cb 0x7f01f8005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42772/0 (socket says 172.21.15.69:42772) 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 -- 172.21.15.69:0/1865909952 learned_addr learned my addr 172.21.15.69:0/1865909952 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f020193a640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e315b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f020213b640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315c9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 -- 172.21.15.69:0/1865909952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 msgr2=0x55e7e315b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e315b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 -- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 msgr2=0x55e7e315c9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315c9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f0201139640 1 -- 172.21.15.69:0/1865909952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7e315cf90 con 0x55e7e314cc40 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.496+0000 7f020213b640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:26:59.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0201139640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009ca0 tx=0x7f01ec009ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9bfb4ec12ff433fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f01ec01e040 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f01f809bd70 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f01ec0037f0 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f01ec033050 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 10 client.16701 fetch_fsmap learned FSMap version 37 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55e7e31c8940 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f01f809bd70 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f01ec019430 con 0x55e7e314cc40 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 10 client.16701 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 10 client.16701 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 20 client.16701 populate_metadata read hostname 'smithi069' 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e31c9180 0x55e7e31cb5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:26:59.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 4 client.16701 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:26:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.497+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7e30b7dc0 con 0x55e7e31c9180 2024-05-09T09:26:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.498+0000 7f020193a640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e31c9180 0x55e7e31cb5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:26:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.498+0000 7f020193a640 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e31c9180 0x55e7e31cb5a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eeeb92b31c2dc7f3 server_cookie=e8f6989c8763f3d6 in_seq=0 out_seq=0 2024-05-09T09:26:59.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.498+0000 7f01e57fa640 10 client.16701 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:26:59 smithi103 ceph-mon[34421]: pgmap v612: 129 pgs: 129 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f01e57fa640 1 -- 172.21.15.69:0/1865909952 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7e30b7dc0 con 0x55e7e31c9180 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f01e57fa640 10 client.16701 handle_command_reply: tid=0 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 client.16701 shutdown 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 msgr2=0x55e7e315b990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f01ec009ca0 tx=0x7f01ec009ba0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 shutdown_connections 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7e3151db0 0x55e7e315c9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7e314cc40 0x55e7e315b990 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7e31c9180 0x55e7e31cb5a0 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 --2- 172.21.15.69:0/1865909952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7e314c250 0x55e7e315b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f01e57fa640 0 client.16701 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 >> 172.21.15.69:0/1865909952 conn(0x55e7e31478c0 msgr2=0x55e7e314a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 shutdown_connections 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 1 -- 172.21.15.69:0/1865909952 wait complete. 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.634+0000 7f0206986740 20 client.16701 trim_cache size 0 max 16384 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:26:59.636 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:26:59.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:26:59.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 msgr2=0x7f01e80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 0x7f01e80a50f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f01f4005e30 tx=0x7f01f4005d80 comp rx=0 tx=0).stop 2024-05-09T09:26:59.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 msgr2=0x7f01fc104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f01f800e9c0 tx=0x7f01f800ee80 comp rx=0 tx=0).stop 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 shutdown_connections 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f01e80a2c30 0x7f01e80a50f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f01fc102380 0x7f01fc1032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f01fc1019c0 0x7f01fc102da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 --2- 172.21.15.69:0/2386689157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f01fc1007d0 0x7f01fc104760 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 >> 172.21.15.69:0/2386689157 conn(0x7f01fc0fbd40 msgr2=0x7f01fc0fd660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 shutdown_connections 2024-05-09T09:26:59.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:26:59.635+0000 7f0203b9e640 1 -- 172.21.15.69:0/2386689157 wait complete. 2024-05-09T09:26:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:26:59 smithi069 ceph-mon[30123]: pgmap v612: 129 pgs: 129 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:26:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:26:59 smithi163 ceph-mon[34354]: pgmap v612: 129 pgs: 129 active+clean; 88 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:27:00.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 469551 7.26 MB/sec execute 298 sec latency 90.235 ms 2024-05-09T09:27:01.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 471896 7.27 MB/sec execute 299 sec latency 39.502 ms 2024-05-09T09:27:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:01 smithi103 ceph-mon[34421]: pgmap v613: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-09T09:27:01.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:01 smithi069 ceph-mon[30123]: pgmap v613: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-09T09:27:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:01 smithi163 ceph-mon[34354]: pgmap v613: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.4 MiB/s wr, 250 op/s 2024-05-09T09:27:02.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 473453 7.27 MB/sec execute 300 sec latency 14.107 ms 2024-05-09T09:27:03.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 476324 7.31 MB/sec execute 301 sec latency 15.826 ms 2024-05-09T09:27:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:03 smithi103 ceph-mon[34421]: pgmap v614: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:27:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:03 smithi069 ceph-mon[30123]: pgmap v614: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:27:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:03 smithi163 ceph-mon[34354]: pgmap v614: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 1019 KiB/s rd, 3.1 MiB/s wr, 351 op/s 2024-05-09T09:27:04.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 478180 7.31 MB/sec execute 302 sec latency 13.031 ms 2024-05-09T09:27:05.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 480275 7.33 MB/sec execute 303 sec latency 13.854 ms 2024-05-09T09:27:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:05 smithi103 ceph-mon[34421]: pgmap v615: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.8 MiB/s wr, 357 op/s 2024-05-09T09:27:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:05 smithi069 ceph-mon[30123]: pgmap v615: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.8 MiB/s wr, 357 op/s 2024-05-09T09:27:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:05 smithi163 ceph-mon[34354]: pgmap v615: 129 pgs: 129 active+clean; 89 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 815 KiB/s rd, 2.8 MiB/s wr, 357 op/s 2024-05-09T09:27:06.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 482251 7.33 MB/sec execute 304 sec latency 66.955 ms 2024-05-09T09:27:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:27:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:27:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:27:06.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:27:06.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:27:06.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:27:06.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:27:06.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:27:06.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:27:06.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:27:06.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:27:06.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:27:07.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 483941 7.34 MB/sec execute 305 sec latency 13.476 ms 2024-05-09T09:27:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:07 smithi103 ceph-mon[34421]: pgmap v616: 129 pgs: 129 active+clean; 90 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 3.0 MiB/s wr, 371 op/s 2024-05-09T09:27:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:07 smithi069 ceph-mon[30123]: pgmap v616: 129 pgs: 129 active+clean; 90 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 3.0 MiB/s wr, 371 op/s 2024-05-09T09:27:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:07 smithi163 ceph-mon[34354]: pgmap v616: 129 pgs: 129 active+clean; 90 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 810 KiB/s rd, 3.0 MiB/s wr, 371 op/s 2024-05-09T09:27:08.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 486833 7.38 MB/sec execute 306 sec latency 12.601 ms 2024-05-09T09:27:09.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 488267 7.37 MB/sec execute 307 sec latency 13.740 ms 2024-05-09T09:27:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:09 smithi103 ceph-mon[34421]: pgmap v617: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.6 MiB/s wr, 364 op/s 2024-05-09T09:27:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:09 smithi069 ceph-mon[30123]: pgmap v617: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.6 MiB/s wr, 364 op/s 2024-05-09T09:27:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:09 smithi163 ceph-mon[34354]: pgmap v617: 129 pgs: 129 active+clean; 91 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 2.6 MiB/s wr, 364 op/s 2024-05-09T09:27:10.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 490702 7.39 MB/sec execute 308 sec latency 11.174 ms 2024-05-09T09:27:11.495 INFO:tasks.workunit.client.0.smithi069.stdout: 1 492343 7.39 MB/sec execute 309 sec latency 39.387 ms 2024-05-09T09:27:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:11 smithi103 ceph-mon[34421]: pgmap v618: 129 pgs: 129 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.6 MiB/s wr, 328 op/s 2024-05-09T09:27:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:11 smithi069 ceph-mon[30123]: pgmap v618: 129 pgs: 129 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.6 MiB/s wr, 328 op/s 2024-05-09T09:27:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:11 smithi163 ceph-mon[34354]: pgmap v618: 129 pgs: 129 active+clean; 92 MiB data, 1.7 GiB used, 1.0 TiB / 1.0 TiB avail; 609 KiB/s rd, 2.6 MiB/s wr, 328 op/s 2024-05-09T09:27:12.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 494387 7.41 MB/sec execute 310 sec latency 17.204 ms 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 -- 172.21.15.69:0/1682238586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278104ca0 msgr2=0x7fb278105120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 --2- 172.21.15.69:0/1682238586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278104ca0 0x7fb278105120 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7fb260009900 tx=0x7fb26002a410 comp rx=0 tx=0).stop 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 -- 172.21.15.69:0/1682238586 shutdown_connections 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 --2- 172.21.15.69:0/1682238586 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278105660 0x7fb27810bef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 --2- 172.21.15.69:0/1682238586 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278104ca0 0x7fb278105120 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 --2- 172.21.15.69:0/1682238586 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 0x7fb278103ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 -- 172.21.15.69:0/1682238586 >> 172.21.15.69:0/1682238586 conn(0x7fb2780ff020 msgr2=0x7fb278101460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 -- 172.21.15.69:0/1682238586 shutdown_connections 2024-05-09T09:27:12.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.731+0000 7fb27d17e640 1 -- 172.21.15.69:0/1682238586 wait complete. 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 Processor -- start 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 -- start start 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 0x7fb278199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278104ca0 0x7fb27819a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb27810f730 con 0x7fb278105660 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb27810f5b0 con 0x7fb278103ab0 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27d17e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb27810f8b0 con 0x7fb278104ca0 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36524/0 (socket says 172.21.15.69:36524) 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb27759e640 1 -- 172.21.15.69:0/2587658967 learned_addr learned my addr 172.21.15.69:0/2587658967 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 0x7fb278199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.732+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278104ca0 0x7fb27819a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 -- 172.21.15.69:0/2587658967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278104ca0 msgr2=0x7fb27819a2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278104ca0 0x7fb27819a2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 -- 172.21.15.69:0/2587658967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 msgr2=0x7fb278199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 0x7fb278199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 -- 172.21.15.69:0/2587658967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2781a1990 con 0x7fb278105660 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27759e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb27000eb10 tx=0x7fb27000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb274d99640 1 -- 172.21.15.69:0/2587658967 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270004b20 con 0x7fb278105660 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb274d99640 1 -- 172.21.15.69:0/2587658967 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb270004cc0 con 0x7fb278105660 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2781a1c80 con 0x7fb278105660 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb274d99640 1 -- 172.21.15.69:0/2587658967 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb270010760 con 0x7fb278105660 2024-05-09T09:27:12.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.733+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2781a21a0 con 0x7fb278105660 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.735+0000 7fb274d99640 1 -- 172.21.15.69:0/2587658967 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb270005020 con 0x7fb278105660 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.735+0000 7fb27ff6f740 1 Processor -- start 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.735+0000 7fb27ff6f740 1 -- start start 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe83f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 0x5582fe847240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5582fe7a3340 con 0x5582fe844db0 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5582fe74b290 con 0x5582fe83f250 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5582fe79f300 con 0x5582fe83fc40 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:49006/0 (socket says 172.21.15.69:49006) 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 -- 172.21.15.69:0/1916230311 learned_addr learned my addr 172.21.15.69:0/1916230311 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb274d99640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 0x7fb2640a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb26ffff640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 0x5582fe847240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb26ffff640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe83f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb27759e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 0x7fb2640a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 -- 172.21.15.69:0/1916230311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 msgr2=0x5582fe83f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe83f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb274d99640 1 -- 172.21.15.69:0/2587658967 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7fb2700cfa60 con 0x7fb278105660 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.736+0000 7fb276d9d640 1 -- 172.21.15.69:0/1916230311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 msgr2=0x5582fe847240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 0x5582fe847240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb276d9d640 1 -- 172.21.15.69:0/1916230311 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fe76f8e0 con 0x5582fe83fc40 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb26ffff640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 0x5582fe847240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb27759e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 0x7fb2640a50f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb27001fff0 tx=0x7fb2700a0040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb268009ca0 tx=0x7fb268011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b4874a9d6a5045e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb26d7fa640 1 -- 172.21.15.69:0/1916230311 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268004220 con 0x5582fe83fc40 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb26d7fa640 1 -- 172.21.15.69:0/1916230311 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2680043c0 con 0x5582fe83fc40 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.737+0000 7fb26d7fa640 1 -- 172.21.15.69:0/1916230311 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2680198b0 con 0x5582fe83fc40 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1916230311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 msgr2=0x5582fe844740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb268009ca0 tx=0x7fb268011810 comp rx=0 tx=0).stop 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1916230311 shutdown_connections 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe844db0 0x5582fe847240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83fc40 0x5582fe844740 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1916230311 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe83f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1916230311 >> 172.21.15.69:0/1916230311 conn(0x5582fe83a8c0 msgr2=0x5582fe83acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1916230311 shutdown_connections 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1916230311 wait complete. 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 -- start start 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 Processor -- start 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.738+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe849950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe850990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5582fe847d60 con 0x5582fe83fc40 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5582fe75e790 con 0x5582fe83f250 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5582fe847ee0 con 0x5582fe844db0 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 10 client.0 _ll_register_callbacks cb 0x5582fe799a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe849950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.745 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:47016/0 (socket says 172.21.15.69:47016) 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 -- 172.21.15.69:0/2119545964 learned_addr learned my addr 172.21.15.69:0/2119545964 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe850990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 -- 172.21.15.69:0/2119545964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 msgr2=0x5582fe850990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe850990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 -- 172.21.15.69:0/2119545964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 msgr2=0x5582fe849950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe849950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 -- 172.21.15.69:0/2119545964 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe4b7210 con 0x5582fe83f250 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe849950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb260031610 tx=0x7fb260005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7d2910dab180b8ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb25e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb260035070 con 0x5582fe83f250 2024-05-09T09:27:12.746 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7fb260004480 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb2680043c0 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.739+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb260053050 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 10 client.26030 fetch_fsmap learned FSMap version 37 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5582fe583620 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2680043c0 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb260004bd0 con 0x5582fe83f250 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 10 client.26030 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 10 client.26030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 20 client.26030 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe854e30 0x5582fe8572a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 4 client.26030 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5582fe7b0a70 con 0x5582fe854e30 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.740+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe854e30 0x5582fe8572a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.741+0000 7fb25e7fc640 10 client.26030 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:12.747 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.741+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe854e30 0x5582fe8572a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fe392f30bfc2bc9 server_cookie=336976efd459e969 in_seq=0 out_seq=0 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.741+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2119545964 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5582fe7b0a70 con 0x5582fe854e30 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.741+0000 7fb25e7fc640 10 client.26030 handle_command_reply: tid=0 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.741+0000 7fb27ff6f740 1 client.26030 shutdown 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 msgr2=0x5582fe849410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fb260031610 tx=0x7fb260005d50 comp rx=0 tx=0).stop 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 shutdown_connections 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe854e30 0x5582fe8572a0 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe850990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe849950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2119545964 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849410 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb25e7fc640 0 client.26030 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 >> 172.21.15.69:0/2119545964 conn(0x5582fe83a8c0 msgr2=0x5582fe839a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 shutdown_connections 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2119545964 wait complete. 2024-05-09T09:27:12.748 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.742+0000 7fb27ff6f740 20 client.26030 trim_cache size 0 max 16384 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 Processor -- start 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 -- start start 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe849560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781aee70 con 0x5582fe83fc40 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781ac740 con 0x5582fe83f250 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781aa590 con 0x5582fe844db0 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb26ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36554/0 (socket says 172.21.15.69:36554) 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb276d9d640 1 -- 172.21.15.69:0/1125693009 learned_addr learned my addr 172.21.15.69:0/1125693009 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.759+0000 7fb26ffff640 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe849560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 -- 172.21.15.69:0/1125693009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 msgr2=0x5582fe849560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe849560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 -- 172.21.15.69:0/1125693009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 msgr2=0x5582fe849020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 -- 172.21.15.69:0/1125693009 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5582fe838f00 con 0x5582fe83fc40 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb26ffff640 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb276d9d640 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb268009230 tx=0x7fb2680096a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5b4632f14f25fa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb26cff9640 1 -- 172.21.15.69:0/1125693009 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268031070 con 0x5582fe83fc40 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb26cff9640 1 -- 172.21.15.69:0/1125693009 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb268003db0 con 0x5582fe83fc40 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.760+0000 7fb26cff9640 1 -- 172.21.15.69:0/1125693009 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb26802cd70 con 0x5582fe83fc40 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1125693009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 msgr2=0x5582fe850950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7fb268009230 tx=0x7fb2680096a0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1125693009 shutdown_connections 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe844db0 0x5582fe849560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe850950 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/1125693009 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe83f250 0x5582fe849020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1125693009 >> 172.21.15.69:0/1125693009 conn(0x5582fe83a8c0 msgr2=0x5582fe83d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1125693009 shutdown_connections 2024-05-09T09:27:12.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- 172.21.15.69:0/1125693009 wait complete. 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- start start 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 Processor -- start 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83f250 0x5582fe84e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe844db0 0x5582fe84f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781aac00 con 0x5582fe83fc40 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781ac070 con 0x5582fe844db0 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2781ab2d0 con 0x5582fe83f250 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb27ff6f740 10 client.0 _ll_register_callbacks cb 0x7fb260009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb276d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36556/0 (socket says 172.21.15.69:36556) 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb276d9d640 1 -- 172.21.15.69:0/2629614152 learned_addr learned my addr 172.21.15.69:0/2629614152 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.761+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83f250 0x5582fe84e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe844db0 0x5582fe84f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 -- 172.21.15.69:0/2629614152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83f250 msgr2=0x5582fe84e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83f250 0x5582fe84e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 -- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe844db0 msgr2=0x5582fe84f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe844db0 0x5582fe84f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 -- 172.21.15.69:0/2629614152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5582fe84fbe0 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb276d9d640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb268014040 tx=0x7fb26802cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbe6121d0bc85e1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb268031030 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7fb26802b070 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb268003db0 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7fb26803e050 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb27ff6f740 10 client.16719 fetch_fsmap learned FSMap version 37 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x5582fe8bb480 con 0x5582fe83fc40 2024-05-09T09:27:12.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb268003db0 con 0x5582fe83fc40 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.762+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7fb2680384c0 con 0x5582fe83fc40 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 10 client.16719 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 10 client.16719 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 20 client.16719 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe8bbc60 0x5582fe8be080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 4 client.16719 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5582fe7aadc0 con 0x5582fe8bbc60 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe8bbc60 0x5582fe8be080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb25e7fc640 10 client.16719 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:12.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.763+0000 7fb26ffff640 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe8bbc60 0x5582fe8be080 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=77741a35f0faf862 server_cookie=7d6f9daf5ff62c24 in_seq=0 out_seq=0 2024-05-09T09:27:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.875+0000 7fb25e7fc640 1 -- 172.21.15.69:0/2629614152 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5582fe7aadc0 con 0x5582fe8bbc60 2024-05-09T09:27:12.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.875+0000 7fb25e7fc640 10 client.16719 handle_command_reply: tid=0 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.875+0000 7fb27ff6f740 1 client.16719 shutdown 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.875+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 msgr2=0x5582fe84e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fb268014040 tx=0x7fb26802cd70 comp rx=0 tx=0).stop 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 shutdown_connections 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5582fe844db0 0x5582fe84f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5582fe83fc40 0x5582fe84e5e0 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5582fe8bbc60 0x5582fe8be080 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 --2- 172.21.15.69:0/2629614152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5582fe83f250 0x5582fe84e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb25e7fc640 0 client.16719 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 >> 172.21.15.69:0/2629614152 conn(0x5582fe83a8c0 msgr2=0x5582fe849020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 shutdown_connections 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 1 -- 172.21.15.69:0/2629614152 wait complete. 2024-05-09T09:27:12.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27ff6f740 20 client.16719 trim_cache size 0 max 16384 2024-05-09T09:27:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:27:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:27:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:27:12.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 msgr2=0x7fb2640a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 0x7fb2640a50f0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fb27001fff0 tx=0x7fb2700a0040 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 msgr2=0x7fb2781a1320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.876+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7fb27000eb10 tx=0x7fb27000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 shutdown_connections 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2640a2c30 0x7fb2640a50f0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb278105660 0x7fb2781a1320 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb278104ca0 0x7fb27819a2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 --2- 172.21.15.69:0/2587658967 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb278103ab0 0x7fb278199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 >> 172.21.15.69:0/2587658967 conn(0x7fb2780ff020 msgr2=0x7fb278107f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:12.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 shutdown_connections 2024-05-09T09:27:12.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:12.877+0000 7fb27d17e640 1 -- 172.21.15.69:0/2587658967 wait complete. 2024-05-09T09:27:13.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 496728 7.42 MB/sec execute 311 sec latency 71.578 ms 2024-05-09T09:27:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:13 smithi103 ceph-mon[34421]: pgmap v619: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-05-09T09:27:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:13 smithi069 ceph-mon[30123]: pgmap v619: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-05-09T09:27:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:13 smithi163 ceph-mon[34354]: pgmap v619: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 731 KiB/s rd, 3.0 MiB/s wr, 382 op/s 2024-05-09T09:27:14.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 498226 7.42 MB/sec execute 312 sec latency 13.715 ms 2024-05-09T09:27:15.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 501159 7.46 MB/sec execute 313 sec latency 12.750 ms 2024-05-09T09:27:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:15 smithi103 ceph-mon[34421]: pgmap v620: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-05-09T09:27:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:15 smithi069 ceph-mon[30123]: pgmap v620: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-05-09T09:27:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:15 smithi163 ceph-mon[34354]: pgmap v620: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 650 KiB/s rd, 2.6 MiB/s wr, 336 op/s 2024-05-09T09:27:16.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 502351 7.45 MB/sec execute 314 sec latency 66.275 ms 2024-05-09T09:27:17.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 504816 7.47 MB/sec execute 315 sec latency 37.401 ms 2024-05-09T09:27:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:17 smithi103 ceph-mon[34421]: pgmap v621: 129 pgs: 129 active+clean; 90 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.0 MiB/s wr, 351 op/s 2024-05-09T09:27:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:17 smithi069 ceph-mon[30123]: pgmap v621: 129 pgs: 129 active+clean; 90 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.0 MiB/s wr, 351 op/s 2024-05-09T09:27:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:17 smithi163 ceph-mon[34354]: pgmap v621: 129 pgs: 129 active+clean; 90 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 701 KiB/s rd, 3.0 MiB/s wr, 351 op/s 2024-05-09T09:27:18.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 506888 7.47 MB/sec execute 316 sec latency 13.112 ms 2024-05-09T09:27:19.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 508765 7.49 MB/sec execute 317 sec latency 13.156 ms 2024-05-09T09:27:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:19 smithi103 ceph-mon[34421]: pgmap v622: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-05-09T09:27:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:19 smithi069 ceph-mon[30123]: pgmap v622: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-05-09T09:27:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:19 smithi163 ceph-mon[34354]: pgmap v622: 129 pgs: 129 active+clean; 93 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.9 MiB/s wr, 370 op/s 2024-05-09T09:27:20.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 511689 7.52 MB/sec execute 318 sec latency 12.589 ms 2024-05-09T09:27:21.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 512839 7.51 MB/sec execute 319 sec latency 49.960 ms 2024-05-09T09:27:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:21 smithi103 ceph-mon[34421]: pgmap v623: 129 pgs: 129 active+clean; 92 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-05-09T09:27:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:21 smithi069 ceph-mon[30123]: pgmap v623: 129 pgs: 129 active+clean; 92 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-05-09T09:27:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:21 smithi163 ceph-mon[34354]: pgmap v623: 129 pgs: 129 active+clean; 92 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 647 KiB/s rd, 2.7 MiB/s wr, 332 op/s 2024-05-09T09:27:22.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 515369 7.53 MB/sec execute 320 sec latency 30.141 ms 2024-05-09T09:27:23.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 517091 7.53 MB/sec execute 321 sec latency 12.408 ms 2024-05-09T09:27:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:23 smithi103 ceph-mon[34421]: pgmap v624: 129 pgs: 129 active+clean; 91 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:27:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:23 smithi069 ceph-mon[30123]: pgmap v624: 129 pgs: 129 active+clean; 91 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:27:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:23 smithi163 ceph-mon[34354]: pgmap v624: 129 pgs: 129 active+clean; 91 MiB data, 1.8 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:27:24.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 519227 7.55 MB/sec execute 322 sec latency 11.291 ms 2024-05-09T09:27:25.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 521874 7.57 MB/sec execute 323 sec latency 14.409 ms 2024-05-09T09:27:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:25 smithi103 ceph-mon[34421]: pgmap v625: 129 pgs: 129 active+clean; 93 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.9 MiB/s wr, 368 op/s 2024-05-09T09:27:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:25 smithi069 ceph-mon[30123]: pgmap v625: 129 pgs: 129 active+clean; 93 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.9 MiB/s wr, 368 op/s 2024-05-09T09:27:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:25 smithi163 ceph-mon[34354]: pgmap v625: 129 pgs: 129 active+clean; 93 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.9 MiB/s wr, 368 op/s 2024-05-09T09:27:25.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 -- 172.21.15.69:0/3888607861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658100790 msgr2=0x7f7658100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 --2- 172.21.15.69:0/3888607861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658100790 0x7f7658100bb0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7f7654009900 tx=0x7f765402a410 comp rx=0 tx=0).stop 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 -- 172.21.15.69:0/3888607861 shutdown_connections 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 --2- 172.21.15.69:0/3888607861 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 0x7f7658108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 --2- 172.21.15.69:0/3888607861 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658101980 0x7f7658101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 --2- 172.21.15.69:0/3888607861 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658100790 0x7f7658100bb0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 -- 172.21.15.69:0/3888607861 >> 172.21.15.69:0/3888607861 conn(0x7f76580fbd00 msgr2=0x7f76580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 -- 172.21.15.69:0/3888607861 shutdown_connections 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 -- 172.21.15.69:0/3888607861 wait complete. 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.973+0000 7f765fe83640 1 Processor -- start 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- start start 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 0x7f7658196a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 0x7f765819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f765810c3e0 con 0x7f7658101980 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f765810c260 con 0x7f7658100790 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f765810c560 con 0x7f7658102340 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765dc1f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 0x7f7658196a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50922/0 (socket says 172.21.15.69:50922) 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765dc1f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 0x7f7658196a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36716/0 (socket says 172.21.15.69:36716) 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 -- 172.21.15.69:0/3508516093 learned_addr learned my addr 172.21.15.69:0/3508516093 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765e420640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 0x7f765819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 -- 172.21.15.69:0/3508516093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 msgr2=0x7f765819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 0x7f765819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 -- 172.21.15.69:0/3508516093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 msgr2=0x7f7658196a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 0x7f7658196a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 -- 172.21.15.69:0/3508516093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f765819e670 con 0x7f7658101980 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765d41e640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f764800e9f0 tx=0x7f764800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f764effd640 1 -- 172.21.15.69:0/3508516093 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764800cd80 con 0x7f7658101980 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f764effd640 1 -- 172.21.15.69:0/3508516093 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7648010910 con 0x7f7658101980 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f764effd640 1 -- 172.21.15.69:0/3508516093 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7648018990 con 0x7f7658101980 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f765819e960 con 0x7f7658101980 2024-05-09T09:27:25.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.974+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f765819eeb0 con 0x7f7658101980 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.977+0000 7f7662b8a740 1 Processor -- start 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.977+0000 7f764effd640 1 -- 172.21.15.69:0/3508516093 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7648010ac0 con 0x7f7658101980 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 -- start start 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee360560 0x55f2ee3609e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 0x55f2ee36c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee2c4340 con 0x55f2ee36a550 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee26c290 con 0x55f2ee3618d0 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f7662b8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee2c0300 con 0x55f2ee360560 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765e420640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 0x55f2ee36c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765e420640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36732/0 (socket says 172.21.15.69:36732) 2024-05-09T09:27:25.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765e420640 1 -- 172.21.15.69:0/1491199816 learned_addr learned my addr 172.21.15.69:0/1491199816 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765dc1f640 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee360560 0x55f2ee3609e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 -- 172.21.15.69:0/1491199816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee360560 msgr2=0x55f2ee3609e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee360560 0x55f2ee3609e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f764effd640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 0x7f76440a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 -- 172.21.15.69:0/1491199816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 msgr2=0x55f2ee360020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765d41e640 1 -- 172.21.15.69:0/1491199816 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ee2908e0 con 0x55f2ee36a550 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.978+0000 7f765e420640 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f765e420640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 0x7f76440a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f764effd640 1 -- 172.21.15.69:0/3508516093 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f76480cf6d0 con 0x7f7658101980 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f765d41e640 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 0x55f2ee36c940 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f76480cce20 tx=0x7f76480d5f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d9da512bb0a86aac server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f7643fff640 1 -- 172.21.15.69:0/1491199816 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764809ba30 con 0x55f2ee36a550 2024-05-09T09:27:25.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f7643fff640 1 -- 172.21.15.69:0/1491199816 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f764809bc60 con 0x55f2ee36a550 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.979+0000 7f7643fff640 1 -- 172.21.15.69:0/1491199816 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f764809b5c0 con 0x55f2ee36a550 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- 172.21.15.69:0/1491199816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 msgr2=0x55f2ee36c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 0x55f2ee36c940 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7f76480cce20 tx=0x7f76480d5f60 comp rx=0 tx=0).stop 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f765e420640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 0x7f76440a50f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55f2ee361460 tx=0x7f7650004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- 172.21.15.69:0/1491199816 shutdown_connections 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee36a550 0x55f2ee36c940 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee360560 0x55f2ee3609e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 --2- 172.21.15.69:0/1491199816 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee360020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- 172.21.15.69:0/1491199816 >> 172.21.15.69:0/1491199816 conn(0x55f2ee35b8c0 msgr2=0x55f2ee35bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- 172.21.15.69:0/1491199816 shutdown_connections 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- 172.21.15.69:0/1491199816 wait complete. 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 -- start start 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.980+0000 7f7662b8a740 1 Processor -- start 2024-05-09T09:27:25.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee375db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee36d090 con 0x55f2ee360560 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee27f790 con 0x55f2ee3618d0 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f2ee36d210 con 0x55f2ee36a550 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f7662b8a740 10 client.0 _ll_register_callbacks cb 0x55f2ee2baa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f765dc1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f765dc1f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50948/0 (socket says 172.21.15.69:50948) 2024-05-09T09:27:25.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.981+0000 7f765dc1f640 1 -- 172.21.15.69:0/293382803 learned_addr learned my addr 172.21.15.69:0/293382803 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.982+0000 7f765dc1f640 1 -- 172.21.15.69:0/293382803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 msgr2=0x55f2ee375db0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.982+0000 7f765dc1f640 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee375db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.982+0000 7f765dc1f640 1 -- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 msgr2=0x55f2ee36ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.982+0000 7f765dc1f640 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.982+0000 7f765dc1f640 1 -- 172.21.15.69:0/293382803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2edfd8210 con 0x55f2ee360560 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.983+0000 7f76417fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.983+0000 7f765dc1f640 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f76540314e0 tx=0x7f7654032e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c01c40b1fd48dd5a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.983+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7654035040 con 0x55f2ee360560 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.983+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f7654031970 con 0x55f2ee360560 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.983+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f764809bc60 con 0x55f2ee360560 2024-05-09T09:27:25.987 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.984+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f765404e050 con 0x55f2ee360560 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.984+0000 7f7662b8a740 10 client.16731 fetch_fsmap learned FSMap version 37 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.984+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f2ee0a4620 con 0x55f2ee360560 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f764809bc60 con 0x55f2ee360560 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f7654004920 con 0x55f2ee360560 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 10 client.16731 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 10 client.16731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 20 client.16731 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee38b380 0x55f2ee38d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 4 client.16731 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.985+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f2ee2d1a70 con 0x55f2ee38b380 2024-05-09T09:27:25.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.988+0000 7f765d41e640 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee38b380 0x55f2ee38d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:25.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.993+0000 7f765d41e640 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee38b380 0x55f2ee38d7b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=673f17baa162fd8f server_cookie=aa95f9190dd48865 in_seq=0 out_seq=0 2024-05-09T09:27:25.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.993+0000 7f76417fa640 10 client.16731 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:25.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.996+0000 7f76417fa640 1 -- 172.21.15.69:0/293382803 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f2ee2d1a70 con 0x55f2ee38b380 2024-05-09T09:27:25.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.996+0000 7f76417fa640 10 client.16731 handle_command_reply: tid=0 2024-05-09T09:27:25.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:25.997+0000 7f7662b8a740 1 client.16731 shutdown 2024-05-09T09:27:26.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 msgr2=0x55f2ee36e830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f76540314e0 tx=0x7f7654032e60 comp rx=0 tx=0).stop 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 shutdown_connections 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee38b380 0x55f2ee38d7b0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee375db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 --2- 172.21.15.69:0/293382803 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee360560 0x55f2ee36e830 secure :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7f76540314e0 tx=0x7f7654032e60 comp rx=0 tx=0).stop 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 >> 172.21.15.69:0/293382803 conn(0x55f2ee35b8c0 msgr2=0x55f2ee35aa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f76417fa640 0 client.16731 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.002+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 shutdown_connections 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.004+0000 7f7662b8a740 1 -- 172.21.15.69:0/293382803 wait complete. 2024-05-09T09:27:26.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.004+0000 7f7662b8a740 20 client.16731 trim_cache size 0 max 16384 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.043+0000 7f7662b8a740 1 Processor -- start 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.043+0000 7f7662b8a740 1 -- start start 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36e440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee36e980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76581aabf0 con 0x55f2ee366140 2024-05-09T09:27:26.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f765e420640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f765e420640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50962/0 (socket says 172.21.15.69:50962) 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f765e420640 1 -- 172.21.15.69:0/3932565990 learned_addr learned my addr 172.21.15.69:0/3932565990 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76581a84c0 con 0x55f2ee3618d0 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.044+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76581a62d0 con 0x55f2ee36a550 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.045+0000 7f765e420640 1 -- 172.21.15.69:0/3932565990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 msgr2=0x55f2ee36e980 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.045+0000 7f765e420640 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee36e980 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.045+0000 7f765e420640 1 -- 172.21.15.69:0/3932565990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 msgr2=0x55f2ee36e440 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.045+0000 7f765e420640 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36e440 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.045+0000 7f765e420640 1 -- 172.21.15.69:0/3932565990 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f2ee359f00 con 0x55f2ee366140 2024-05-09T09:27:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.046+0000 7f765e420640 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7650011630 tx=0x7f7650019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15f02bf2bdea6566 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.047+0000 7f76437fe640 1 -- 172.21.15.69:0/3932565990 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f765001e0d0 con 0x55f2ee366140 2024-05-09T09:27:26.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.047+0000 7f76437fe640 1 -- 172.21.15.69:0/3932565990 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f765001e270 con 0x55f2ee366140 2024-05-09T09:27:26.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.047+0000 7f76437fe640 1 -- 172.21.15.69:0/3932565990 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650005280 con 0x55f2ee366140 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 msgr2=0x55f2ee35a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7f7650011630 tx=0x7f7650019000 comp rx=0 tx=0).stop 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 shutdown_connections 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee366140 0x55f2ee35a170 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee36a550 0x55f2ee36e980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3932565990 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee3618d0 0x55f2ee36e440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 >> 172.21.15.69:0/3932565990 conn(0x55f2ee35b8c0 msgr2=0x55f2ee35bca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.048+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 shutdown_connections 2024-05-09T09:27:26.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.049+0000 7f7662b8a740 1 -- 172.21.15.69:0/3932565990 wait complete. 2024-05-09T09:27:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.049+0000 7f7662b8a740 1 -- start start 2024-05-09T09:27:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.049+0000 7f7662b8a740 1 Processor -- start 2024-05-09T09:27:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.050+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 0x55f2ee365bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.050+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee36a550 0x55f2ee373d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f7662b8a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f765e420640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f765dc1f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 0x55f2ee365bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f765e420640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50966/0 (socket says 172.21.15.69:50966) 2024-05-09T09:27:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f765d41e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee36a550 0x55f2ee373d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:26.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f765e420640 1 -- 172.21.15.69:0/3517454866 learned_addr learned my addr 172.21.15.69:0/3517454866 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76581a6980 con 0x55f2ee3742c0 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76581a7df0 con 0x55f2ee36a550 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76581a7050 con 0x55f2ee3618d0 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.051+0000 7f7662b8a740 10 client.0 _ll_register_callbacks cb 0x7f76500093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 -- 172.21.15.69:0/3517454866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 msgr2=0x55f2ee365bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 0x55f2ee365bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 -- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee36a550 msgr2=0x55f2ee373d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:26.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee36a550 0x55f2ee373d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f2ee374f70 con 0x55f2ee3742c0 2024-05-09T09:27:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765dc1f640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 0x55f2ee365bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f765e420640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7650011fd0 tx=0x7f765001eb50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=628d4a02c9b8598e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:26.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.052+0000 7f76417fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7650003bb0 con 0x55f2ee3742c0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(97..97 src has 1..97) ==== 9129+0+0 (secure 0 0 0) 0x7f7650004cc0 con 0x55f2ee3742c0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f765001e270 con 0x55f2ee3742c0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=37) ==== 24+0+0 (secure 0 0 0) 0x7f765003c050 con 0x55f2ee3742c0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f7662b8a740 10 client.16737 fetch_fsmap learned FSMap version 37 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.053+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=37,monmap=4+}) -- 0x55f2ee3ed910 con 0x55f2ee3742c0 2024-05-09T09:27:26.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.054+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7650005500 con 0x55f2ee3742c0 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.054+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 37) ==== 8346+0+0 (secure 0 0 0) 0x7f76500285b0 con 0x55f2ee3742c0 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 10 client.16737 fetch_fsmap finished waiting for FSMap version 37 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 10 client.16737 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 20 client.16737 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee3ee250 0x55f2ee3f0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 4 client.16737 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:27:26.054 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.055+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f765001e270 con 0x55f2ee3ee250 2024-05-09T09:27:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.056+0000 7f765dc1f640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee3ee250 0x55f2ee3f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.056+0000 7f765dc1f640 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee3ee250 0x55f2ee3f0670 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9856f6b08964159 server_cookie=12ab2a0a7bfddb56 in_seq=0 out_seq=0 2024-05-09T09:27:26.055 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.056+0000 7f76417fa640 10 client.16737 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:26.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f76417fa640 1 -- 172.21.15.69:0/3517454866 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f765001e270 con 0x55f2ee3ee250 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f76417fa640 10 client.16737 handle_command_reply: tid=0 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 client.16737 shutdown 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 msgr2=0x55f2ee374740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7f7650011fd0 tx=0x7f765001eb50 comp rx=0 tx=0).stop 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 shutdown_connections 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f2ee3742c0 0x55f2ee374740 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f2ee36a550 0x55f2ee373d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f2ee3ee250 0x55f2ee3f0670 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 --2- 172.21.15.69:0/3517454866 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f2ee3618d0 0x55f2ee365bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f76417fa640 0 client.16737 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 >> 172.21.15.69:0/3517454866 conn(0x55f2ee35b8c0 msgr2=0x55f2ee36e340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:26.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 shutdown_connections 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 1 -- 172.21.15.69:0/3517454866 wait complete. 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.229+0000 7f7662b8a740 20 client.16737 trim_cache size 0 max 16384 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:27:26.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 msgr2=0x7f76440a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 0x7f76440a50f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x55f2ee361460 tx=0x7f7650004a40 comp rx=0 tx=0).stop 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 msgr2=0x7f7658196f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7f764800e9f0 tx=0x7f764800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 shutdown_connections 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76440a2c30 0x7f76440a50f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7658102340 0x7f765819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7658101980 0x7f7658196f90 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 --2- 172.21.15.69:0/3508516093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7658100790 0x7f7658196a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 >> 172.21.15.69:0/3508516093 conn(0x7f76580fbd00 msgr2=0x7f7658104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 shutdown_connections 2024-05-09T09:27:26.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:26.230+0000 7f765fe83640 1 -- 172.21.15.69:0/3508516093 wait complete. 2024-05-09T09:27:26.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 522935 7.56 MB/sec execute 324 sec latency 70.508 ms 2024-05-09T09:27:27.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 525578 7.58 MB/sec execute 325 sec latency 34.361 ms 2024-05-09T09:27:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:27 smithi069 ceph-mon[30123]: daemon mds.c restarted 2024-05-09T09:27:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:27 smithi163 ceph-mon[34354]: daemon mds.c restarted 2024-05-09T09:27:27.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:27:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:27:27.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:27:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:27:27.556 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:27:27 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/4167992524 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x56500f3a3a40 con 0x56500f377000 2024-05-09T09:27:27.556 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:27:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:27:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:27 smithi103 ceph-mon[34421]: daemon mds.c restarted 2024-05-09T09:27:28.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 526948 7.58 MB/sec execute 326 sec latency 15.671 ms 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/3681372184,v1:172.21.15.163:6837/3681372184] up:boot 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:27:28.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:28 smithi069 ceph-mon[30123]: pgmap v627: 129 pgs: 129 active+clean; 99 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/3681372184,v1:172.21.15.163:6837/3681372184] up:boot 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:27:28.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:28 smithi163 ceph-mon[34354]: pgmap v627: 129 pgs: 129 active+clean; 99 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-05-09T09:27:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: osdmap e98: 12 total, 12 up, 12 in 2024-05-09T09:27:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/3681372184,v1:172.21.15.163:6837/3681372184] up:boot 2024-05-09T09:27:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 2 up:standby-replay 7 up:standby 2024-05-09T09:27:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:27:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:27:28.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:28 smithi103 ceph-mon[34421]: pgmap v627: 129 pgs: 129 active+clean; 99 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 654 KiB/s rd, 3.6 MiB/s wr, 321 op/s 2024-05-09T09:27:29.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 529577 7.61 MB/sec execute 327 sec latency 16.210 ms 2024-05-09T09:27:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:29 smithi103 ceph-mon[34421]: pgmap v628: 129 pgs: 129 active+clean; 102 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-05-09T09:27:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:29 smithi069 ceph-mon[30123]: pgmap v628: 129 pgs: 129 active+clean; 102 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-05-09T09:27:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:29 smithi163 ceph-mon[34354]: pgmap v628: 129 pgs: 129 active+clean; 102 MiB data, 1.9 GiB used, 1.0 TiB / 1.0 TiB avail; 572 KiB/s rd, 4.0 MiB/s wr, 335 op/s 2024-05-09T09:27:30.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 531287 7.60 MB/sec execute 328 sec latency 17.855 ms 2024-05-09T09:27:31.496 INFO:tasks.workunit.client.0.smithi069.stdout: 1 533276 7.62 MB/sec execute 329 sec latency 52.420 ms 2024-05-09T09:27:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:31 smithi103 ceph-mon[34421]: pgmap v629: 129 pgs: 129 active+clean; 107 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.5 MiB/s wr, 349 op/s 2024-05-09T09:27:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:31 smithi069 ceph-mon[30123]: pgmap v629: 129 pgs: 129 active+clean; 107 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.5 MiB/s wr, 349 op/s 2024-05-09T09:27:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:31 smithi163 ceph-mon[34354]: pgmap v629: 129 pgs: 129 active+clean; 107 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 4.5 MiB/s wr, 349 op/s 2024-05-09T09:27:32.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 535296 7.62 MB/sec execute 330 sec latency 39.109 ms 2024-05-09T09:27:33.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 537033 7.63 MB/sec execute 331 sec latency 11.886 ms 2024-05-09T09:27:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:33 smithi103 ceph-mon[34421]: pgmap v630: 129 pgs: 129 active+clean; 109 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 4.7 MiB/s wr, 398 op/s 2024-05-09T09:27:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:33 smithi069 ceph-mon[30123]: pgmap v630: 129 pgs: 129 active+clean; 109 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 4.7 MiB/s wr, 398 op/s 2024-05-09T09:27:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:33 smithi163 ceph-mon[34354]: pgmap v630: 129 pgs: 129 active+clean; 109 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 922 KiB/s rd, 4.7 MiB/s wr, 398 op/s 2024-05-09T09:27:34.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 539724 7.65 MB/sec execute 332 sec latency 13.670 ms 2024-05-09T09:27:35.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 540906 7.65 MB/sec execute 333 sec latency 49.319 ms 2024-05-09T09:27:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:35 smithi103 ceph-mon[34421]: pgmap v631: 129 pgs: 129 active+clean; 111 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.5 MiB/s wr, 343 op/s 2024-05-09T09:27:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:35 smithi069 ceph-mon[30123]: pgmap v631: 129 pgs: 129 active+clean; 111 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.5 MiB/s wr, 343 op/s 2024-05-09T09:27:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:35 smithi163 ceph-mon[34354]: pgmap v631: 129 pgs: 129 active+clean; 111 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 4.5 MiB/s wr, 343 op/s 2024-05-09T09:27:36.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 543601 7.67 MB/sec execute 334 sec latency 27.690 ms 2024-05-09T09:27:37.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 544704 7.66 MB/sec execute 335 sec latency 27.754 ms 2024-05-09T09:27:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:37 smithi103 ceph-mon[34421]: pgmap v632: 129 pgs: 129 active+clean; 112 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 4.1 MiB/s wr, 374 op/s 2024-05-09T09:27:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:37 smithi069 ceph-mon[30123]: pgmap v632: 129 pgs: 129 active+clean; 112 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 4.1 MiB/s wr, 374 op/s 2024-05-09T09:27:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:37 smithi163 ceph-mon[34354]: pgmap v632: 129 pgs: 129 active+clean; 112 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 886 KiB/s rd, 4.1 MiB/s wr, 374 op/s 2024-05-09T09:27:38.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 547366 7.69 MB/sec execute 336 sec latency 22.613 ms 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 -- 172.21.15.69:0/2797877616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc100790 msgr2=0x7f84dc100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 --2- 172.21.15.69:0/2797877616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc100790 0x7f84dc100bb0 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009900 tx=0x7f84d802a410 comp rx=0 tx=0).stop 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 -- 172.21.15.69:0/2797877616 shutdown_connections 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 --2- 172.21.15.69:0/2797877616 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc102340 0x7f84dc108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 --2- 172.21.15.69:0/2797877616 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc101980 0x7f84dc101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 --2- 172.21.15.69:0/2797877616 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc100790 0x7f84dc100bb0 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 -- 172.21.15.69:0/2797877616 >> 172.21.15.69:0/2797877616 conn(0x7f84dc0fbd00 msgr2=0x7f84dc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 -- 172.21.15.69:0/2797877616 shutdown_connections 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.323+0000 7f84e329e640 1 -- 172.21.15.69:0/2797877616 wait complete. 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 Processor -- start 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 -- start start 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc100790 0x7f84dc197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc102340 0x7f84dc19f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc10c400 con 0x7f84dc101980 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc10c280 con 0x7f84dc100790 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e329e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc10c580 con 0x7f84dc102340 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48584/0 (socket says 172.21.15.69:48584) 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 -- 172.21.15.69:0/929189526 learned_addr learned my addr 172.21.15.69:0/929189526 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e103a640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc100790 0x7f84dc197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e183b640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc102340 0x7f84dc19f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 -- 172.21.15.69:0/929189526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc102340 msgr2=0x7f84dc19f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc102340 0x7f84dc19f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 -- 172.21.15.69:0/929189526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc100790 msgr2=0x7f84dc197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc100790 0x7f84dc197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 -- 172.21.15.69:0/929189526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f84dc19f770 con 0x7f84dc101980 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.324+0000 7f84e0839640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d770 tx=0x7f84cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.325+0000 7f84d27fc640 1 -- 172.21.15.69:0/929189526 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc017070 con 0x7f84dc101980 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.325+0000 7f84d27fc640 1 -- 172.21.15.69:0/929189526 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84cc005500 con 0x7f84dc101980 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.325+0000 7f84d27fc640 1 -- 172.21.15.69:0/929189526 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc004e00 con 0x7f84dc101980 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.325+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f84dc19fa60 con 0x7f84dc101980 2024-05-09T09:27:39.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.325+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f84dc19ff60 con 0x7f84dc101980 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.327+0000 7f84d27fc640 1 -- 172.21.15.69:0/929189526 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f84cc004fa0 con 0x7f84dc101980 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 Processor -- start 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 -- start start 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 0x55b59f8fdf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f900aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b59f854340 con 0x55b59f8f9170 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b59f7fc290 con 0x55b59f8fe610 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e5faa740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b59f850300 con 0x55b59f8fbb30 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e183b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e103a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 0x55b59f8fdf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e0839640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f900aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e183b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48596/0 (socket says 172.21.15.69:48596) 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84e183b640 1 -- 172.21.15.69:0/2945924607 learned_addr learned my addr 172.21.15.69:0/2945924607 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.328+0000 7f84d27fc640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 0x7f84c80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 -- 172.21.15.69:0/2945924607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 msgr2=0x55b59f8fdf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 0x55b59f8fdf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 -- 172.21.15.69:0/2945924607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 msgr2=0x55b59f900aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f900aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 -- 172.21.15.69:0/2945924607 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b59f8208e0 con 0x55b59f8f9170 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e103a640 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 0x55b59f8fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84d27fc640 1 -- 172.21.15.69:0/929189526 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f84cc0cee40 con 0x7f84dc101980 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 0x7f84c80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f84d40104f0 tx=0x7f84d40109b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e14b5eb2b16b79 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84c77fe640 1 -- 172.21.15.69:0/2945924607 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4019070 con 0x55b59f8f9170 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84c77fe640 1 -- 172.21.15.69:0/2945924607 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d400ede0 con 0x55b59f8f9170 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84e183b640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 0x7f84c80a50f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f84d4021000 tx=0x7f84d401e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.329+0000 7f84c77fe640 1 -- 172.21.15.69:0/2945924607 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4011b40 con 0x55b59f8f9170 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- 172.21.15.69:0/2945924607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 msgr2=0x55b59f8fb560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7f84d40104f0 tx=0x7f84d40109b0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- 172.21.15.69:0/2945924607 shutdown_connections 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f900aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fbb30 0x55b59f8fdf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 --2- 172.21.15.69:0/2945924607 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8f9170 0x55b59f8fb560 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- 172.21.15.69:0/2945924607 >> 172.21.15.69:0/2945924607 conn(0x55b59f8eb840 msgr2=0x55b59f8ebc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- 172.21.15.69:0/2945924607 shutdown_connections 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- 172.21.15.69:0/2945924607 wait complete. 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 -- start start 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.330+0000 7f84e5faa740 1 Processor -- start 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f902c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 0x55b59f9031c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b59f9015c0 con 0x55b59f8fe610 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b59f80f790 con 0x55b59f8fbb30 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b59f901740 con 0x55b59f8f9170 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 10 client.0 _ll_register_callbacks cb 0x55b59f84aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e103a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f902c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e0839640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 0x55b59f9031c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48600/0 (socket says 172.21.15.69:48600) 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 -- 172.21.15.69:0/33576105 learned_addr learned my addr 172.21.15.69:0/33576105 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e103a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f902c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:44454/0 (socket says 172.21.15.69:44454) 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 -- 172.21.15.69:0/33576105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 msgr2=0x55b59f902c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f902c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 -- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 msgr2=0x55b59f9031c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 0x55b59f9031c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 -- 172.21.15.69:0/33576105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b59f568210 con 0x55b59f8fe610 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e0839640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 0x55b59f9031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e183b640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009fd0 tx=0x7f84d4011ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d37d5394c402d17f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:27:39.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d4019070 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f84d4011ad0 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.331+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f84d400ede0 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f84d4013580 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 10 client.16755 fetch_fsmap learned FSMap version 39 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b59f634620 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84d400ede0 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f84d402e070 con 0x55b59f8fe610 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 10 client.16755 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 10 client.16755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 20 client.16755 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f90e5e0 0x55b59f9109d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 4 client.16755 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b59f861a70 con 0x55b59f90e5e0 2024-05-09T09:27:39.340 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.332+0000 7f84e103a640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f90e5e0 0x55b59f9109d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84c4ff9640 10 client.16755 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e103a640 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f90e5e0 0x55b59f9109d0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d045a3f7d753ff server_cookie=cf4e30c06f8ffe47 in_seq=0 out_seq=0 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84c4ff9640 1 -- 172.21.15.69:0/33576105 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b59f861a70 con 0x55b59f90e5e0 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84c4ff9640 10 client.16755 handle_command_reply: tid=0 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 client.16755 shutdown 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 msgr2=0x55b59f90a200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7f84d4009fd0 tx=0x7f84d4011ed0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 shutdown_connections 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fe610 0x55b59f90a200 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fbb30 0x55b59f9031c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f90e5e0 0x55b59f9109d0 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 --2- 172.21.15.69:0/33576105 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f902c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84c4ff9640 0 client.16755 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.333+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 >> 172.21.15.69:0/33576105 conn(0x55b59f8eb840 msgr2=0x55b59f8f8800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.334+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 shutdown_connections 2024-05-09T09:27:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.334+0000 7f84e5faa740 1 -- 172.21.15.69:0/33576105 wait complete. 2024-05-09T09:27:39.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.334+0000 7f84e5faa740 20 client.16755 trim_cache size 0 max 16384 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 Processor -- start 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 -- start start 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8f9170 0x55b59f8fa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fe610 0x55b59f90a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1aaa30 con 0x55b59f8fbb30 2024-05-09T09:27:39.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1a8300 con 0x55b59f8f9170 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e5faa740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1a6150 con 0x55b59f8fe610 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48606/0 (socket says 172.21.15.69:48606) 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e0839640 1 -- 172.21.15.69:0/1843727789 learned_addr learned my addr 172.21.15.69:0/1843727789 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e103a640 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8f9170 0x55b59f8fa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.351+0000 7f84e183b640 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fe610 0x55b59f90a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 -- 172.21.15.69:0/1843727789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fe610 msgr2=0x55b59f90a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fe610 0x55b59f90a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 -- 172.21.15.69:0/1843727789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8f9170 msgr2=0x55b59f8fa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8f9170 0x55b59f8fa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 -- 172.21.15.69:0/1843727789 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b59f911000 con 0x55b59f8fbb30 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84e0839640 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f84cc097780 tx=0x7f84cc0ab040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d95fb0fb4d3d4aa9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84c6ffd640 1 -- 172.21.15.69:0/1843727789 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc020040 con 0x55b59f8fbb30 2024-05-09T09:27:39.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84c6ffd640 1 -- 172.21.15.69:0/1843727789 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f84cc0046c0 con 0x55b59f8fbb30 2024-05-09T09:27:39.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.352+0000 7f84c6ffd640 1 -- 172.21.15.69:0/1843727789 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84cc099620 con 0x55b59f8fbb30 2024-05-09T09:27:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- 172.21.15.69:0/1843727789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 msgr2=0x55b59f8fa6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7f84cc097780 tx=0x7f84cc0ab040 comp rx=0 tx=0).stop 2024-05-09T09:27:39.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- 172.21.15.69:0/1843727789 shutdown_connections 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8fe610 0x55b59f90a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f8fa6d0 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- 172.21.15.69:0/1843727789 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8f9170 0x55b59f8fa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- 172.21.15.69:0/1843727789 >> 172.21.15.69:0/1843727789 conn(0x55b59f8eb840 msgr2=0x55b59f902c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- 172.21.15.69:0/1843727789 shutdown_connections 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- 172.21.15.69:0/1843727789 wait complete. 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- start start 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 Processor -- start 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f907670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f907bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f908b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1a67c0 con 0x55b59f8fbb30 2024-05-09T09:27:39.358 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1a7c30 con 0x55b59f8fe610 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f84dc1a6e90 con 0x55b59f8f9170 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.353+0000 7f84e5faa740 10 client.0 _ll_register_callbacks cb 0x7f84d4020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f907bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f907670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e0839640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f907bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48610/0 (socket says 172.21.15.69:48610) 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e0839640 1 -- 172.21.15.69:0/3479014488 learned_addr learned my addr 172.21.15.69:0/3479014488 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e183b640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f908b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 -- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 msgr2=0x55b59f908b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f908b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 -- 172.21.15.69:0/3479014488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 msgr2=0x55b59f907bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f907bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 -- 172.21.15.69:0/3479014488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b59f909140 con 0x55b59f8f9170 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e103a640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f907670 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009230 tx=0x7f84d80041a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7f7e99da235513af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:39.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84c4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f84d8035040 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55b59f85bdc0 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f84d8046dc0 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f84d8055050 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e5faa740 10 client.25993 fetch_fsmap learned FSMap version 39 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.354+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55b59f90a4c0 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55b59f85bdc0 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f84d8004780 con 0x55b59f8f9170 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 10 client.25993 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 10 client.25993 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 20 client.25993 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f9751b0 0x55b59f9775d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:39.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 4 client.25993 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:27:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b59f7d98e0 con 0x55b59f9751b0 2024-05-09T09:27:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e0839640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f9751b0 0x55b59f9775d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84c4ff9640 10 client.25993 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:39.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.355+0000 7f84e0839640 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f9751b0 0x55b59f9775d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3318482231eef789 server_cookie=4bdea7b65e3032ad in_seq=0 out_seq=0 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84c4ff9640 1 -- 172.21.15.69:0/3479014488 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b59f7d98e0 con 0x55b59f9751b0 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84c4ff9640 10 client.25993 handle_command_reply: tid=0 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 client.25993 shutdown 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 msgr2=0x55b59f907670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f907670 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f84d8009230 tx=0x7f84d80041a0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 shutdown_connections 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b59f8fe610 0x55b59f908b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b59f8fbb30 0x55b59f907bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b59f9751b0 0x55b59f9775d0 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 --2- 172.21.15.69:0/3479014488 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b59f8f9170 0x55b59f907670 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84c4ff9640 0 client.25993 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 >> 172.21.15.69:0/3479014488 conn(0x55b59f8eb840 msgr2=0x55b59f902c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 shutdown_connections 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 1 -- 172.21.15.69:0/3479014488 wait complete. 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.466+0000 7f84e5faa740 20 client.25993 trim_cache size 0 max 16384 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:27:39.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:27:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 msgr2=0x7f84c80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 0x7f84c80a50f0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f84d4021000 tx=0x7f84d401e000 comp rx=0 tx=0).stop 2024-05-09T09:27:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 msgr2=0x7f84dc198090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7f84cc00d770 tx=0x7f84cc00dc30 comp rx=0 tx=0).stop 2024-05-09T09:27:39.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 shutdown_connections 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f84c80a2c30 0x7f84c80a50f0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f84dc102340 0x7f84dc19f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f84dc101980 0x7f84dc198090 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 --2- 172.21.15.69:0/929189526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f84dc100790 0x7f84dc197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 >> 172.21.15.69:0/929189526 conn(0x7f84dc0fbd00 msgr2=0x7f84dc068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 shutdown_connections 2024-05-09T09:27:39.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:39.467+0000 7f84e329e640 1 -- 172.21.15.69:0/929189526 wait complete. 2024-05-09T09:27:39.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 548774 7.68 MB/sec execute 337 sec latency 79.440 ms 2024-05-09T09:27:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:39 smithi103 ceph-mon[34421]: pgmap v633: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.6 MiB/s wr, 350 op/s 2024-05-09T09:27:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:39 smithi069 ceph-mon[30123]: pgmap v633: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.6 MiB/s wr, 350 op/s 2024-05-09T09:27:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:39 smithi163 ceph-mon[34354]: pgmap v633: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 911 KiB/s rd, 3.6 MiB/s wr, 350 op/s 2024-05-09T09:27:40.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 551077 7.70 MB/sec execute 338 sec latency 15.078 ms 2024-05-09T09:27:41.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 552999 7.70 MB/sec execute 339 sec latency 100.954 ms 2024-05-09T09:27:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:41 smithi103 ceph-mon[34421]: pgmap v634: 129 pgs: 129 active+clean; 115 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-09T09:27:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:41 smithi069 ceph-mon[30123]: pgmap v634: 129 pgs: 129 active+clean; 115 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-09T09:27:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:41 smithi163 ceph-mon[34354]: pgmap v634: 129 pgs: 129 active+clean; 115 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 806 KiB/s rd, 3.1 MiB/s wr, 293 op/s 2024-05-09T09:27:42.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 554735 7.71 MB/sec execute 340 sec latency 17.751 ms 2024-05-09T09:27:43.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 557159 7.72 MB/sec execute 341 sec latency 16.202 ms 2024-05-09T09:27:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:43 smithi103 ceph-mon[34421]: pgmap v635: 129 pgs: 129 active+clean; 113 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:27:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:43 smithi069 ceph-mon[30123]: pgmap v635: 129 pgs: 129 active+clean; 113 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:27:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:43 smithi163 ceph-mon[34354]: pgmap v635: 129 pgs: 129 active+clean; 113 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1005 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:27:44.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 558695 7.72 MB/sec execute 342 sec latency 12.172 ms 2024-05-09T09:27:45.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 561535 7.75 MB/sec execute 343 sec latency 12.581 ms 2024-05-09T09:27:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:45 smithi103 ceph-mon[34421]: pgmap v636: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:27:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:45 smithi069 ceph-mon[30123]: pgmap v636: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:27:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:45 smithi163 ceph-mon[34354]: pgmap v636: 129 pgs: 129 active+clean; 114 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 846 KiB/s rd, 3.0 MiB/s wr, 370 op/s 2024-05-09T09:27:46.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 563146 7.75 MB/sec execute 344 sec latency 36.462 ms 2024-05-09T09:27:47.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 565222 7.77 MB/sec execute 345 sec latency 41.579 ms 2024-05-09T09:27:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:47 smithi103 ceph-mon[34421]: pgmap v637: 129 pgs: 129 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-05-09T09:27:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:47 smithi069 ceph-mon[30123]: pgmap v637: 129 pgs: 129 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-05-09T09:27:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:47 smithi163 ceph-mon[34354]: pgmap v637: 129 pgs: 129 active+clean; 118 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 3.0 MiB/s wr, 346 op/s 2024-05-09T09:27:48.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 567390 7.77 MB/sec execute 346 sec latency 27.978 ms 2024-05-09T09:27:49.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 569136 7.78 MB/sec execute 347 sec latency 13.393 ms 2024-05-09T09:27:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:49 smithi103 ceph-mon[34421]: pgmap v638: 129 pgs: 129 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-05-09T09:27:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:49 smithi069 ceph-mon[30123]: pgmap v638: 129 pgs: 129 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-05-09T09:27:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:49 smithi163 ceph-mon[34354]: pgmap v638: 129 pgs: 129 active+clean; 120 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 356 op/s 2024-05-09T09:27:50.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 572044 7.80 MB/sec execute 348 sec latency 12.100 ms 2024-05-09T09:27:51.497 INFO:tasks.workunit.client.0.smithi069.stdout: 1 573340 7.80 MB/sec execute 349 sec latency 30.207 ms 2024-05-09T09:27:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:51 smithi069 ceph-mon[30123]: pgmap v639: 129 pgs: 129 active+clean; 121 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-05-09T09:27:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:51 smithi163 ceph-mon[34354]: pgmap v639: 129 pgs: 129 active+clean; 121 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-05-09T09:27:51.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:51 smithi103 ceph-mon[34421]: pgmap v639: 129 pgs: 129 active+clean; 121 MiB data, 2.0 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.8 MiB/s wr, 337 op/s 2024-05-09T09:27:52.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 575719 7.82 MB/sec execute 350 sec latency 51.543 ms 2024-05-09T09:27:52.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 -- 172.21.15.69:0/1398753377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881019a0 msgr2=0x7fb688101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 --2- 172.21.15.69:0/1398753377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881019a0 0x7fb688101e20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fb678009860 tx=0x7fb67802a340 comp rx=0 tx=0).stop 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 -- 172.21.15.69:0/1398753377 shutdown_connections 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 --2- 172.21.15.69:0/1398753377 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb688108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 --2- 172.21.15.69:0/1398753377 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881019a0 0x7fb688101e20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 --2- 172.21.15.69:0/1398753377 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881007b0 0x7fb688100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 -- 172.21.15.69:0/1398753377 >> 172.21.15.69:0/1398753377 conn(0x7fb6880fbd40 msgr2=0x7fb6880fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 -- 172.21.15.69:0/1398753377 shutdown_connections 2024-05-09T09:27:52.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.585+0000 7fb69044d640 1 -- 172.21.15.69:0/1398753377 wait complete. 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 Processor -- start 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 -- start start 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881007b0 0x7fb688199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881019a0 0x7fb68819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb68810b390 con 0x7fb688102360 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb68810b210 con 0x7fb6881019a0 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.586+0000 7fb69044d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb68810b510 con 0x7fb6881007b0 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48666/0 (socket says 172.21.15.69:48666) 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 -- 172.21.15.69:0/2885939833 learned_addr learned my addr 172.21.15.69:0/2885939833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881007b0 0x7fb688199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68d9e8640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881019a0 0x7fb68819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 -- 172.21.15.69:0/2885939833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881007b0 msgr2=0x7fb688199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881007b0 0x7fb688199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 -- 172.21.15.69:0/2885939833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881019a0 msgr2=0x7fb68819a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881019a0 0x7fb68819a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 -- 172.21.15.69:0/2885939833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb6881a1960 con 0x7fb688102360 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.587+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb68000efc0 tx=0x7fb68000c800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.588+0000 7fb67f7fe640 1 -- 172.21.15.69:0/2885939833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb680003970 con 0x7fb688102360 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.588+0000 7fb67f7fe640 1 -- 172.21.15.69:0/2885939833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb680004020 con 0x7fb688102360 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.588+0000 7fb67f7fe640 1 -- 172.21.15.69:0/2885939833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68000b9f0 con 0x7fb688102360 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.588+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6881a1c50 con 0x7fb688102360 2024-05-09T09:27:52.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.588+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6881a2090 con 0x7fb688102360 2024-05-09T09:27:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.592+0000 7fb692dfd740 1 Processor -- start 2024-05-09T09:27:52.592 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.592+0000 7fb67f7fe640 1 -- 172.21.15.69:0/2885939833 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb68001d020 con 0x7fb688102360 2024-05-09T09:27:52.593 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.593+0000 7fb692dfd740 1 -- start start 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb67f7fe640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 0x7fb6740a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d2b4529340 con 0x55d2b45c5250 2024-05-09T09:27:52.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2b44d1290 con 0x55d2b45cadb0 2024-05-09T09:27:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb692dfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d2b4525300 con 0x55d2b45c5c40 2024-05-09T09:27:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.594+0000 7fb67f7fe640 1 -- 172.21.15.69:0/2885939833 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb6800cf290 con 0x7fb688102360 2024-05-09T09:27:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48678/0 (socket says 172.21.15.69:48678) 2024-05-09T09:27:52.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1030205802 learned_addr learned my addr 172.21.15.69:0/1030205802 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1030205802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 msgr2=0x55d2b45ca740 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45ca740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1030205802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 msgr2=0x55d2b45cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1030205802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b44f58e0 con 0x55d2b45c5250 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.595+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:27:52.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.596+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb684009c60 tx=0x7fb684011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80707bbd2fd7caec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.596+0000 7fb68d9e8640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 0x7fb6740a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.596+0000 7fb67cbfe640 1 -- 172.21.15.69:0/1030205802 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684004220 con 0x55d2b45c5250 2024-05-09T09:27:52.597 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.596+0000 7fb67cbfe640 1 -- 172.21.15.69:0/1030205802 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6840043c0 con 0x55d2b45c5250 2024-05-09T09:27:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.597+0000 7fb67cbfe640 1 -- 172.21.15.69:0/1030205802 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684004df0 con 0x55d2b45c5250 2024-05-09T09:27:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.597+0000 7fb692dfd740 1 -- 172.21.15.69:0/1030205802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 msgr2=0x55d2b45c5670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.597+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fb684009c60 tx=0x7fb684011810 comp rx=0 tx=0).stop 2024-05-09T09:27:52.598 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.598+0000 7fb68d9e8640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 0x7fb6740a51f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb678009830 tx=0x7fb678030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 -- 172.21.15.69:0/1030205802 shutdown_connections 2024-05-09T09:27:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1030205802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c5670 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 -- 172.21.15.69:0/1030205802 >> 172.21.15.69:0/1030205802 conn(0x55d2b45c08c0 msgr2=0x55d2b45c0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 -- 172.21.15.69:0/1030205802 shutdown_connections 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 -- 172.21.15.69:0/1030205802 wait complete. 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.600+0000 7fb692dfd740 1 -- start start 2024-05-09T09:27:52.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 Processor -- start 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45cf4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5c40 0x55d2b45cf9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d2b45cddb0 con 0x55d2b45c5c40 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d2b44e4790 con 0x55d2b45c5250 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.601+0000 7fb692dfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d2b45cdf30 con 0x55d2b45cadb0 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb692dfd740 10 client.0 _ll_register_callbacks cb 0x55d2b451fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:50782/0 (socket says 172.21.15.69:50782) 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 -- 172.21.15.69:0/1317100633 learned_addr learned my addr 172.21.15.69:0/1317100633 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:52.603 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68d9e8640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5c40 0x55d2b45cf9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45cf4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 -- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 msgr2=0x55d2b45cf4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45cf4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 -- 172.21.15.69:0/1317100633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5c40 msgr2=0x55d2b45cf9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5c40 0x55d2b45cf9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.602+0000 7fb68e9ea640 1 -- 172.21.15.69:0/1317100633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b423d210 con 0x55d2b45cadb0 2024-05-09T09:27:52.604 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb68009b640 tx=0x7fb6800a4440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=157d18b439cdd95f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb671ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68009cd90 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb6840043c0 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb6800860c0 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb6800862c0 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb692dfd740 10 client.26002 fetch_fsmap learned FSMap version 39 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.603+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d2b4309620 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb6840043c0 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fb68000b040 con 0x55d2b45cadb0 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb692dfd740 10 client.26002 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb692dfd740 10 client.26002 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb692dfd740 20 client.26002 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.604+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b45dacb0 0x55d2b45dd120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.605+0000 7fb692dfd740 4 client.26002 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:27:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.605+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2b4536a70 con 0x55d2b45dacb0 2024-05-09T09:27:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.605+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b45dacb0 0x55d2b45dd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.607+0000 7fb671ffb640 10 client.26002 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:52.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.608+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b45dacb0 0x55d2b45dd120 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fa266737f5ef50ea server_cookie=b6a37df174b7c4d7 in_seq=0 out_seq=0 2024-05-09T09:27:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.609+0000 7fb671ffb640 1 -- 172.21.15.69:0/1317100633 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2b4536a70 con 0x55d2b45dacb0 2024-05-09T09:27:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.609+0000 7fb671ffb640 10 client.26002 handle_command_reply: tid=0 2024-05-09T09:27:52.609 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.609+0000 7fb692dfd740 1 client.26002 shutdown 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 msgr2=0x55d2b45d6a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fb68009b640 tx=0x7fb6800a4440 comp rx=0 tx=0).stop 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 shutdown_connections 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b45dacb0 0x55d2b45dd120 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45cadb0 0x55d2b45d6a20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5c40 0x55d2b45cf9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1317100633 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45cf4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.611 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.610+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 >> 172.21.15.69:0/1317100633 conn(0x55d2b45c08c0 msgr2=0x55d2b45bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.611+0000 7fb671ffb640 0 client.26002 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.611+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 shutdown_connections 2024-05-09T09:27:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.612+0000 7fb692dfd740 1 -- 172.21.15.69:0/1317100633 wait complete. 2024-05-09T09:27:52.612 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.612+0000 7fb692dfd740 20 client.26002 trim_cache size 0 max 16384 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 Processor -- start 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 -- start start 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45c7330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c0124e0 con 0x55d2b45c5250 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c00fdb0 con 0x55d2b45cadb0 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb692dfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c00dc00 con 0x55d2b45c5c40 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48690/0 (socket says 172.21.15.69:48690) 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1872627131 learned_addr learned my addr 172.21.15.69:0/1872627131 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68d9e8640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45c7330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.637 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1872627131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 msgr2=0x55d2b45d69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1872627131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 msgr2=0x55d2b45c7330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45c7330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e1e9640 1 -- 172.21.15.69:0/1872627131 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2b45c6020 con 0x55d2b45c5250 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.636+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.637+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb6840099f0 tx=0x7fb6840048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea15801b2ac89b0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.637+0000 7fb673fff640 1 -- 172.21.15.69:0/1872627131 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb68401d040 con 0x55d2b45c5250 2024-05-09T09:27:52.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.637+0000 7fb673fff640 1 -- 172.21.15.69:0/1872627131 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb684026e00 con 0x55d2b45c5250 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.637+0000 7fb673fff640 1 -- 172.21.15.69:0/1872627131 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb684024ab0 con 0x55d2b45c5250 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- 172.21.15.69:0/1872627131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 msgr2=0x55d2b45c6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fb6840099f0 tx=0x7fb6840048b0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- 172.21.15.69:0/1872627131 shutdown_connections 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45cadb0 0x55d2b45c7330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- 172.21.15.69:0/1872627131 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45c5250 0x55d2b45c6df0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- 172.21.15.69:0/1872627131 >> 172.21.15.69:0/1872627131 conn(0x55d2b45c08c0 msgr2=0x55d2b45ca3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- 172.21.15.69:0/1872627131 shutdown_connections 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- 172.21.15.69:0/1872627131 wait complete. 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- start start 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 Processor -- start 2024-05-09T09:27:52.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45d68f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d33e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c00e270 con 0x55d2b45cadb0 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c00f6e0 con 0x55d2b45c5250 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb66c00e940 con 0x55d2b45c5c40 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb692dfd740 10 client.0 _ll_register_callbacks cb 0x7fb6800099a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb68e9ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48694/0 (socket says 172.21.15.69:48694) 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb68e1e9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45d68f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb68d9e8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.638+0000 7fb68e9ea640 1 -- 172.21.15.69:0/818574478 learned_addr learned my addr 172.21.15.69:0/818574478 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 -- 172.21.15.69:0/818574478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 msgr2=0x55d2b45d33e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d33e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 -- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 msgr2=0x55d2b45d68f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45d68f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 -- 172.21.15.69:0/818574478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2b463f2e0 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb68e9ea640 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb68009b8d0 tx=0x7fb6800a1190 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7ae925dcd5fe74f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb671ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb6800b9060 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb68009cb90 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb684026e00 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fb68009b710 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb692dfd740 10 client.16779 fetch_fsmap learned FSMap version 39 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55d2b4640610 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.639+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb684026e00 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fb68009ae20 con 0x55d2b45cadb0 2024-05-09T09:27:52.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 10 client.16779 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 10 client.16779 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 20 client.16779 populate_metadata read hostname 'smithi069' 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b4640e60 0x55d2b4643280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 4 client.16779 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d2b4530dc0 con 0x55d2b4640e60 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b4640e60 0x55d2b4643280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb671ffb640 10 client.16779 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:52.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.640+0000 7fb68e1e9640 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b4640e60 0x55d2b4643280 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8995fc4b8577e8ff server_cookie=915bc07ef809025b in_seq=0 out_seq=0 2024-05-09T09:27:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb671ffb640 1 -- 172.21.15.69:0/818574478 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d2b4530dc0 con 0x55d2b4640e60 2024-05-09T09:27:52.769 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb671ffb640 10 client.16779 handle_command_reply: tid=0 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 client.16779 shutdown 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 msgr2=0x55d2b45d3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fb68009b8d0 tx=0x7fb6800a1190 comp rx=0 tx=0).stop 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 shutdown_connections 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d2b45cadb0 0x55d2b45d3920 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2b4640e60 0x55d2b4643280 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2b45c5c40 0x55d2b45d33e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb671ffb640 0 client.16779 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 --2- 172.21.15.69:0/818574478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2b45c5250 0x55d2b45d68f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.768+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 >> 172.21.15.69:0/818574478 conn(0x55d2b45c08c0 msgr2=0x55d2b45cf3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 shutdown_connections 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb692dfd740 1 -- 172.21.15.69:0/818574478 wait complete. 2024-05-09T09:27:52.770 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb692dfd740 20 client.16779 trim_cache size 0 max 16384 2024-05-09T09:27:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:27:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:27:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:27:52.771 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 msgr2=0x7fb6740a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 0x7fb6740a51f0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7fb678009830 tx=0x7fb678030550 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 msgr2=0x7fb6881a12c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fb68000efc0 tx=0x7fb68000c800 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 shutdown_connections 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6740a2d30 0x7fb6740a51f0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb688102360 0x7fb6881a12c0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb6881019a0 0x7fb68819a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 --2- 172.21.15.69:0/2885939833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb6881007b0 0x7fb688199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.769+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 >> 172.21.15.69:0/2885939833 conn(0x7fb6880fbd40 msgr2=0x7fb6880fc740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.770+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 shutdown_connections 2024-05-09T09:27:52.772 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:27:52.770+0000 7fb69044d640 1 -- 172.21.15.69:0/2885939833 wait complete. 2024-05-09T09:27:53.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 577043 7.81 MB/sec execute 351 sec latency 73.671 ms 2024-05-09T09:27:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:53 smithi069 ceph-mon[30123]: pgmap v640: 129 pgs: 129 active+clean; 119 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-05-09T09:27:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:53 smithi163 ceph-mon[34354]: pgmap v640: 129 pgs: 129 active+clean; 119 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-05-09T09:27:53.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:53 smithi103 ceph-mon[34421]: pgmap v640: 129 pgs: 129 active+clean; 119 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 3.1 MiB/s wr, 340 op/s 2024-05-09T09:27:54.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 579404 7.83 MB/sec execute 352 sec latency 11.338 ms 2024-05-09T09:27:55.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 581742 7.84 MB/sec execute 353 sec latency 14.671 ms 2024-05-09T09:27:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:55 smithi069 ceph-mon[30123]: pgmap v641: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-05-09T09:27:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:55 smithi163 ceph-mon[34354]: pgmap v641: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-05-09T09:27:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:55 smithi103 ceph-mon[34421]: pgmap v641: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 2.8 MiB/s wr, 347 op/s 2024-05-09T09:27:56.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 583329 7.84 MB/sec execute 354 sec latency 27.036 ms 2024-05-09T09:27:57.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 585890 7.86 MB/sec execute 355 sec latency 54.705 ms 2024-05-09T09:27:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:57 smithi103 ceph-mon[34421]: pgmap v642: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:27:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:57 smithi069 ceph-mon[30123]: pgmap v642: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:27:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:57 smithi163 ceph-mon[34354]: pgmap v642: 129 pgs: 129 active+clean; 121 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 3.0 MiB/s wr, 345 op/s 2024-05-09T09:27:58.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 587064 7.85 MB/sec execute 356 sec latency 31.477 ms 2024-05-09T09:27:59.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 589858 7.88 MB/sec execute 357 sec latency 17.075 ms 2024-05-09T09:27:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:27:59 smithi103 ceph-mon[34421]: pgmap v643: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 383 op/s 2024-05-09T09:27:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:27:59 smithi069 ceph-mon[30123]: pgmap v643: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 383 op/s 2024-05-09T09:27:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:27:59 smithi163 ceph-mon[34354]: pgmap v643: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 995 KiB/s rd, 3.0 MiB/s wr, 383 op/s 2024-05-09T09:28:00.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 591396 7.87 MB/sec execute 358 sec latency 13.853 ms 2024-05-09T09:28:01.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 593573 7.89 MB/sec execute 359 sec latency 17.850 ms 2024-05-09T09:28:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:01 smithi069 ceph-mon[30123]: pgmap v644: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-05-09T09:28:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:01 smithi163 ceph-mon[34354]: pgmap v644: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-05-09T09:28:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:01 smithi103 ceph-mon[34421]: pgmap v644: 129 pgs: 129 active+clean; 123 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 705 KiB/s rd, 2.6 MiB/s wr, 317 op/s 2024-05-09T09:28:02.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 595414 7.88 MB/sec execute 360 sec latency 52.941 ms 2024-05-09T09:28:03.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 597140 7.90 MB/sec execute 361 sec latency 19.602 ms 2024-05-09T09:28:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:03 smithi069 ceph-mon[30123]: pgmap v645: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.9 MiB/s wr, 336 op/s 2024-05-09T09:28:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:03 smithi163 ceph-mon[34354]: pgmap v645: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.9 MiB/s wr, 336 op/s 2024-05-09T09:28:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:03 smithi103 ceph-mon[34421]: pgmap v645: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 807 KiB/s rd, 2.9 MiB/s wr, 336 op/s 2024-05-09T09:28:04.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 599429 7.91 MB/sec execute 362 sec latency 15.237 ms 2024-05-09T09:28:05.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 601044 7.91 MB/sec execute 363 sec latency 12.719 ms 2024-05-09T09:28:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:05 smithi069 ceph-mon[30123]: pgmap v646: 129 pgs: 129 active+clean; 124 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-09T09:28:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:05 smithi163 ceph-mon[34354]: pgmap v646: 129 pgs: 129 active+clean; 124 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-09T09:28:05.871 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:05 smithi103 ceph-mon[34421]: pgmap v646: 129 pgs: 129 active+clean; 124 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 892 KiB/s rd, 2.7 MiB/s wr, 352 op/s 2024-05-09T09:28:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/858147690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac102ea0 msgr2=0x7fe7ac1032c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/858147690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac1032c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fe79c009900 tx=0x7fe79c02a410 comp rx=0 tx=0).stop 2024-05-09T09:28:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/858147690 shutdown_connections 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/858147690 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/858147690 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac0ff910 0x7fe7ac0ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/858147690 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac1032c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/858147690 >> 172.21.15.69:0/858147690 conn(0x7fe7ac0fb4b0 msgr2=0x7fe7ac0fd8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/858147690 shutdown_connections 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.865+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/858147690 wait complete. 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 Processor -- start 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 -- start start 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac0ff910 0x7fe7ac197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac10d520 con 0x7fe7ac1002d0 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac10d3a0 con 0x7fe7ac102ea0 2024-05-09T09:28:05.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7b1b9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac10d6a0 con 0x7fe7ac0ff910 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44414/0 (socket says 172.21.15.69:44414) 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7a3fff640 1 -- 172.21.15.69:0/3230107981 learned_addr learned my addr 172.21.15.69:0/3230107981 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac0ff910 0x7fe7ac197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.866+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 -- 172.21.15.69:0/3230107981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac0ff910 msgr2=0x7fe7ac197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac0ff910 0x7fe7ac197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 -- 172.21.15.69:0/3230107981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac102ea0 msgr2=0x7fe7ac19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 -- 172.21.15.69:0/3230107981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe7ac19f750 con 0x7fe7ac1002d0 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe79400e9f0 tx=0x7fe79400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a97fa640 1 -- 172.21.15.69:0/3230107981 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79400cd80 con 0x7fe7ac1002d0 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe7ac19fa40 con 0x7fe7ac1002d0 2024-05-09T09:28:05.882 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a97fa640 1 -- 172.21.15.69:0/3230107981 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe794010910 con 0x7fe7ac1002d0 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7a97fa640 1 -- 172.21.15.69:0/3230107981 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7940188f0 con 0x7fe7ac1002d0 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.867+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe7ac19ff90 con 0x7fe7ac1002d0 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.869+0000 7fe7a97fa640 1 -- 172.21.15.69:0/3230107981 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe794010ac0 con 0x7fe7ac1002d0 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 Processor -- start 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 -- start start 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.883 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c14740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c17240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7a97fa640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 0x7fe7980a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5602f7b73340 con 0x5602f7c14db0 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5602f7b1b290 con 0x5602f7c0fc40 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7b4988740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602f7b6f300 con 0x5602f7c0f250 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 0x7fe7980a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c17240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:48210/0 (socket says 172.21.15.69:48210) 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 -- 172.21.15.69:0/1691412507 learned_addr learned my addr 172.21.15.69:0/1691412507 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:05.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7a97fa640 1 -- 172.21.15.69:0/3230107981 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe7940cf770 con 0x7fe7ac1002d0 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c14740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 -- 172.21.15.69:0/1691412507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 msgr2=0x5602f7c14740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c14740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 -- 172.21.15.69:0/1691412507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 msgr2=0x5602f7c17240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c17240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.870+0000 7fe7abfff640 1 -- 172.21.15.69:0/1691412507 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f7b3f8e0 con 0x5602f7c0f250 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.871+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 0x7fe7980a50f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400fc30 tx=0x7fe7a4017000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.871+0000 7fe7abfff640 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4012d90 tx=0x7fe7a400f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3c3a863b5d860f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.871+0000 7fe7a1ffb640 1 -- 172.21.15.69:0/1691412507 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a40125a0 con 0x5602f7c0f250 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.871+0000 7fe7a1ffb640 1 -- 172.21.15.69:0/1691412507 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a4003e50 con 0x5602f7c0f250 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.871+0000 7fe7a1ffb640 1 -- 172.21.15.69:0/1691412507 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4004320 con 0x5602f7c0f250 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- 172.21.15.69:0/1691412507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 msgr2=0x5602f7c0f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4012d90 tx=0x7fe7a400f730 comp rx=0 tx=0).stop 2024-05-09T09:28:05.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- 172.21.15.69:0/1691412507 shutdown_connections 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c17240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c14740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1691412507 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c0f670 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- 172.21.15.69:0/1691412507 >> 172.21.15.69:0/1691412507 conn(0x5602f7c0a8c0 msgr2=0x5602f7c0acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- 172.21.15.69:0/1691412507 shutdown_connections 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- 172.21.15.69:0/1691412507 wait complete. 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- start start 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 Processor -- start 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c19400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c19940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5602f7c17d00 con 0x5602f7c14db0 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5602f7b2e790 con 0x5602f7c0fc40 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.872+0000 7fe7b4988740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5602f7c17e80 con 0x5602f7c0f250 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7b4988740 10 client.0 _ll_register_callbacks cb 0x5602f7b69a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7ab7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c19400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44436/0 (socket says 172.21.15.69:44436) 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 -- 172.21.15.69:0/3464421252 learned_addr learned my addr 172.21.15.69:0/3464421252 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c19940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 -- 172.21.15.69:0/3464421252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 msgr2=0x5602f7c19400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c19400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 -- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 msgr2=0x5602f7c19940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c19940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 -- 172.21.15.69:0/3464421252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f7887210 con 0x5602f7c14db0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c19940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7abfff640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006010 tx=0x7fe7a400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98b4a9b1971149fd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe78affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7a4003b80 con 0x5602f7c14db0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe7a4005120 con 0x5602f7c14db0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe7a4003e50 con 0x5602f7c14db0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe7a4038050 con 0x5602f7c14db0 2024-05-09T09:28:05.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7b4988740 10 client.16791 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.873+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5602f7953620 con 0x5602f7c14db0 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe7a4003e50 con 0x5602f7c14db0 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fe7a400f040 con 0x5602f7c14db0 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 10 client.16791 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 10 client.16791 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 20 client.16791 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c27060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 4 client.16791 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5602f7b80a70 con 0x5602f7c24c70 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c27060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe78affd640 10 client.16791 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.874+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c27060 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10553ae9d5c9180c server_cookie=173cc2774f13b688 in_seq=0 out_seq=0 2024-05-09T09:28:05.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.875+0000 7fe78affd640 1 -- 172.21.15.69:0/3464421252 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5602f7b80a70 con 0x5602f7c24c70 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.875+0000 7fe78affd640 10 client.16791 handle_command_reply: tid=0 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.875+0000 7fe7b4988740 1 client.16791 shutdown 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.875+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 msgr2=0x5602f7c20980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.875+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7fe7a4006010 tx=0x7fe7a400cce0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 shutdown_connections 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c14db0 0x5602f7c20980 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c0fc40 0x5602f7c19940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c27060 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe78affd640 0 client.16791 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 --2- 172.21.15.69:0/3464421252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c19400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 >> 172.21.15.69:0/3464421252 conn(0x5602f7c0a8c0 msgr2=0x5602f7c09a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:05.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 shutdown_connections 2024-05-09T09:28:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 1 -- 172.21.15.69:0/3464421252 wait complete. 2024-05-09T09:28:05.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.876+0000 7fe7b4988740 20 client.16791 trim_cache size 0 max 16384 2024-05-09T09:28:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 Processor -- start 2024-05-09T09:28:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 -- start start 2024-05-09T09:28:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0fc40 0x5602f7c20940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c096b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1acd10 con 0x5602f7c0f250 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1aa5e0 con 0x5602f7c14db0 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7b4988740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1a8430 con 0x5602f7c0fc40 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44444/0 (socket says 172.21.15.69:44444) 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 -- 172.21.15.69:0/675081062 learned_addr learned my addr 172.21.15.69:0/675081062 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7abfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c096b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0fc40 0x5602f7c20940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 -- 172.21.15.69:0/675081062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0fc40 msgr2=0x5602f7c20940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0fc40 0x5602f7c20940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 -- 172.21.15.69:0/675081062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 msgr2=0x5602f7c096b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c096b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 -- 172.21.15.69:0/675081062 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5602f7c08f00 con 0x5602f7c0f250 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7abfff640 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.893+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe79c005ab0 tx=0x7fe79c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61d43cfde291065e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7a17fa640 1 -- 172.21.15.69:0/675081062 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c035040 con 0x5602f7c0f250 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7a17fa640 1 -- 172.21.15.69:0/675081062 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe79c04a070 con 0x5602f7c0f250 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7a17fa640 1 -- 172.21.15.69:0/675081062 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe79c0054a0 con 0x5602f7c0f250 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 -- 172.21.15.69:0/675081062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 msgr2=0x5602f7c09170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:05.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7fe79c005ab0 tx=0x7fe79c000bc0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 -- 172.21.15.69:0/675081062 shutdown_connections 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0fc40 0x5602f7c20940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 --2- 172.21.15.69:0/675081062 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0f250 0x5602f7c09170 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 -- 172.21.15.69:0/675081062 >> 172.21.15.69:0/675081062 conn(0x5602f7c0a8c0 msgr2=0x5602f7c10e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.894+0000 7fe7b4988740 1 -- 172.21.15.69:0/675081062 shutdown_connections 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 -- 172.21.15.69:0/675081062 wait complete. 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 -- start start 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 Processor -- start 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c1dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c1f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1a8aa0 con 0x5602f7c0fc40 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1a9f10 con 0x5602f7c14db0 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe7ac1a9170 con 0x5602f7c0f250 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7b4988740 10 client.0 _ll_register_callbacks cb 0x7fe794096680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7a3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44452/0 (socket says 172.21.15.69:44452) 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7a3fff640 1 -- 172.21.15.69:0/1202219980 learned_addr learned my addr 172.21.15.69:0/1202219980 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c1dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.895+0000 7fe7abfff640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c1f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 -- 172.21.15.69:0/1202219980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 msgr2=0x5602f7c1dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c1dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 -- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 msgr2=0x5602f7c1f2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:05.903 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c1f2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 -- 172.21.15.69:0/1202219980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5602f7c1f8d0 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7a3fff640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe79409bc90 tx=0x7fe7940911e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21398b500cecda81 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe7940a4630 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe79409b150 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe79c04a070 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fe7940b1070 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 10 client.16797 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5602f7c20ce0 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe79c04a070 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fe79406bce0 con 0x5602f7c0fc40 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 10 client.16797 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:05.904 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 10 client.16797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 20 client.16797 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c8da10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 4 client.16797 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.896+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5602f7b7adc0 con 0x5602f7c24c70 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.897+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c8da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.897+0000 7fe78affd640 10 client.16797 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:05.905 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:05.897+0000 7fe7ab7fe640 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c8da10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eceeaf02209061bf server_cookie=a137d6cdad11f276 in_seq=0 out_seq=0 2024-05-09T09:28:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe78affd640 1 -- 172.21.15.69:0/1202219980 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5602f7b7adc0 con 0x5602f7c24c70 2024-05-09T09:28:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe78affd640 10 client.16797 handle_command_reply: tid=0 2024-05-09T09:28:06.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 client.16797 shutdown 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 msgr2=0x5602f7c1e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7fe79409bc90 tx=0x7fe7940911e0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 shutdown_connections 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5602f7c14db0 0x5602f7c1f2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5602f7c0fc40 0x5602f7c1e2d0 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5602f7c24c70 0x5602f7c8da10 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 --2- 172.21.15.69:0/1202219980 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5602f7c0f250 0x5602f7c1dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe78affd640 0 client.16797 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.027+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 >> 172.21.15.69:0/1202219980 conn(0x5602f7c0a8c0 msgr2=0x5602f7c192d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 shutdown_connections 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b4988740 1 -- 172.21.15.69:0/1202219980 wait complete. 2024-05-09T09:28:06.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b4988740 20 client.16797 trim_cache size 0 max 16384 2024-05-09T09:28:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:28:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:28:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:28:06.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 msgr2=0x7fe7980a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 0x7fe7980a50f0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7fe7a400fc30 tx=0x7fe7a4017000 comp rx=0 tx=0).stop 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 msgr2=0x7fe7ac198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7fe79400e9f0 tx=0x7fe79400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 shutdown_connections 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe7980a2c30 0x7fe7980a50f0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe7ac102ea0 0x7fe7ac19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe7ac1002d0 0x7fe7ac198070 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 --2- 172.21.15.69:0/3230107981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe7ac0ff910 0x7fe7ac197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 >> 172.21.15.69:0/3230107981 conn(0x7fe7ac0fb4b0 msgr2=0x7fe7ac105f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 shutdown_connections 2024-05-09T09:28:06.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:06.028+0000 7fe7b1b9e640 1 -- 172.21.15.69:0/3230107981 wait complete. 2024-05-09T09:28:06.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 603448 7.92 MB/sec execute 364 sec latency 82.015 ms 2024-05-09T09:28:06.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:06 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:28:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:06 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:28:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:06 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:28:07.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 604558 7.92 MB/sec execute 365 sec latency 76.973 ms 2024-05-09T09:28:07.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:28:07.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:28:07.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:07 smithi103 ceph-mon[34421]: pgmap v647: 129 pgs: 129 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.8 MiB/s wr, 331 op/s 2024-05-09T09:28:07.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:07 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:28:08.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:28:08.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:28:08.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:07 smithi069 ceph-mon[30123]: pgmap v647: 129 pgs: 129 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.8 MiB/s wr, 331 op/s 2024-05-09T09:28:08.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:07 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:28:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:28:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:28:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:07 smithi163 ceph-mon[34354]: pgmap v647: 129 pgs: 129 active+clean; 125 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.8 MiB/s wr, 331 op/s 2024-05-09T09:28:08.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:07 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:28:08.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 606977 7.93 MB/sec execute 366 sec latency 33.350 ms 2024-05-09T09:28:09.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 608288 7.93 MB/sec execute 367 sec latency 18.574 ms 2024-05-09T09:28:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:09 smithi069 ceph-mon[30123]: pgmap v648: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:28:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:09 smithi163 ceph-mon[34354]: pgmap v648: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:28:09.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:09 smithi103 ceph-mon[34421]: pgmap v648: 129 pgs: 129 active+clean; 122 MiB data, 2.1 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:28:10.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 611115 7.95 MB/sec execute 368 sec latency 13.656 ms 2024-05-09T09:28:11.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 612534 7.95 MB/sec execute 369 sec latency 20.054 ms 2024-05-09T09:28:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:11 smithi069 ceph-mon[30123]: pgmap v649: 129 pgs: 129 active+clean; 122 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-05-09T09:28:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:11 smithi163 ceph-mon[34354]: pgmap v649: 129 pgs: 129 active+clean; 122 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-05-09T09:28:11.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:11 smithi103 ceph-mon[34421]: pgmap v649: 129 pgs: 129 active+clean; 122 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 698 KiB/s rd, 2.4 MiB/s wr, 309 op/s 2024-05-09T09:28:12.511 INFO:tasks.workunit.client.0.smithi069.stdout: 1 614749 7.96 MB/sec execute 370 sec latency 60.550 ms 2024-05-09T09:28:13.498 INFO:tasks.workunit.client.0.smithi069.stdout: 1 616059 7.96 MB/sec execute 371 sec latency 32.486 ms 2024-05-09T09:28:13.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:13 smithi103 ceph-mon[34421]: pgmap v650: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:28:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:13 smithi069 ceph-mon[30123]: pgmap v650: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:28:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:13 smithi163 ceph-mon[34354]: pgmap v650: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 917 KiB/s rd, 2.9 MiB/s wr, 373 op/s 2024-05-09T09:28:14.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 618376 7.97 MB/sec execute 372 sec latency 12.306 ms 2024-05-09T09:28:15.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 620543 7.97 MB/sec execute 373 sec latency 15.377 ms 2024-05-09T09:28:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:15 smithi069 ceph-mon[30123]: pgmap v651: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:28:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:15 smithi163 ceph-mon[34354]: pgmap v651: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:28:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:15 smithi103 ceph-mon[34421]: pgmap v651: 129 pgs: 129 active+clean; 125 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:28:16.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 622250 7.98 MB/sec execute 374 sec latency 13.770 ms 2024-05-09T09:28:17.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 624720 8.00 MB/sec execute 375 sec latency 44.096 ms 2024-05-09T09:28:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:17 smithi069 ceph-mon[30123]: pgmap v652: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 3.0 MiB/s wr, 379 op/s 2024-05-09T09:28:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:17 smithi163 ceph-mon[34354]: pgmap v652: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 3.0 MiB/s wr, 379 op/s 2024-05-09T09:28:17.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:17 smithi103 ceph-mon[34421]: pgmap v652: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 3.0 MiB/s wr, 379 op/s 2024-05-09T09:28:18.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 625818 7.99 MB/sec execute 376 sec latency 30.978 ms 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- 172.21.15.69:0/1713746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101320 msgr2=0x7f57f4108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 --2- 172.21.15.69:0/1713746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101320 0x7f57f4108be0 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f57ec009ca0 tx=0x7f57ec02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- 172.21.15.69:0/1713746141 shutdown_connections 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 --2- 172.21.15.69:0/1713746141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101320 0x7f57f4108be0 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 --2- 172.21.15.69:0/1713746141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f4100de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 --2- 172.21.15.69:0/1713746141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101d30 0x7f57f4102150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- 172.21.15.69:0/1713746141 >> 172.21.15.69:0/1713746141 conn(0x7f57f40fb490 msgr2=0x7f57f40fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- 172.21.15.69:0/1713746141 shutdown_connections 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- 172.21.15.69:0/1713746141 wait complete. 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 Processor -- start 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.119+0000 7f57f8aee640 1 -- start start 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101320 0x7f57f4196f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57f410c3f0 con 0x7f57f4101d30 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57f410c270 con 0x7f57f4100980 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f410c570 con 0x7f57f4101320 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101320 0x7f57f4196f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:48920/0 (socket says 172.21.15.69:48920) 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59028/0 (socket says 172.21.15.69:59028) 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 -- 172.21.15.69:0/971361259 learned_addr learned my addr 172.21.15.69:0/971361259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 -- 172.21.15.69:0/971361259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101320 msgr2=0x7f57f4196f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101320 0x7f57f4196f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 -- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 msgr2=0x7f57f419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 -- 172.21.15.69:0/971361259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f57f419e5f0 con 0x7f57f4100980 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f259c640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57e400e9f0 tx=0x7f57e400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57eb7fe640 1 -- 172.21.15.69:0/971361259 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e400cd80 con 0x7f57f4100980 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f57f419e8e0 con 0x7f57f4100980 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57eb7fe640 1 -- 172.21.15.69:0/971361259 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57e4005500 con 0x7f57f4100980 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.120+0000 7f57eb7fe640 1 -- 172.21.15.69:0/971361259 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57e4010690 con 0x7f57f4100980 2024-05-09T09:28:19.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.121+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f57f419ee30 con 0x7f57f4100980 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57eb7fe640 1 -- 172.21.15.69:0/971361259 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f57e40046c0 con 0x7f57f4100980 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 Processor -- start 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 -- start start 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eec0480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eec0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55842ee29340 con 0x55842eec1d30 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55842edd1290 con 0x55842eecf570 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.123+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55842ee25300 con 0x55842eec09c0 2024-05-09T09:28:19.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eec0480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57f259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eec0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:48936/0 (socket says 172.21.15.69:48936) 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eec0480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59034/0 (socket says 172.21.15.69:59034) 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 -- 172.21.15.69:0/2710801874 learned_addr learned my addr 172.21.15.69:0/2710801874 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57f259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eec0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53590/0 (socket says 172.21.15.69:53590) 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57eb7fe640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 0x7f57e00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 -- 172.21.15.69:0/2710801874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 msgr2=0x55842eec0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eec0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 -- 172.21.15.69:0/2710801874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 msgr2=0x55842eec0480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eec0480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 -- 172.21.15.69:0/2710801874 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842edf58e0 con 0x55842eecf570 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57eb7fe640 1 -- 172.21.15.69:0/971361259 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f57e40cf7a0 con 0x7f57f4100980 2024-05-09T09:28:19.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 0x7f57e00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009c90 tx=0x7f57dc011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72a23776a5476995 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57e8ff9640 1 -- 172.21.15.69:0/2710801874 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01e070 con 0x55842eecf570 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57e8ff9640 1 -- 172.21.15.69:0/2710801874 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc019b10 con 0x55842eecf570 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.124+0000 7f57e8ff9640 1 -- 172.21.15.69:0/2710801874 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc00ea70 con 0x55842eecf570 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2710801874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 msgr2=0x55842eed1960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 0x7f57e00a50f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f57ec005e30 tx=0x7f57ec0095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f57dc009c90 tx=0x7f57dc011810 comp rx=0 tx=0).stop 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2710801874 shutdown_connections 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed1960 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eec0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2710801874 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eec0480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2710801874 >> 172.21.15.69:0/2710801874 conn(0x55842eec2ad0 msgr2=0x55842eec2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2710801874 shutdown_connections 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2710801874 wait complete. 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.125+0000 7f57fb7fb740 1 -- start start 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 Processor -- start 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55842eed2110 con 0x55842eecf570 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55842ede4790 con 0x55842eec1d30 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55842eed2290 con 0x55842eec09c0 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57fb7fb740 10 client.0 _ll_register_callbacks cb 0x55842ee1fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:48950/0 (socket says 172.21.15.69:48950) 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f259c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53596/0 (socket says 172.21.15.69:53596) 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 -- 172.21.15.69:0/1931573984 learned_addr learned my addr 172.21.15.69:0/1931573984 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f2d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59048/0 (socket says 172.21.15.69:59048) 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 -- 172.21.15.69:0/1931573984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 msgr2=0x55842eed37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 -- 172.21.15.69:0/1931573984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 msgr2=0x55842eedad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 -- 172.21.15.69:0/1931573984 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842eb3d210 con 0x55842eec1d30 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f259c640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57ebfff640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f57dc019710 tx=0x7f57dc019740 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d3e8002df92b0126 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.126+0000 7f57d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:28:19.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc029030 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57dc019b10 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f57dc004020 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f57dc033050 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 10 client.26120 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55842ec09620 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc019b10 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f57dc00e790 con 0x55842eec1d30 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 10 client.26120 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 10 client.26120 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 20 client.26120 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842eedf080 0x55842eee1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 4 client.26120 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55842ee36a70 con 0x55842eedf080 2024-05-09T09:28:19.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.127+0000 7f57f259c640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842eedf080 0x55842eee1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.128+0000 7f57d9ffb640 10 client.26120 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.128+0000 7f57f259c640 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842eedf080 0x55842eee1470 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fe184c3fe220531 server_cookie=c63d9236cfa18316 in_seq=0 out_seq=0 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.128+0000 7f57d9ffb640 1 -- 172.21.15.69:0/1931573984 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55842ee36a70 con 0x55842eedf080 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.128+0000 7f57d9ffb640 10 client.26120 handle_command_reply: tid=0 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.128+0000 7f57fb7fb740 1 client.26120 shutdown 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 msgr2=0x55842eed3d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f57dc019710 tx=0x7f57dc019740 comp rx=0 tx=0).stop 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 shutdown_connections 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eecf570 0x55842eedad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec1d30 0x55842eed3d00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842eedf080 0x55842eee1470 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/1931573984 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec09c0 0x55842eed37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57d9ffb640 0 client.26120 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:19.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 >> 172.21.15.69:0/1931573984 conn(0x55842eec2ad0 msgr2=0x55842eebfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 shutdown_connections 2024-05-09T09:28:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 1 -- 172.21.15.69:0/1931573984 wait complete. 2024-05-09T09:28:19.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.129+0000 7f57fb7fb740 20 client.26120 trim_cache size 0 max 16384 2024-05-09T09:28:19.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 Processor -- start 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 -- start start 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec09c0 0x55842eed2de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eecf570 0x55842eed37c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41aaa70 con 0x55842eec1d30 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41a8340 con 0x55842eec09c0 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.144+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41a6190 con 0x55842eecf570 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59050/0 (socket says 172.21.15.69:59050) 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 -- 172.21.15.69:0/2555041958 learned_addr learned my addr 172.21.15.69:0/2555041958 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eecf570 0x55842eed37c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57f259c640 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec09c0 0x55842eed2de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 -- 172.21.15.69:0/2555041958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eecf570 msgr2=0x55842eed37c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eecf570 0x55842eed37c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 -- 172.21.15.69:0/2555041958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec09c0 msgr2=0x55842eed2de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec09c0 0x55842eed2de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 -- 172.21.15.69:0/2555041958 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55842eebef00 con 0x55842eec1d30 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57ebfff640 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb80 tx=0x7f57dc005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f88613df72f0ee2a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57dbfff640 1 -- 172.21.15.69:0/2555041958 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc029070 con 0x55842eec1d30 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57dbfff640 1 -- 172.21.15.69:0/2555041958 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc005040 con 0x55842eec1d30 2024-05-09T09:28:19.147 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.145+0000 7f57dbfff640 1 -- 172.21.15.69:0/2555041958 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc01e040 con 0x55842eec1d30 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2555041958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 msgr2=0x55842eedad00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb80 tx=0x7f57dc005fd0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2555041958 shutdown_connections 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eecf570 0x55842eed37c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec1d30 0x55842eedad00 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/2555041958 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eec09c0 0x55842eed2de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2555041958 >> 172.21.15.69:0/2555041958 conn(0x55842eec2ad0 msgr2=0x55842eec2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2555041958 shutdown_connections 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- 172.21.15.69:0/2555041958 wait complete. 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- start start 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 Processor -- start 2024-05-09T09:28:19.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 0x55842eed7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed7ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41a6800 con 0x55842eec09c0 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.146+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41a7c70 con 0x55842eecf570 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57fb7fb740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f57f41a6ed0 con 0x55842eec1d30 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57fb7fb740 10 client.0 _ll_register_callbacks cb 0x7f57dc01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57f259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57f259c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59060/0 (socket says 172.21.15.69:59060) 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 0x55842eed7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57f2d9d640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed7ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57f259c640 1 -- 172.21.15.69:0/3383145926 learned_addr learned my addr 172.21.15.69:0/3383145926 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:19.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 -- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 msgr2=0x55842eed7ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed7ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 -- 172.21.15.69:0/3383145926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 msgr2=0x55842eedaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 -- 172.21.15.69:0/3383145926 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55842eed80a0 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57f259c640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57ebfff640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 0x55842eed7570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb00 tx=0x7f57dc0054a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ae4a92d60ab4b3d4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57d9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f57dc03d070 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f57dc00e7c0 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f57dc005040 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f57dc049050 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57fb7fb740 10 client.26044 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.147+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55842ef44610 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f57dc005040 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f57dc0046f0 con 0x55842eec1d30 2024-05-09T09:28:19.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 10 client.26044 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 10 client.26044 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 20 client.26044 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842ef44fb0 0x55842ef473d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 4 client.26044 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55842ee30dc0 con 0x55842ef44fb0 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57f259c640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842ef44fb0 0x55842ef473d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57d9ffb640 10 client.26044 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:19.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.148+0000 7f57f259c640 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842ef44fb0 0x55842ef473d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=84f95fa58c09aa48 server_cookie=96c413d26ad4d88a in_seq=0 out_seq=0 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57d9ffb640 1 -- 172.21.15.69:0/3383145926 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55842ee30dc0 con 0x55842ef44fb0 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57d9ffb640 10 client.26044 handle_command_reply: tid=0 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 client.26044 shutdown 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 msgr2=0x55842eed7570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 0x55842eed7570 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f57dc00eb00 tx=0x7f57dc0054a0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 shutdown_connections 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55842eecf570 0x55842eed7ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55842ef44fb0 0x55842ef473d0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55842eec1d30 0x55842eed7570 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 --2- 172.21.15.69:0/3383145926 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55842eec09c0 0x55842eedaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57d9ffb640 0 client.26044 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 >> 172.21.15.69:0/3383145926 conn(0x55842eec2ad0 msgr2=0x55842eec5850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 shutdown_connections 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.256+0000 7f57fb7fb740 1 -- 172.21.15.69:0/3383145926 wait complete. 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57fb7fb740 20 client.26044 trim_cache size 0 max 16384 2024-05-09T09:28:19.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:28:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:28:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:28:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:28:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 msgr2=0x7f57e00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 0x7f57e00a50f0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f57ec005e30 tx=0x7f57ec0095c0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 msgr2=0x7f57f41969d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f57e400e9f0 tx=0x7f57e400edf0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 shutdown_connections 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f57e00a2c30 0x7f57e00a50f0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f57f4101d30 0x7f57f419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f57f4101320 0x7f57f4196f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 --2- 172.21.15.69:0/971361259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f57f4100980 0x7f57f41969d0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 >> 172.21.15.69:0/971361259 conn(0x7f57f40fb490 msgr2=0x7f57f4104c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 shutdown_connections 2024-05-09T09:28:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:19.257+0000 7f57f8aee640 1 -- 172.21.15.69:0/971361259 wait complete. 2024-05-09T09:28:19.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 628244 8.00 MB/sec execute 377 sec latency 61.375 ms 2024-05-09T09:28:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:19 smithi069 ceph-mon[30123]: pgmap v653: 129 pgs: 129 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-05-09T09:28:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:19 smithi163 ceph-mon[34354]: pgmap v653: 129 pgs: 129 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-05-09T09:28:19.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:19 smithi103 ceph-mon[34421]: pgmap v653: 129 pgs: 129 active+clean; 128 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 854 KiB/s rd, 2.8 MiB/s wr, 330 op/s 2024-05-09T09:28:20.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 629540 8.00 MB/sec execute 378 sec latency 13.842 ms 2024-05-09T09:28:21.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 632375 8.03 MB/sec execute 379 sec latency 13.120 ms 2024-05-09T09:28:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:21 smithi069 ceph-mon[30123]: pgmap v654: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.3 MiB/s wr, 259 op/s 2024-05-09T09:28:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:21 smithi163 ceph-mon[34354]: pgmap v654: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.3 MiB/s wr, 259 op/s 2024-05-09T09:28:21.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:21 smithi103 ceph-mon[34421]: pgmap v654: 129 pgs: 129 active+clean; 127 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 661 KiB/s rd, 2.3 MiB/s wr, 259 op/s 2024-05-09T09:28:22.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 633551 8.02 MB/sec execute 380 sec latency 43.782 ms 2024-05-09T09:28:23.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 635931 8.03 MB/sec execute 381 sec latency 32.020 ms 2024-05-09T09:28:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:23 smithi069 ceph-mon[30123]: pgmap v655: 129 pgs: 129 active+clean; 130 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-09T09:28:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:23 smithi163 ceph-mon[34354]: pgmap v655: 129 pgs: 129 active+clean; 130 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-09T09:28:23.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:23 smithi103 ceph-mon[34421]: pgmap v655: 129 pgs: 129 active+clean; 130 MiB data, 2.2 GiB used, 1.0 TiB / 1.0 TiB avail; 982 KiB/s rd, 3.0 MiB/s wr, 364 op/s 2024-05-09T09:28:24.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 637239 8.03 MB/sec execute 382 sec latency 19.885 ms 2024-05-09T09:28:25.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 639630 8.04 MB/sec execute 383 sec latency 10.914 ms 2024-05-09T09:28:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:25 smithi069 ceph-mon[30123]: pgmap v656: 129 pgs: 129 active+clean; 126 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.5 MiB/s wr, 315 op/s 2024-05-09T09:28:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:25 smithi163 ceph-mon[34354]: pgmap v656: 129 pgs: 129 active+clean; 126 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.5 MiB/s wr, 315 op/s 2024-05-09T09:28:25.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:25 smithi103 ceph-mon[34421]: pgmap v656: 129 pgs: 129 active+clean; 126 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 753 KiB/s rd, 2.5 MiB/s wr, 315 op/s 2024-05-09T09:28:26.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 641646 8.04 MB/sec execute 384 sec latency 15.140 ms 2024-05-09T09:28:27.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 643281 8.05 MB/sec execute 385 sec latency 65.274 ms 2024-05-09T09:28:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:27 smithi069 ceph-mon[30123]: pgmap v657: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-05-09T09:28:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:27 smithi163 ceph-mon[34354]: pgmap v657: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-05-09T09:28:27.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:27 smithi103 ceph-mon[34421]: pgmap v657: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.9 MiB/s wr, 359 op/s 2024-05-09T09:28:28.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 645217 8.05 MB/sec execute 386 sec latency 42.770 ms 2024-05-09T09:28:29.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 646947 8.06 MB/sec execute 387 sec latency 15.449 ms 2024-05-09T09:28:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:29 smithi069 ceph-mon[30123]: pgmap v658: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:28:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:29 smithi163 ceph-mon[34354]: pgmap v658: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:28:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:29 smithi103 ceph-mon[34421]: pgmap v658: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:28:30.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 649359 8.07 MB/sec execute 388 sec latency 16.850 ms 2024-05-09T09:28:31.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 650700 8.07 MB/sec execute 389 sec latency 52.612 ms 2024-05-09T09:28:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:31 smithi069 ceph-mon[30123]: pgmap v659: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-05-09T09:28:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:31 smithi163 ceph-mon[34354]: pgmap v659: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-05-09T09:28:31.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:31 smithi103 ceph-mon[34421]: pgmap v659: 129 pgs: 129 active+clean; 129 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 631 KiB/s rd, 2.5 MiB/s wr, 287 op/s 2024-05-09T09:28:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- 172.21.15.69:0/3861017535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4101980 msgr2=0x7f1da4101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3861017535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4101980 0x7f1da4101e00 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f1d94009900 tx=0x7f1d9402a410 comp rx=0 tx=0).stop 2024-05-09T09:28:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- 172.21.15.69:0/3861017535 shutdown_connections 2024-05-09T09:28:32.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3861017535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4102340 0x7f1da4108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3861017535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4101980 0x7f1da4101e00 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3861017535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 0x7f1da4100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- 172.21.15.69:0/3861017535 >> 172.21.15.69:0/3861017535 conn(0x7f1da40fbd00 msgr2=0x7f1da40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- 172.21.15.69:0/3861017535 shutdown_connections 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- 172.21.15.69:0/3861017535 wait complete. 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 Processor -- start 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- start start 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 0x7f1da4104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4101980 0x7f1da4102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410c3f0 con 0x7f1da4102340 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410c270 con 0x7f1da4101980 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.352+0000 7f1daac4d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1da410c570 con 0x7f1da4100790 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39604/0 (socket says 172.21.15.69:39604) 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 -- 172.21.15.69:0/3102389795 learned_addr learned my addr 172.21.15.69:0/3102389795 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da89e9640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 0x7f1da4104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4101980 0x7f1da4102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 -- 172.21.15.69:0/3102389795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 msgr2=0x7f1da4104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 0x7f1da4104720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 -- 172.21.15.69:0/3102389795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4101980 msgr2=0x7f1da4102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4101980 0x7f1da4102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 -- 172.21.15.69:0/3102389795 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da4103b30 con 0x7f1da4102340 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.356 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1d99ffb640 1 -- 172.21.15.69:0/3102389795 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c00ccc0 con 0x7f1da4102340 2024-05-09T09:28:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da41a1c20 con 0x7f1da4102340 2024-05-09T09:28:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1d99ffb640 1 -- 172.21.15.69:0/3102389795 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d9c00ce60 con 0x7f1da4102340 2024-05-09T09:28:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1d99ffb640 1 -- 172.21.15.69:0/3102389795 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c010490 con 0x7f1da4102340 2024-05-09T09:28:32.357 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.353+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da41a2140 con 0x7f1da4102340 2024-05-09T09:28:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.355+0000 7f1d99ffb640 1 -- 172.21.15.69:0/3102389795 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1d9c005020 con 0x7f1da4102340 2024-05-09T09:28:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 Processor -- start 2024-05-09T09:28:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 -- start start 2024-05-09T09:28:32.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d99ffb640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 0x7f1d8c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397da9740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397dac240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563397d08340 con 0x563397da4250 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563397cb0290 con 0x563397da9db0 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1dada39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563397d04300 con 0x563397da4c40 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1da89e9640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 0x7f1d8c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39608/0 (socket says 172.21.15.69:39608) 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397dac240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 -- 172.21.15.69:0/2841506476 learned_addr learned my addr 172.21.15.69:0/2841506476 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d99ffb640 1 -- 172.21.15.69:0/3102389795 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1d9c0cf860 con 0x7f1da4102340 2024-05-09T09:28:32.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397da9740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 -- 172.21.15.69:0/2841506476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 msgr2=0x563397da9740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397da9740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 -- 172.21.15.69:0/2841506476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 msgr2=0x563397dac240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397dac240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.356+0000 7f1d9bfff640 1 -- 172.21.15.69:0/2841506476 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563397cd48e0 con 0x563397da4250 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1d94031610 tx=0x7f1d94005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2f318967fddd9ef server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1da89e9640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 0x7f1d8c0a50f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1da41017e0 tx=0x7f1da0009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1d8affd640 1 -- 172.21.15.69:0/2841506476 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94044030 con 0x563397da4250 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1d8affd640 1 -- 172.21.15.69:0/2841506476 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d940317b0 con 0x563397da4250 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1d8affd640 1 -- 172.21.15.69:0/2841506476 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d94035040 con 0x563397da4250 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 -- 172.21.15.69:0/2841506476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 msgr2=0x563397da4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f1d94031610 tx=0x7f1d94005b50 comp rx=0 tx=0).stop 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 -- 172.21.15.69:0/2841506476 shutdown_connections 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397dac240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397da9740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 --2- 172.21.15.69:0/2841506476 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4250 0x563397da4670 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 -- 172.21.15.69:0/2841506476 >> 172.21.15.69:0/2841506476 conn(0x563397d9f8c0 msgr2=0x563397d9fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 -- 172.21.15.69:0/2841506476 shutdown_connections 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.357+0000 7f1dada39740 1 -- 172.21.15.69:0/2841506476 wait complete. 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 -- start start 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 Processor -- start 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397dae440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397db59c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563397dacd00 con 0x563397da4c40 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563397cc3790 con 0x563397da9db0 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563397dace80 con 0x563397da4250 2024-05-09T09:28:32.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1dada39740 10 client.0 _ll_register_callbacks cb 0x563397cfea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1da89e9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397dae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39624/0 (socket says 172.21.15.69:39624) 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1d9bfff640 1 -- 172.21.15.69:0/2207647974 learned_addr learned my addr 172.21.15.69:0/2207647974 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.358+0000 7f1da91ea640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397db59c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 -- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 msgr2=0x563397db59c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397db59c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 -- 172.21.15.69:0/2207647974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 msgr2=0x563397dae980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 -- 172.21.15.69:0/2207647974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563397a1c210 con 0x563397da4250 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da91ea640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397db59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397dae440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1da0005fd0 tx=0x7f1da0004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=80f38e35e82ecbc7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d83fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1da0023070 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1da0004b20 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d940317b0 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1da002e050 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1dada39740 10 client.26056 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563397ae8620 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.359+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d940317b0 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f1da000f040 con 0x563397da4250 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 10 client.26056 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 10 client.26056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 20 client.26056 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397db9f90 0x563397dbc430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 4 client.26056 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563397d15a70 con 0x563397db9f90 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397db9f90 0x563397dbc430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397db9f90 0x563397dbc430 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a639668aec137ce0 server_cookie=a96dff2dc362c6a2 in_seq=0 out_seq=0 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.360+0000 7f1d83fff640 10 client.26056 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.361+0000 7f1d83fff640 1 -- 172.21.15.69:0/2207647974 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563397d15a70 con 0x563397db9f90 2024-05-09T09:28:32.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.361+0000 7f1d83fff640 10 client.26056 handle_command_reply: tid=0 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.361+0000 7f1dada39740 1 client.26056 shutdown 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.361+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 msgr2=0x563397dae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.361+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397dae440 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1da0005fd0 tx=0x7f1da0004a40 comp rx=0 tx=0).stop 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 shutdown_connections 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da9db0 0x563397db59c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da4c40 0x563397dae980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397db9f90 0x563397dbc430 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 --2- 172.21.15.69:0/2207647974 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397dae440 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 >> 172.21.15.69:0/2207647974 conn(0x563397d9f8c0 msgr2=0x563397d9ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1d83fff640 0 client.26056 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 shutdown_connections 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 1 -- 172.21.15.69:0/2207647974 wait complete. 2024-05-09T09:28:32.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.362+0000 7f1dada39740 20 client.26056 trim_cache size 0 max 16384 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 Processor -- start 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 -- start start 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397d9fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4c40 0x563397db5980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41aee20 con 0x563397da9db0 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41ac6f0 con 0x563397da4c40 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1dada39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41aa540 con 0x563397da4250 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39634/0 (socket says 172.21.15.69:39634) 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1da91ea640 1 -- 172.21.15.69:0/3002708450 learned_addr learned my addr 172.21.15.69:0/3002708450 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:32.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1da89e9640 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397d9fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.377+0000 7f1d9bfff640 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4c40 0x563397db5980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 -- 172.21.15.69:0/3002708450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 msgr2=0x563397d9fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397d9fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 -- 172.21.15.69:0/3002708450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4c40 msgr2=0x563397db5980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4c40 0x563397db5980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 -- 172.21.15.69:0/3002708450 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563397da3e90 con 0x563397da9db0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1da91ea640 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e940 tx=0x7f1d9c007840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2a9431762426e07 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1d8a7fc640 1 -- 172.21.15.69:0/3002708450 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c0ca650 con 0x563397da9db0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1d8a7fc640 1 -- 172.21.15.69:0/3002708450 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d9c0a5870 con 0x563397da9db0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.378+0000 7f1d8a7fc640 1 -- 172.21.15.69:0/3002708450 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c004750 con 0x563397da9db0 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- 172.21.15.69:0/3002708450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 msgr2=0x563397dae050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e940 tx=0x7f1d9c007840 comp rx=0 tx=0).stop 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- 172.21.15.69:0/3002708450 shutdown_connections 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397dae050 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4c40 0x563397db5980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- 172.21.15.69:0/3002708450 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4250 0x563397d9fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- 172.21.15.69:0/3002708450 >> 172.21.15.69:0/3002708450 conn(0x563397d9f8c0 msgr2=0x563397da94c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- 172.21.15.69:0/3002708450 shutdown_connections 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- 172.21.15.69:0/3002708450 wait complete. 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- start start 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 Processor -- start 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4250 0x563397db24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397db2a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41aabb0 con 0x563397da9db0 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41ac020 con 0x563397da4250 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1da41ab280 con 0x563397da4c40 2024-05-09T09:28:32.384 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.379+0000 7f1dada39740 10 client.0 _ll_register_callbacks cb 0x7f1da0016070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d9bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397db2a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39636/0 (socket says 172.21.15.69:39636) 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 -- 172.21.15.69:0/2281432812 learned_addr learned my addr 172.21.15.69:0/2281432812 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4250 0x563397db24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 -- 172.21.15.69:0/2281432812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 msgr2=0x563397db2a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397db2a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 -- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4250 msgr2=0x563397db24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4250 0x563397db24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 -- 172.21.15.69:0/2281432812 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563397db5ae0 con 0x563397da9db0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1da91ea640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097d0 tx=0x7f1d9c09c520 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc80987e0b380117 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9c09c1a0 con 0x563397da9db0 2024-05-09T09:28:32.385 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1d9c0a4520 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1d9c0a5870 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f1d9c0a91d0 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1dada39740 10 client.16833 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563397e1f9c0 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d9c0a5870 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.380+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f1d9c09b190 con 0x563397da9db0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 10 client.16833 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 10 client.16833 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 20 client.16833 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397e20230 0x563397e22650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 4 client.16833 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563397d0fdc0 con 0x563397e20230 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397e20230 0x563397e22650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:32.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1d83fff640 10 client.16833 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:32.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.381+0000 7f1da89e9640 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397e20230 0x563397e22650 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c06849dafbf37e82 server_cookie=c21ab26345039135 in_seq=0 out_seq=0 2024-05-09T09:28:32.499 INFO:tasks.workunit.client.0.smithi069.stdout: 1 652914 8.08 MB/sec execute 390 sec latency 81.406 ms 2024-05-09T09:28:32.511 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.510+0000 7f1d83fff640 1 -- 172.21.15.69:0/2281432812 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563397d0fdc0 con 0x563397e20230 2024-05-09T09:28:32.511 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.510+0000 7f1d83fff640 10 client.16833 handle_command_reply: tid=0 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.510+0000 7f1dada39740 1 client.16833 shutdown 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 msgr2=0x563397e1e5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c0097d0 tx=0x7f1d9c09c520 comp rx=0 tx=0).stop 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 shutdown_connections 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563397da9db0 0x563397e1e5a0 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563397e20230 0x563397e22650 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563397da4c40 0x563397db2a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1d83fff640 0 client.16833 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 --2- 172.21.15.69:0/2281432812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563397da4250 0x563397db24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 >> 172.21.15.69:0/2281432812 conn(0x563397d9f8c0 msgr2=0x563397da9090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 shutdown_connections 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 1 -- 172.21.15.69:0/2281432812 wait complete. 2024-05-09T09:28:32.512 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1dada39740 20 client.16833 trim_cache size 0 max 16384 2024-05-09T09:28:32.513 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:28:32.513 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:28:32.513 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:28:32.513 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 msgr2=0x7f1d8c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 0x7f1d8c0a50f0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7f1da41017e0 tx=0x7f1da0009670 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 msgr2=0x7f1da41032a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.511+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f1d9c00e970 tx=0x7f1d9c00ed70 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 shutdown_connections 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2c30 0x7f1d8c0a50f0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da4102340 0x7f1da41032a0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da4101980 0x7f1da4102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 --2- 172.21.15.69:0/3102389795 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da4100790 0x7f1da4104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 >> 172.21.15.69:0/3102389795 conn(0x7f1da40fbd00 msgr2=0x7f1da40fd750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 shutdown_connections 2024-05-09T09:28:32.514 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:32.512+0000 7f1daac4d640 1 -- 172.21.15.69:0/3102389795 wait complete. 2024-05-09T09:28:33.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 654173 8.07 MB/sec execute 391 sec latency 39.429 ms 2024-05-09T09:28:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:33 smithi069 ceph-mon[30123]: pgmap v660: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:28:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:33 smithi163 ceph-mon[34354]: pgmap v660: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:28:33.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:33 smithi103 ceph-mon[34421]: pgmap v660: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 960 KiB/s rd, 3.2 MiB/s wr, 368 op/s 2024-05-09T09:28:34.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 656774 8.09 MB/sec execute 392 sec latency 30.051 ms 2024-05-09T09:28:35.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 657926 8.08 MB/sec execute 393 sec latency 21.950 ms 2024-05-09T09:28:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:35 smithi069 ceph-mon[30123]: pgmap v661: 129 pgs: 129 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.6 MiB/s wr, 301 op/s 2024-05-09T09:28:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:35 smithi163 ceph-mon[34354]: pgmap v661: 129 pgs: 129 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.6 MiB/s wr, 301 op/s 2024-05-09T09:28:35.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:35 smithi103 ceph-mon[34421]: pgmap v661: 129 pgs: 129 active+clean; 134 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 696 KiB/s rd, 2.6 MiB/s wr, 301 op/s 2024-05-09T09:28:36.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 660535 8.11 MB/sec execute 394 sec latency 141.788 ms 2024-05-09T09:28:37.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 661780 8.09 MB/sec execute 395 sec latency 13.610 ms 2024-05-09T09:28:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:37 smithi069 ceph-mon[30123]: pgmap v662: 129 pgs: 129 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.0 MiB/s wr, 343 op/s 2024-05-09T09:28:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:37 smithi163 ceph-mon[34354]: pgmap v662: 129 pgs: 129 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.0 MiB/s wr, 343 op/s 2024-05-09T09:28:37.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:37 smithi103 ceph-mon[34421]: pgmap v662: 129 pgs: 129 active+clean; 131 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 762 KiB/s rd, 3.0 MiB/s wr, 343 op/s 2024-05-09T09:28:38.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 664297 8.12 MB/sec execute 396 sec latency 52.399 ms 2024-05-09T09:28:39.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 665526 8.11 MB/sec execute 397 sec latency 21.985 ms 2024-05-09T09:28:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:39 smithi069 ceph-mon[30123]: pgmap v663: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:28:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:39 smithi163 ceph-mon[34354]: pgmap v663: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:28:39.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:39 smithi103 ceph-mon[34421]: pgmap v663: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:28:40.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 667981 8.12 MB/sec execute 398 sec latency 10.839 ms 2024-05-09T09:28:41.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 669513 8.12 MB/sec execute 399 sec latency 55.680 ms 2024-05-09T09:28:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:41 smithi069 ceph-mon[30123]: pgmap v664: 129 pgs: 129 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:28:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:41 smithi163 ceph-mon[34354]: pgmap v664: 129 pgs: 129 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:28:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:41 smithi103 ceph-mon[34421]: pgmap v664: 129 pgs: 129 active+clean; 132 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 758 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:28:42.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 671621 8.13 MB/sec execute 400 sec latency 10.206 ms 2024-05-09T09:28:43.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 673596 8.13 MB/sec execute 401 sec latency 67.151 ms 2024-05-09T09:28:43.668 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:43 smithi103 ceph-mon[34421]: pgmap v665: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:28:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:43 smithi069 ceph-mon[30123]: pgmap v665: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:28:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:43 smithi163 ceph-mon[34354]: pgmap v665: 129 pgs: 129 active+clean; 133 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 904 KiB/s rd, 2.9 MiB/s wr, 355 op/s 2024-05-09T09:28:44.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 675323 8.14 MB/sec execute 402 sec latency 22.171 ms 2024-05-09T09:28:45.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 677961 8.15 MB/sec execute 403 sec latency 19.782 ms 2024-05-09T09:28:45.616 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 -- 172.21.15.69:0/54557994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 msgr2=0x7fd32c109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 --2- 172.21.15.69:0/54557994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c109ce0 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7fd324009ca0 tx=0x7fd32402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 -- 172.21.15.69:0/54557994 shutdown_connections 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 --2- 172.21.15.69:0/54557994 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c109ce0 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 --2- 172.21.15.69:0/54557994 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c102a90 0x7fd32c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 --2- 172.21.15.69:0/54557994 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c1018a0 0x7fd32c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 -- 172.21.15.69:0/54557994 >> 172.21.15.69:0/54557994 conn(0x7fd32c0fce10 msgr2=0x7fd32c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 -- 172.21.15.69:0/54557994 shutdown_connections 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.614+0000 7fd330bf9640 1 -- 172.21.15.69:0/54557994 wait complete. 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 Processor -- start 2024-05-09T09:28:45.617 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 -- start start 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c1018a0 0x7fd32c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c102a90 0x7fd32c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10d520 con 0x7fd32c103450 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10d3a0 con 0x7fd32c102a90 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd330bf9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c10d6a0 con 0x7fd32c1018a0 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54878/0 (socket says 172.21.15.69:54878) 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd32ad9d640 1 -- 172.21.15.69:0/3938993526 learned_addr learned my addr 172.21.15.69:0/3938993526 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd329d9b640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c102a90 0x7fd32c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.615+0000 7fd32a59c640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c1018a0 0x7fd32c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 -- 172.21.15.69:0/3938993526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c1018a0 msgr2=0x7fd32c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c1018a0 0x7fd32c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 -- 172.21.15.69:0/3938993526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c102a90 msgr2=0x7fd32c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c102a90 0x7fd32c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 -- 172.21.15.69:0/3938993526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd32c19f6c0 con 0x7fd32c103450 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd324031ae0 tx=0x7fd324031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd31b7fe640 1 -- 172.21.15.69:0/3938993526 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd324003c30 con 0x7fd32c103450 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd31b7fe640 1 -- 172.21.15.69:0/3938993526 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd324003dd0 con 0x7fd32c103450 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd31b7fe640 1 -- 172.21.15.69:0/3938993526 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32402fcb0 con 0x7fd32c103450 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd32c19f950 con 0x7fd32c103450 2024-05-09T09:28:45.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.616+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd32c19fea0 con 0x7fd32c103450 2024-05-09T09:28:45.626 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.618+0000 7fd31b7fe640 1 -- 172.21.15.69:0/3938993526 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd324004050 con 0x7fd32c103450 2024-05-09T09:28:45.626 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 Processor -- start 2024-05-09T09:28:45.626 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 -- start start 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd31b7fe640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 0x7fd3140a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742abf4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55742ab50340 con 0x55742abecc40 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55742aaf8290 con 0x55742abec250 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.619+0000 7fd3339e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55742ab4c300 con 0x55742abf1db0 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd31b7fe640 1 -- 172.21.15.69:0/3938993526 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd3240f05e0 con 0x7fd32c103450 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd329d9b640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 0x7fd3140a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:50746/0 (socket says 172.21.15.69:50746) 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 -- 172.21.15.69:0/3297324408 learned_addr learned my addr 172.21.15.69:0/3297324408 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:45.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742abf4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd329d9b640 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 -- 172.21.15.69:0/3297324408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 msgr2=0x55742abf4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742abf4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 -- 172.21.15.69:0/3297324408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 msgr2=0x55742abf1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 -- 172.21.15.69:0/3297324408 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742ab1c8e0 con 0x55742abec250 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd329d9b640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 0x7fd3140a50f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd32000b620 tx=0x7fd32000bb50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd32a59c640 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009ca0 tx=0x7fd31c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e725b8491e385f56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd318ff9640 1 -- 172.21.15.69:0/3297324408 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c004220 con 0x55742abec250 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd318ff9640 1 -- 172.21.15.69:0/3297324408 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd31c0043c0 con 0x55742abec250 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.620+0000 7fd318ff9640 1 -- 172.21.15.69:0/3297324408 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c0198d0 con 0x55742abec250 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- 172.21.15.69:0/3297324408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 msgr2=0x55742abec670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009ca0 tx=0x7fd31c011810 comp rx=0 tx=0).stop 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- 172.21.15.69:0/3297324408 shutdown_connections 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742abf4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 --2- 172.21.15.69:0/3297324408 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abec250 0x55742abec670 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- 172.21.15.69:0/3297324408 >> 172.21.15.69:0/3297324408 conn(0x55742abe78c0 msgr2=0x55742abe7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- 172.21.15.69:0/3297324408 shutdown_connections 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- 172.21.15.69:0/3297324408 wait complete. 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 -- start start 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.621+0000 7fd3339e4740 1 Processor -- start 2024-05-09T09:28:45.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abf6460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abfd9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55742abf4d60 con 0x55742abecc40 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55742ab0b790 con 0x55742abf1db0 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55742abf4ee0 con 0x55742abec250 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 10 client.0 _ll_register_callbacks cb 0x55742ab46a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd32ad9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abfd9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54902/0 (socket says 172.21.15.69:54902) 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 -- 172.21.15.69:0/1382001550 learned_addr learned my addr 172.21.15.69:0/1382001550 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abf6460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 -- 172.21.15.69:0/1382001550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 msgr2=0x55742abf6460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abf6460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 -- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 msgr2=0x55742abfd9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abfd9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 -- 172.21.15.69:0/1382001550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742a864210 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abfd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd320012460 tx=0x7fd32000b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42fcc033b36b34b2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd311ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32001ec00 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd31c0043c0 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.622+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd32001eda0 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd320033050 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 10 client.16845 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55742a930620 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd31c0043c0 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fd32001f070 con 0x55742abecc40 2024-05-09T09:28:45.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 10 client.16845 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 10 client.16845 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 20 client.16845 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac01fa0 0x55742ac043d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 4 client.16845 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55742ab5da70 con 0x55742ac01fa0 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.623+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac01fa0 0x55742ac043d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd311ffb640 10 client.16845 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac01fa0 0x55742ac043d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f775b494c38ba54 server_cookie=1381a83e7c92f33a in_seq=0 out_seq=0 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd311ffb640 1 -- 172.21.15.69:0/1382001550 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55742ab5da70 con 0x55742ac01fa0 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd311ffb640 10 client.16845 handle_command_reply: tid=0 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 client.16845 shutdown 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 msgr2=0x55742abf69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7fd320012460 tx=0x7fd32000b390 comp rx=0 tx=0).stop 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 shutdown_connections 2024-05-09T09:28:45.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abfd9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abf69a0 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac01fa0 0x55742ac043d0 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1382001550 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abf6460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd311ffb640 0 client.16845 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 >> 172.21.15.69:0/1382001550 conn(0x55742abe78c0 msgr2=0x55742abe6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 shutdown_connections 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.624+0000 7fd3339e4740 1 -- 172.21.15.69:0/1382001550 wait complete. 2024-05-09T09:28:45.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.625+0000 7fd3339e4740 20 client.16845 trim_cache size 0 max 16384 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 Processor -- start 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 -- start start 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abede20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abea870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1acc50 con 0x55742abecc40 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1aa520 con 0x55742abf1db0 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd3339e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1a8370 con 0x55742abec250 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54906/0 (socket says 172.21.15.69:54906) 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 -- 172.21.15.69:0/1468360661 learned_addr learned my addr 172.21.15.69:0/1468360661 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abea870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abede20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 -- 172.21.15.69:0/1468360661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 msgr2=0x55742abede20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abede20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 -- 172.21.15.69:0/1468360661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 msgr2=0x55742abea870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abea870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 -- 172.21.15.69:0/1468360661 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55742abed020 con 0x55742abecc40 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd32000b5f0 tx=0x7fd320005c30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7b0f992c8de5d2d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.640+0000 7fd313fff640 1 -- 172.21.15.69:0/1468360661 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32003c070 con 0x55742abecc40 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd313fff640 1 -- 172.21.15.69:0/1468360661 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd320020400 con 0x55742abecc40 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd313fff640 1 -- 172.21.15.69:0/1468360661 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd32002b920 con 0x55742abecc40 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- 172.21.15.69:0/1468360661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 msgr2=0x55742abee360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7fd32000b5f0 tx=0x7fd320005c30 comp rx=0 tx=0).stop 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- 172.21.15.69:0/1468360661 shutdown_connections 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abf1db0 0x55742abea870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abecc40 0x55742abee360 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1468360661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abec250 0x55742abede20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- 172.21.15.69:0/1468360661 >> 172.21.15.69:0/1468360661 conn(0x55742abe78c0 msgr2=0x55742abf6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- 172.21.15.69:0/1468360661 shutdown_connections 2024-05-09T09:28:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- 172.21.15.69:0/1468360661 wait complete. 2024-05-09T09:28:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.641+0000 7fd3339e4740 1 -- start start 2024-05-09T09:28:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 Processor -- start 2024-05-09T09:28:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 0x55742ac63820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742ac6a860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1a89e0 con 0x55742abec250 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1a9e50 con 0x55742abecc40 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd32c1a90b0 con 0x55742abf1db0 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd3339e4740 10 client.0 _ll_register_callbacks cb 0x7fd32000bd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54920/0 (socket says 172.21.15.69:54920) 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 -- 172.21.15.69:0/1712325596 learned_addr learned my addr 172.21.15.69:0/1712325596 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:45.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 0x55742ac63820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32ad9d640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742ac6a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 -- 172.21.15.69:0/1712325596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 msgr2=0x55742ac6a860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742ac6a860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 -- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 msgr2=0x55742ac63820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 0x55742ac63820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 -- 172.21.15.69:0/1712325596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55742ac6ae50 con 0x55742abec250 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 0x55742ac63820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd32a59c640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009250 tx=0x7fd31c005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12dbcffa6ef7a2e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd311ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.642+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd31c032020 con 0x55742abec250 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd31c01f040 con 0x55742abec250 2024-05-09T09:28:45.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd320020400 con 0x55742abec250 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd31c03d050 con 0x55742abec250 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 10 client.16851 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55742ac6c160 con 0x55742abec250 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd320020400 con 0x55742abec250 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fd31c003680 con 0x55742abec250 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 10 client.16851 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:45.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 10 client.16851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 20 client.16851 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac6c9f0 0x55742ac6ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 4 client.16851 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55742ab57dc0 con 0x55742ac6c9f0 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.643+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac6c9f0 0x55742ac6ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.644+0000 7fd311ffb640 10 client.16851 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:45.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.644+0000 7fd329d9b640 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac6c9f0 0x55742ac6ee10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8a72520d050ab850 server_cookie=59b82d492e6b1660 in_seq=0 out_seq=0 2024-05-09T09:28:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:45 smithi069 ceph-mon[30123]: pgmap v666: 129 pgs: 129 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd311ffb640 1 -- 172.21.15.69:0/1712325596 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55742ab57dc0 con 0x55742ac6c9f0 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd311ffb640 10 client.16851 handle_command_reply: tid=0 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 client.16851 shutdown 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 msgr2=0x55742ac632e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7fd31c009250 tx=0x7fd31c005d30 comp rx=0 tx=0).stop 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 shutdown_connections 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55742ac6c9f0 0x55742ac6ee10 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55742abf1db0 0x55742ac6a860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55742abecc40 0x55742ac63820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 --2- 172.21.15.69:0/1712325596 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55742abec250 0x55742ac632e0 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 >> 172.21.15.69:0/1712325596 conn(0x55742abe78c0 msgr2=0x55742abf6460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd311ffb640 0 client.16851 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 shutdown_connections 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 1 -- 172.21.15.69:0/1712325596 wait complete. 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.764+0000 7fd3339e4740 20 client.16851 trim_cache size 0 max 16384 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:28:45.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:28:45.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 msgr2=0x7fd3140a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 0x7fd3140a50f0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fd32000b620 tx=0x7fd32000bb50 comp rx=0 tx=0).stop 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 msgr2=0x7fd32c19f050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7fd324031ae0 tx=0x7fd324031ee0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 shutdown_connections 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3140a2c30 0x7fd3140a50f0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd32c103450 0x7fd32c19f050 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd32c102a90 0x7fd32c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 --2- 172.21.15.69:0/3938993526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd32c1018a0 0x7fd32c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 >> 172.21.15.69:0/3938993526 conn(0x7fd32c0fce10 msgr2=0x7fd32c105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 shutdown_connections 2024-05-09T09:28:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:45.765+0000 7fd330bf9640 1 -- 172.21.15.69:0/3938993526 wait complete. 2024-05-09T09:28:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:45 smithi163 ceph-mon[34354]: pgmap v666: 129 pgs: 129 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:28:45.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:45 smithi103 ceph-mon[34421]: pgmap v666: 129 pgs: 129 active+clean; 135 MiB data, 2.3 GiB used, 1.0 TiB / 1.0 TiB avail; 723 KiB/s rd, 2.6 MiB/s wr, 309 op/s 2024-05-09T09:28:46.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 679066 8.15 MB/sec execute 404 sec latency 118.283 ms 2024-05-09T09:28:47.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 681936 8.17 MB/sec execute 405 sec latency 14.619 ms 2024-05-09T09:28:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:47 smithi069 ceph-mon[30123]: pgmap v667: 129 pgs: 129 active+clean; 134 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T09:28:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:47 smithi163 ceph-mon[34354]: pgmap v667: 129 pgs: 129 active+clean; 134 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T09:28:47.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:47 smithi103 ceph-mon[34421]: pgmap v667: 129 pgs: 129 active+clean; 134 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 3.1 MiB/s wr, 361 op/s 2024-05-09T09:28:48.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 683036 8.16 MB/sec execute 406 sec latency 60.412 ms 2024-05-09T09:28:49.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 685623 8.18 MB/sec execute 407 sec latency 38.076 ms 2024-05-09T09:28:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:49 smithi069 ceph-mon[30123]: pgmap v668: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:28:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:49 smithi163 ceph-mon[34354]: pgmap v668: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:28:49.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:49 smithi103 ceph-mon[34421]: pgmap v668: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 841 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:28:50.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 687059 8.17 MB/sec execute 408 sec latency 12.888 ms 2024-05-09T09:28:51.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 689367 8.19 MB/sec execute 409 sec latency 10.459 ms 2024-05-09T09:28:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:51 smithi069 ceph-mon[30123]: pgmap v669: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-05-09T09:28:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:51 smithi163 ceph-mon[34354]: pgmap v669: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-05-09T09:28:51.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:51 smithi103 ceph-mon[34421]: pgmap v669: 129 pgs: 129 active+clean; 133 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 535 KiB/s rd, 2.6 MiB/s wr, 282 op/s 2024-05-09T09:28:52.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 691404 8.18 MB/sec execute 410 sec latency 14.379 ms 2024-05-09T09:28:53.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 693077 8.19 MB/sec execute 411 sec latency 70.123 ms 2024-05-09T09:28:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:53 smithi069 ceph-mon[30123]: pgmap v670: 129 pgs: 129 active+clean; 136 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-05-09T09:28:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:53 smithi163 ceph-mon[34354]: pgmap v670: 129 pgs: 129 active+clean; 136 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-05-09T09:28:53.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:53 smithi103 ceph-mon[34421]: pgmap v670: 129 pgs: 129 active+clean; 136 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 887 KiB/s rd, 3.2 MiB/s wr, 370 op/s 2024-05-09T09:28:54.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 695385 8.20 MB/sec execute 412 sec latency 33.315 ms 2024-05-09T09:28:55.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 696778 8.20 MB/sec execute 413 sec latency 16.858 ms 2024-05-09T09:28:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:55 smithi069 ceph-mon[30123]: pgmap v671: 129 pgs: 129 active+clean; 135 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-05-09T09:28:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:55 smithi163 ceph-mon[34354]: pgmap v671: 129 pgs: 129 active+clean; 135 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-05-09T09:28:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:55 smithi103 ceph-mon[34421]: pgmap v671: 129 pgs: 129 active+clean; 135 MiB data, 2.4 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.8 MiB/s wr, 334 op/s 2024-05-09T09:28:56.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 699401 8.21 MB/sec execute 414 sec latency 38.673 ms 2024-05-09T09:28:57.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 700469 8.21 MB/sec execute 415 sec latency 30.330 ms 2024-05-09T09:28:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:57 smithi069 ceph-mon[30123]: pgmap v672: 129 pgs: 129 active+clean; 137 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 2.9 MiB/s wr, 354 op/s 2024-05-09T09:28:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:57 smithi163 ceph-mon[34354]: pgmap v672: 129 pgs: 129 active+clean; 137 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 2.9 MiB/s wr, 354 op/s 2024-05-09T09:28:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:57 smithi103 ceph-mon[34421]: pgmap v672: 129 pgs: 129 active+clean; 137 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 2.9 MiB/s wr, 354 op/s 2024-05-09T09:28:58.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 703110 8.23 MB/sec execute 416 sec latency 34.724 ms 2024-05-09T09:28:58.857 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.857+0000 7f3325561640 1 -- 172.21.15.69:0/1807101058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320102340 msgr2=0x7f3320108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.857 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.857+0000 7f3325561640 1 --2- 172.21.15.69:0/1807101058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320102340 0x7f3320108bd0 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f3318009ca0 tx=0x7f331802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- 172.21.15.69:0/1807101058 shutdown_connections 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- 172.21.15.69:0/1807101058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320102340 0x7f3320108bd0 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- 172.21.15.69:0/1807101058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320101980 0x7f3320101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- 172.21.15.69:0/1807101058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320100790 0x7f3320100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- 172.21.15.69:0/1807101058 >> 172.21.15.69:0/1807101058 conn(0x7f33200fbd00 msgr2=0x7f33200fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- 172.21.15.69:0/1807101058 shutdown_connections 2024-05-09T09:28:58.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- 172.21.15.69:0/1807101058 wait complete. 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 Processor -- start 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- start start 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 0x7f332019f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f332010d530 con 0x7f3320101980 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f332010d3b0 con 0x7f3320100790 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.858+0000 7f3325561640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f332010d6b0 con 0x7f3320102340 2024-05-09T09:28:58.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40194/0 (socket says 172.21.15.69:40194) 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:42688/0 (socket says 172.21.15.69:42688) 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 -- 172.21.15.69:0/1889438088 learned_addr learned my addr 172.21.15.69:0/1889438088 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 0x7f332019f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 -- 172.21.15.69:0/1889438088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 msgr2=0x7f332019f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 0x7f332019f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 -- 172.21.15.69:0/1889438088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 msgr2=0x7f3320197a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 -- 172.21.15.69:0/1889438088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f332019f6b0 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 0x7f332019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331effd640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f331e7fc640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f331400e9f0 tx=0x7f331400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f3307fff640 1 -- 172.21.15.69:0/1889438088 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331400cd80 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f332019f9a0 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f332019ff40 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f3307fff640 1 -- 172.21.15.69:0/1889438088 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3314010910 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.859+0000 7f3307fff640 1 -- 172.21.15.69:0/1889438088 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3314018980 con 0x7f3320101980 2024-05-09T09:28:58.865 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.861+0000 7f3307fff640 1 -- 172.21.15.69:0/1889438088 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3314010ac0 con 0x7f3320101980 2024-05-09T09:28:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 Processor -- start 2024-05-09T09:28:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 -- start start 2024-05-09T09:28:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35ada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b2b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f3307fff640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 0x7f33080a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e9f350d340 con 0x55e9f35b31c0 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e9f34b5290 con 0x55e9f35ad660 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f332826f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e9f3509300 con 0x55e9f35ae050 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40202/0 (socket says 172.21.15.69:40202) 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331f7fe640 1 -- 172.21.15.69:0/1570461701 learned_addr learned my addr 172.21.15.69:0/1570461701 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331effd640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b2b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331e7fc640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 0x7f33080a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.862+0000 7f3307fff640 1 -- 172.21.15.69:0/1889438088 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f33140cf840 con 0x7f3320101980 2024-05-09T09:28:58.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 -- 172.21.15.69:0/1570461701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 msgr2=0x55e9f35ada80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35ada80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 -- 172.21.15.69:0/1570461701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 msgr2=0x55e9f35b5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 -- 172.21.15.69:0/1570461701 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f34d98e0 con 0x55e9f35ae050 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331e7fc640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 0x7f33080a50f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f33140986f0 tx=0x7f3314013b60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f331effd640 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b2b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3310009fa0 tx=0x7f33100118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=99f634b57e26033 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f33057fa640 1 -- 172.21.15.69:0/1570461701 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310004030 con 0x55e9f35ae050 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f33057fa640 1 -- 172.21.15.69:0/1570461701 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33100041d0 con 0x55e9f35ae050 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.863+0000 7f33057fa640 1 -- 172.21.15.69:0/1570461701 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f331000f800 con 0x55e9f35ae050 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- 172.21.15.69:0/1570461701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 msgr2=0x55e9f35b2b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b2b50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f3310009fa0 tx=0x7f33100118d0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- 172.21.15.69:0/1570461701 shutdown_connections 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35b5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b2b50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- 172.21.15.69:0/1570461701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35ada80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- 172.21.15.69:0/1570461701 >> 172.21.15.69:0/1570461701 conn(0x55e9f35a48c0 msgr2=0x55e9f35a4cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- 172.21.15.69:0/1570461701 shutdown_connections 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- 172.21.15.69:0/1570461701 wait complete. 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- start start 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 Processor -- start 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35b7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b7db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35bedf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e9f35b6080 con 0x55e9f35b31c0 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e9f34c8790 con 0x55e9f35ad660 2024-05-09T09:28:58.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e9f35b6200 con 0x55e9f35ae050 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.864+0000 7f332826f740 10 client.0 _ll_register_callbacks cb 0x55e9f3503a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35b7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b7db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35b7870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:42704/0 (socket says 172.21.15.69:42704) 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331f7fe640 1 -- 172.21.15.69:0/97566632 learned_addr learned my addr 172.21.15.69:0/97566632 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331f7fe640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35bedf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 -- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 msgr2=0x55e9f35b7870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35b7870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 -- 172.21.15.69:0/97566632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 msgr2=0x55e9f35bedf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35bedf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 -- 172.21.15.69:0/97566632 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f3221210 con 0x55e9f35ae050 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331f7fe640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35bedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:28:58.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f331effd640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b7db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55e9f35a5830 tx=0x7f3310009ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d6f43f991f291a34 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f3302ff5640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310027070 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f3310005040 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f33100041d0 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f3310033050 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f332826f740 10 client.26092 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.865+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e9f32ed620 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f33100041d0 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f331000f5a0 con 0x55e9f35ae050 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 10 client.26092 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 10 client.26092 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 20 client.26092 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f35c3190 0x55e9f35c5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 4 client.26092 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:28:58.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e9f351aa70 con 0x55e9f35c3190 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f331f7fe640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f35c3190 0x55e9f35c5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f331f7fe640 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f35c3190 0x55e9f35c5580 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f66e3603a183366c server_cookie=40507f6ec87baa02 in_seq=0 out_seq=0 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.866+0000 7f3302ff5640 10 client.26092 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f3302ff5640 1 -- 172.21.15.69:0/97566632 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e9f351aa70 con 0x55e9f35c3190 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f3302ff5640 10 client.26092 handle_command_reply: tid=0 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 client.26092 shutdown 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 msgr2=0x55e9f35b7db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b7db0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x55e9f35a5830 tx=0x7f3310009ce0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 shutdown_connections 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35bedf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f35c3190 0x55e9f35c5580 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ae050 0x55e9f35b7db0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 --2- 172.21.15.69:0/97566632 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35b7870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 >> 172.21.15.69:0/97566632 conn(0x55e9f35a48c0 msgr2=0x55e9f35a3a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f3302ff5640 0 client.26092 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 shutdown_connections 2024-05-09T09:28:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 1 -- 172.21.15.69:0/97566632 wait complete. 2024-05-09T09:28:58.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.867+0000 7f332826f740 20 client.26092 trim_cache size 0 max 16384 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 Processor -- start 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 -- start start 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ad660 0x55e9f35a7950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35a7e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33201aaa20 con 0x55e9f35b31c0 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33201a82f0 con 0x55e9f35ae050 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f332826f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33201a6140 con 0x55e9f35ad660 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35a7e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:42712/0 (socket says 172.21.15.69:42712) 2024-05-09T09:28:58.885 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 -- 172.21.15.69:0/4072989636 learned_addr learned my addr 172.21.15.69:0/4072989636 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331f7fe640 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ad660 0x55e9f35a7950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 -- 172.21.15.69:0/4072989636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ad660 msgr2=0x55e9f35a7950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ad660 0x55e9f35a7950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 -- 172.21.15.69:0/4072989636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 msgr2=0x55e9f35a7e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35a7e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.883+0000 7f331effd640 1 -- 172.21.15.69:0/4072989636 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e9f35c58f0 con 0x55e9f35ae050 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f331effd640 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3310009380 tx=0x7f33100057d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61c33032a417729f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f3304ff9640 1 -- 172.21.15.69:0/4072989636 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310027040 con 0x55e9f35ae050 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f3304ff9640 1 -- 172.21.15.69:0/4072989636 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f331000fbe0 con 0x55e9f35ae050 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f3304ff9640 1 -- 172.21.15.69:0/4072989636 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f33100054f0 con 0x55e9f35ae050 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 -- 172.21.15.69:0/4072989636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 msgr2=0x55e9f35bedb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f3310009380 tx=0x7f33100057d0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 -- 172.21.15.69:0/4072989636 shutdown_connections 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35b31c0 0x55e9f35a7e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ae050 0x55e9f35bedb0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 --2- 172.21.15.69:0/4072989636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35ad660 0x55e9f35a7950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 -- 172.21.15.69:0/4072989636 >> 172.21.15.69:0/4072989636 conn(0x55e9f35a48c0 msgr2=0x55e9f35b66a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.884+0000 7f332826f740 1 -- 172.21.15.69:0/4072989636 shutdown_connections 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 -- 172.21.15.69:0/4072989636 wait complete. 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 -- start start 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 Processor -- start 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35bc420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35b31c0 0x55e9f35bd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f33201a67b0 con 0x55e9f35ae050 2024-05-09T09:28:58.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f33201a7c20 con 0x55e9f35ad660 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f33201a6e80 con 0x55e9f35b31c0 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f332826f740 10 client.0 _ll_register_callbacks cb 0x7f3314005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35bc420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35bc420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:42716/0 (socket says 172.21.15.69:42716) 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40230/0 (socket says 172.21.15.69:40230) 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331f7fe640 1 -- 172.21.15.69:0/1970474839 learned_addr learned my addr 172.21.15.69:0/1970474839 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35b31c0 0x55e9f35bd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 -- 172.21.15.69:0/1970474839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35b31c0 msgr2=0x55e9f35bd970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35b31c0 0x55e9f35bd970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 -- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 msgr2=0x55e9f35bc420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35bc420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.885+0000 7f331effd640 1 -- 172.21.15.69:0/1970474839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e9f35bdf60 con 0x55e9f35ae050 2024-05-09T09:28:58.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f331effd640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f331002e040 tx=0x7f33100092a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da30ba2c815e06a9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3310027040 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f3310005270 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f331000fbe0 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f33100054f0 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 10 client.16869 fetch_fsmap learned FSMap version 39 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x55e9f36296b0 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f331000fbe0 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f33100049e0 con 0x55e9f35ae050 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 10 client.16869 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 10 client.16869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:28:58.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 20 client.16869 populate_metadata read hostname 'smithi069' 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f3629ee0 0x55e9f362c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 4 client.16869 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e9f3514dc0 con 0x55e9f3629ee0 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.886+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f3629ee0 0x55e9f362c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.887+0000 7f3302ff5640 10 client.16869 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:58.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:58.887+0000 7f331f7fe640 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f3629ee0 0x55e9f362c300 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c87b2156f5aa8648 server_cookie=93dfc5dbc0f128d2 in_seq=0 out_seq=0 2024-05-09T09:28:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.015+0000 7f3302ff5640 1 -- 172.21.15.69:0/1970474839 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e9f3514dc0 con 0x55e9f3629ee0 2024-05-09T09:28:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.015+0000 7f3302ff5640 10 client.16869 handle_command_reply: tid=0 2024-05-09T09:28:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 client.16869 shutdown 2024-05-09T09:28:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 msgr2=0x55e9f35bc960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:59.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f331002e040 tx=0x7f33100092a0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 shutdown_connections 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e9f3629ee0 0x55e9f362c300 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f3302ff5640 0 client.16869 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e9f35b31c0 0x55e9f35bd970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e9f35ae050 0x55e9f35bc960 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 --2- 172.21.15.69:0/1970474839 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e9f35ad660 0x55e9f35bc420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 >> 172.21.15.69:0/1970474839 conn(0x55e9f35a48c0 msgr2=0x55e9f35b2480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:59.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 shutdown_connections 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 1 -- 172.21.15.69:0/1970474839 wait complete. 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.016+0000 7f332826f740 20 client.16869 trim_cache size 0 max 16384 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:28:59.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 msgr2=0x7f33080a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 0x7f33080a50f0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f33140986f0 tx=0x7f3314013b60 comp rx=0 tx=0).stop 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 msgr2=0x7f3320197fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f331400e9f0 tx=0x7f331400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 shutdown_connections 2024-05-09T09:28:59.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f33080a2c30 0x7f33080a50f0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3320102340 0x7f332019f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3320101980 0x7f3320197fd0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 --2- 172.21.15.69:0/1889438088 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3320100790 0x7f3320197a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.017+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 >> 172.21.15.69:0/1889438088 conn(0x7f33200fbd00 msgr2=0x7f3320104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.018+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 shutdown_connections 2024-05-09T09:28:59.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:28:59.018+0000 7f3325561640 1 -- 172.21.15.69:0/1889438088 wait complete. 2024-05-09T09:28:59.500 INFO:tasks.workunit.client.0.smithi069.stdout: 1 704044 8.21 MB/sec execute 417 sec latency 100.819 ms 2024-05-09T09:28:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:28:59 smithi069 ceph-mon[30123]: pgmap v673: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-09T09:28:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:28:59 smithi163 ceph-mon[34354]: pgmap v673: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-09T09:28:59.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:28:59 smithi103 ceph-mon[34421]: pgmap v673: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 855 KiB/s rd, 2.6 MiB/s wr, 311 op/s 2024-05-09T09:29:00.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 706782 8.24 MB/sec execute 418 sec latency 21.060 ms 2024-05-09T09:29:01.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 708098 8.23 MB/sec execute 419 sec latency 19.433 ms 2024-05-09T09:29:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:01 smithi069 ceph-mon[30123]: pgmap v674: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-05-09T09:29:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:01 smithi163 ceph-mon[34354]: pgmap v674: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-05-09T09:29:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:01 smithi103 ceph-mon[34421]: pgmap v674: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 824 KiB/s rd, 2.4 MiB/s wr, 288 op/s 2024-05-09T09:29:02.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 710498 8.24 MB/sec execute 420 sec latency 14.623 ms 2024-05-09T09:29:03.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 711818 8.24 MB/sec execute 421 sec latency 63.484 ms 2024-05-09T09:29:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:03 smithi069 ceph-mon[30123]: pgmap v675: 129 pgs: 129 active+clean; 135 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 341 op/s 2024-05-09T09:29:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:03 smithi163 ceph-mon[34354]: pgmap v675: 129 pgs: 129 active+clean; 135 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 341 op/s 2024-05-09T09:29:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:03 smithi103 ceph-mon[34421]: pgmap v675: 129 pgs: 129 active+clean; 135 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 947 KiB/s rd, 2.7 MiB/s wr, 341 op/s 2024-05-09T09:29:04.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 714076 8.25 MB/sec execute 422 sec latency 26.770 ms 2024-05-09T09:29:05.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 715965 8.25 MB/sec execute 423 sec latency 13.999 ms 2024-05-09T09:29:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:05 smithi069 ceph-mon[30123]: pgmap v676: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-05-09T09:29:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:05 smithi163 ceph-mon[34354]: pgmap v676: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-05-09T09:29:05.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:05 smithi103 ceph-mon[34421]: pgmap v676: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.6 MiB/s wr, 335 op/s 2024-05-09T09:29:06.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 717788 8.26 MB/sec execute 424 sec latency 30.307 ms 2024-05-09T09:29:07.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 720311 8.27 MB/sec execute 425 sec latency 14.743 ms 2024-05-09T09:29:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:07 smithi069 ceph-mon[30123]: pgmap v677: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-05-09T09:29:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:07 smithi163 ceph-mon[34354]: pgmap v677: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-05-09T09:29:07.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:07 smithi103 ceph-mon[34421]: pgmap v677: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 858 KiB/s rd, 2.8 MiB/s wr, 346 op/s 2024-05-09T09:29:08.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 721637 8.26 MB/sec execute 426 sec latency 56.977 ms 2024-05-09T09:29:08.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:08 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:29:08.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:08 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:29:08.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:08 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:29:09.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 723877 8.27 MB/sec execute 427 sec latency 67.129 ms 2024-05-09T09:29:10.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 725378 8.27 MB/sec execute 428 sec latency 14.344 ms 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: pgmap v678: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: pgmap v678: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: pgmap v678: 129 pgs: 129 active+clean; 139 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:11.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 728158 8.29 MB/sec execute 429 sec latency 83.851 ms 2024-05-09T09:29:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:11 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:29:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:11 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:29:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:11 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:11 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:29:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:11 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:29:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:11 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:11 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:29:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:11 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:29:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:11 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:29:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 -- 172.21.15.69:0/2799921626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920101460 msgr2=0x7f89201018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 --2- 172.21.15.69:0/2799921626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920101460 0x7f89201018e0 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f8918009ca0 tx=0x7f891802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 -- 172.21.15.69:0/2799921626 shutdown_connections 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 --2- 172.21.15.69:0/2799921626 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920101460 0x7f89201018e0 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 --2- 172.21.15.69:0/2799921626 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920100ac0 0x7f8920100f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 --2- 172.21.15.69:0/2799921626 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920100580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 -- 172.21.15.69:0/2799921626 >> 172.21.15.69:0/2799921626 conn(0x7f89200fbd00 msgr2=0x7f89200fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 -- 172.21.15.69:0/2799921626 shutdown_connections 2024-05-09T09:29:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.126+0000 7f89253d5640 1 -- 172.21.15.69:0/2799921626 wait complete. 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 Processor -- start 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 -- start start 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 0x7f892019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920101460 0x7f89201a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f892010f6e0 con 0x7f8920100ac0 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f892010f560 con 0x7f8920101460 2024-05-09T09:29:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f89253d5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f892010f860 con 0x7f8920100160 2024-05-09T09:29:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f891e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 0x7f892019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f891effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f891e7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 0x7f892019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56402/0 (socket says 172.21.15.69:56402) 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f891e7fc640 1 -- 172.21.15.69:0/1671854892 learned_addr learned my addr 172.21.15.69:0/1671854892 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.127+0000 7f891f7fe640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920101460 0x7f89201a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 -- 172.21.15.69:0/1671854892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920101460 msgr2=0x7f89201a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920101460 0x7f89201a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 -- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 msgr2=0x7f892019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 0x7f892019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 -- 172.21.15.69:0/1671854892 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f89201a18e0 con 0x7f8920100160 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f891effd640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920199ca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f891000e9f0 tx=0x7f891000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f8907fff640 1 -- 172.21.15.69:0/1671854892 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891000ccc0 con 0x7f8920100160 2024-05-09T09:29:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f8907fff640 1 -- 172.21.15.69:0/1671854892 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8910010910 con 0x7f8920100160 2024-05-09T09:29:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f8907fff640 1 -- 172.21.15.69:0/1671854892 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8910018960 con 0x7f8920100160 2024-05-09T09:29:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f89201a1bd0 con 0x7f8920100160 2024-05-09T09:29:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.128+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f89201a2120 con 0x7f8920100160 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.131+0000 7f8907fff640 1 -- 172.21.15.69:0/1671854892 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8910010ac0 con 0x7f8920100160 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 Processor -- start 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 -- start start 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a91670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 0x563537a96740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a96db0 0x563537a99240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5635379f5340 con 0x563537a91250 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56353799d290 con 0x563537a96db0 2024-05-09T09:29:12.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f89281c3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5635379f1300 con 0x563537a91c40 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a91670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a91670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56404/0 (socket says 172.21.15.69:56404) 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 0x563537a96740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891f7fe640 1 -- 172.21.15.69:0/6388143 learned_addr learned my addr 172.21.15.69:0/6388143 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891e7fc640 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a96db0 0x563537a99240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f8907fff640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 0x7f89080a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 -- 172.21.15.69:0/6388143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a96db0 msgr2=0x563537a99240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a96db0 0x563537a99240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 -- 172.21.15.69:0/6388143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 msgr2=0x563537a91670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a91670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891effd640 1 -- 172.21.15.69:0/6388143 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5635379c18e0 con 0x563537a91c40 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.132+0000 7f891f7fe640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 0x7f89080a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.133+0000 7f8907fff640 1 -- 172.21.15.69:0/1671854892 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f89100cf770 con 0x7f8920100160 2024-05-09T09:29:12.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.133+0000 7f891effd640 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 0x563537a96740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89100cce70 tx=0x7f8910066960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dc48514cae8559b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.133+0000 7f89057fa640 1 -- 172.21.15.69:0/6388143 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891009b820 con 0x563537a91c40 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.133+0000 7f89057fa640 1 -- 172.21.15.69:0/6388143 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8910026050 con 0x563537a91c40 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.133+0000 7f89057fa640 1 -- 172.21.15.69:0/6388143 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891009b640 con 0x563537a91c40 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- 172.21.15.69:0/6388143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 msgr2=0x563537a96740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891f7fe640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 0x7f89080a50f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x563537a924a0 tx=0x7f89180095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 0x563537a96740 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f89100cce70 tx=0x7f8910066960 comp rx=0 tx=0).stop 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- 172.21.15.69:0/6388143 shutdown_connections 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a96db0 0x563537a99240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91c40 0x563537a96740 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.153 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- 172.21.15.69:0/6388143 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a91670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- 172.21.15.69:0/6388143 >> 172.21.15.69:0/6388143 conn(0x563537a8c8c0 msgr2=0x563537a8ccc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- 172.21.15.69:0/6388143 shutdown_connections 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- 172.21.15.69:0/6388143 wait complete. 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- start start 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 Processor -- start 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537a9b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563537a99d00 con 0x563537a96db0 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5635379b0790 con 0x563537a91c40 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563537a99e80 con 0x563537a91250 2024-05-09T09:29:12.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f89281c3740 10 client.0 _ll_register_callbacks cb 0x5635379eba20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:46394/0 (socket says 172.21.15.69:46394) 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56416/0 (socket says 172.21.15.69:56416) 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891effd640 1 -- 172.21.15.69:0/105860320 learned_addr learned my addr 172.21.15.69:0/105860320 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.134+0000 7f891e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537a9b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 -- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 msgr2=0x563537a9b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537a9b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 -- 172.21.15.69:0/105860320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 msgr2=0x563537aa2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 -- 172.21.15.69:0/105860320 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563537709210 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891f7fe640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f891effd640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f891009bc70 tx=0x7f891009bce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=339e8a5a8ca105e8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f8902ff5640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891009b640 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8910097b70 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8910026050 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f891009b820 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f89281c3740 10 client.26110 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5635377d5620 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.135+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8910026050 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f89100ccf50 con 0x563537a91250 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 10 client.26110 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 10 client.26110 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 20 client.26110 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:12.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537aa6b00 0x563537aa8ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 4 client.26110 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563537a02a70 con 0x563537aa6b00 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f891e7fc640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537aa6b00 0x563537aa8ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f891e7fc640 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537aa6b00 0x563537aa8ef0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f2d252d35131f6d9 server_cookie=a4216a730b2453ca in_seq=0 out_seq=0 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.136+0000 7f8902ff5640 10 client.26110 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f8902ff5640 1 -- 172.21.15.69:0/105860320 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563537a02a70 con 0x563537aa6b00 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f8902ff5640 10 client.26110 handle_command_reply: tid=0 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 client.26110 shutdown 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 msgr2=0x563537a9b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f891009bc70 tx=0x7f891009bce0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 shutdown_connections 2024-05-09T09:29:12.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a96db0 0x563537aa2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537a9b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537aa6b00 0x563537aa8ef0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 --2- 172.21.15.69:0/105860320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a91250 0x563537a9b3b0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f8902ff5640 0 client.26110 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 >> 172.21.15.69:0/105860320 conn(0x563537a8c8c0 msgr2=0x563537a8ba00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 shutdown_connections 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 1 -- 172.21.15.69:0/105860320 wait complete. 2024-05-09T09:29:12.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.137+0000 7f89281c3740 20 client.26110 trim_cache size 0 max 16384 2024-05-09T09:29:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 Processor -- start 2024-05-09T09:29:12.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 -- start start 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa28c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89201aee80 con 0x563537a91250 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89201ac750 con 0x563537a91c40 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.153+0000 7f89281c3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f89201aa580 con 0x563537a96db0 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56422/0 (socket says 172.21.15.69:56422) 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aae720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:46396/0 (socket says 172.21.15.69:46396) 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891e7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 -- 172.21.15.69:0/3820860636 learned_addr learned my addr 172.21.15.69:0/3820860636 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 -- 172.21.15.69:0/3820860636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 msgr2=0x563537aae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 -- 172.21.15.69:0/3820860636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 msgr2=0x563537aa28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 -- 172.21.15.69:0/3820860636 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563537a8af00 con 0x563537a91250 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891e7fc640 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f891effd640 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f891009b9e0 tx=0x7f89100a47b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e12074622ef38363 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.165 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f8904ff9640 1 -- 172.21.15.69:0/3820860636 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891009b640 con 0x563537a91250 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f8904ff9640 1 -- 172.21.15.69:0/3820860636 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89100a48a0 con 0x563537a91250 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.154+0000 7f8904ff9640 1 -- 172.21.15.69:0/3820860636 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f89100d5120 con 0x563537a91250 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- 172.21.15.69:0/3820860636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 msgr2=0x563537a9b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f891009b9e0 tx=0x7f89100a47b0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- 172.21.15.69:0/3820860636 shutdown_connections 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 --2- 172.21.15.69:0/3820860636 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537a9b6d0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- 172.21.15.69:0/3820860636 >> 172.21.15.69:0/3820860636 conn(0x563537a8c8c0 msgr2=0x563537a9afc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- 172.21.15.69:0/3820860636 shutdown_connections 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- 172.21.15.69:0/3820860636 wait complete. 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.155+0000 7f89281c3740 1 -- start start 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 Processor -- start 2024-05-09T09:29:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa06c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aa16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f89201aac10 con 0x563537a91250 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f89201ac080 con 0x563537a91c40 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f89201ab2e0 con 0x563537a96db0 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f89281c3740 10 client.0 _ll_register_callbacks cb 0x7f8910005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f891effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f891effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56436/0 (socket says 172.21.15.69:56436) 2024-05-09T09:29:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f891effd640 1 -- 172.21.15.69:0/1417948252 learned_addr learned my addr 172.21.15.69:0/1417948252 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f891e7fc640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa06c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.156+0000 7f891f7fe640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aa16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 -- 172.21.15.69:0/1417948252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 msgr2=0x563537aa16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aa16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 -- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 msgr2=0x563537aa06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa06c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 -- 172.21.15.69:0/1417948252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563537aa1cc0 con 0x563537a91250 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f891effd640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8910098140 tx=0x7f891000cf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2333ba35a53ea06 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f8902ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f891009b640 con 0x563537a91250 2024-05-09T09:29:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f891009b330 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f89100a48a0 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f89100a4600 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f89281c3740 10 client.16887 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x563537aa1cc0 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.157+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f89100a48a0 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f8910018b90 con 0x563537a91250 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 10 client.16887 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 10 client.16887 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 20 client.16887 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537b0d6a0 0x563537b0fac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 4 client.16887 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:29:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5635379fcdc0 con 0x563537b0d6a0 2024-05-09T09:29:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f891e7fc640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537b0d6a0 0x563537b0fac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f8902ff5640 10 client.16887 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.158+0000 7f891e7fc640 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537b0d6a0 0x563537b0fac0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2d76a29ad9b5541 server_cookie=8af0c81de0038cbc in_seq=0 out_seq=0 2024-05-09T09:29:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f8902ff5640 1 -- 172.21.15.69:0/1417948252 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5635379fcdc0 con 0x563537b0d6a0 2024-05-09T09:29:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f8902ff5640 10 client.16887 handle_command_reply: tid=0 2024-05-09T09:29:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 client.16887 shutdown 2024-05-09T09:29:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 msgr2=0x563537aa0180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.291 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7f8910098140 tx=0x7f891000cf60 comp rx=0 tx=0).stop 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 shutdown_connections 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563537b0d6a0 0x563537b0fac0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563537a96db0 0x563537aa16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563537a91c40 0x563537aa06c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 --2- 172.21.15.69:0/1417948252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563537a91250 0x563537aa0180 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f8902ff5640 0 client.16887 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 >> 172.21.15.69:0/1417948252 conn(0x563537a8c8c0 msgr2=0x563537a8b170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 shutdown_connections 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 1 -- 172.21.15.69:0/1417948252 wait complete. 2024-05-09T09:29:12.292 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.290+0000 7f89281c3740 20 client.16887 trim_cache size 0 max 16384 2024-05-09T09:29:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:29:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:29:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:29:12.293 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:29:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 msgr2=0x7f89080a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 0x7f89080a50f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x563537a924a0 tx=0x7f89180095c0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 msgr2=0x7f8920199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:12.294 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920199ca0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f891000e9f0 tx=0x7f891000eeb0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 shutdown_connections 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f89080a2c30 0x7f89080a50f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8920101460 0x7f89201a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8920100ac0 0x7f892019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 --2- 172.21.15.69:0/1671854892 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8920100160 0x7f8920199ca0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 >> 172.21.15.69:0/1671854892 conn(0x7f89200fbd00 msgr2=0x7f89200fd800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:12.295 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 shutdown_connections 2024-05-09T09:29:12.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:12.291+0000 7f89253d5640 1 -- 172.21.15.69:0/1671854892 wait complete. 2024-05-09T09:29:12.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 729452 8.29 MB/sec execute 430 sec latency 98.452 ms 2024-05-09T09:29:12.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:12 smithi069 ceph-mon[30123]: pgmap v679: 129 pgs: 129 active+clean; 141 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:29:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:12 smithi163 ceph-mon[34354]: pgmap v679: 129 pgs: 129 active+clean; 141 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:29:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:12 smithi103 ceph-mon[34421]: pgmap v679: 129 pgs: 129 active+clean; 141 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 797 KiB/s rd, 2.7 MiB/s wr, 312 op/s 2024-05-09T09:29:13.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 731894 8.30 MB/sec execute 431 sec latency 9.834 ms 2024-05-09T09:29:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:13 smithi069 ceph-mon[30123]: pgmap v680: 129 pgs: 129 active+clean; 142 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:13 smithi163 ceph-mon[34354]: pgmap v680: 129 pgs: 129 active+clean; 142 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:13 smithi103 ceph-mon[34421]: pgmap v680: 129 pgs: 129 active+clean; 142 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:14.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 733322 8.29 MB/sec execute 432 sec latency 39.029 ms 2024-05-09T09:29:15.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 735564 8.31 MB/sec execute 433 sec latency 18.091 ms 2024-05-09T09:29:15.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:15 smithi069 ceph-mon[30123]: pgmap v681: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.8 MiB/s wr, 328 op/s 2024-05-09T09:29:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:15 smithi163 ceph-mon[34354]: pgmap v681: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.8 MiB/s wr, 328 op/s 2024-05-09T09:29:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:15 smithi103 ceph-mon[34421]: pgmap v681: 129 pgs: 129 active+clean; 138 MiB data, 2.5 GiB used, 1.0 TiB / 1.0 TiB avail; 819 KiB/s rd, 2.8 MiB/s wr, 328 op/s 2024-05-09T09:29:16.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 738047 8.32 MB/sec execute 434 sec latency 13.816 ms 2024-05-09T09:29:17.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 739756 8.32 MB/sec execute 435 sec latency 12.862 ms 2024-05-09T09:29:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:17 smithi069 ceph-mon[30123]: pgmap v682: 129 pgs: 129 active+clean; 140 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:29:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:17 smithi163 ceph-mon[34354]: pgmap v682: 129 pgs: 129 active+clean; 140 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:29:17.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:17 smithi103 ceph-mon[34421]: pgmap v682: 129 pgs: 129 active+clean; 140 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 799 KiB/s rd, 2.7 MiB/s wr, 342 op/s 2024-05-09T09:29:18.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 742451 8.34 MB/sec execute 436 sec latency 14.887 ms 2024-05-09T09:29:19.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 743787 8.33 MB/sec execute 437 sec latency 57.426 ms 2024-05-09T09:29:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:19 smithi069 ceph-mon[30123]: pgmap v683: 129 pgs: 129 active+clean; 141 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-05-09T09:29:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:19 smithi163 ceph-mon[34354]: pgmap v683: 129 pgs: 129 active+clean; 141 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-05-09T09:29:19.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:19 smithi103 ceph-mon[34421]: pgmap v683: 129 pgs: 129 active+clean; 141 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 825 KiB/s rd, 2.5 MiB/s wr, 330 op/s 2024-05-09T09:29:20.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 746127 8.34 MB/sec execute 438 sec latency 24.674 ms 2024-05-09T09:29:21.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 748591 8.35 MB/sec execute 439 sec latency 13.878 ms 2024-05-09T09:29:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:21 smithi069 ceph-mon[30123]: pgmap v684: 129 pgs: 129 active+clean; 142 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.3 MiB/s wr, 331 op/s 2024-05-09T09:29:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:21 smithi163 ceph-mon[34354]: pgmap v684: 129 pgs: 129 active+clean; 142 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.3 MiB/s wr, 331 op/s 2024-05-09T09:29:21.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:21 smithi103 ceph-mon[34421]: pgmap v684: 129 pgs: 129 active+clean; 142 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.3 MiB/s wr, 331 op/s 2024-05-09T09:29:22.501 INFO:tasks.workunit.client.0.smithi069.stdout: 1 750199 8.35 MB/sec execute 440 sec latency 13.097 ms 2024-05-09T09:29:23.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 753043 8.37 MB/sec execute 441 sec latency 13.315 ms 2024-05-09T09:29:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:23 smithi069 ceph-mon[30123]: pgmap v685: 129 pgs: 129 active+clean; 144 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:29:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:23 smithi163 ceph-mon[34354]: pgmap v685: 129 pgs: 129 active+clean; 144 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:29:23.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:23 smithi103 ceph-mon[34421]: pgmap v685: 129 pgs: 129 active+clean; 144 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 814 KiB/s rd, 2.6 MiB/s wr, 343 op/s 2024-05-09T09:29:24.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 754291 8.37 MB/sec execute 442 sec latency 74.070 ms 2024-05-09T09:29:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 -- 172.21.15.69:0/905968424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 msgr2=0x7fd13c109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 --2- 172.21.15.69:0/905968424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c109ce0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fd134009ca0 tx=0x7fd13402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 -- 172.21.15.69:0/905968424 shutdown_connections 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 --2- 172.21.15.69:0/905968424 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c109ce0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 --2- 172.21.15.69:0/905968424 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 --2- 172.21.15.69:0/905968424 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 0x7fd13c101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 -- 172.21.15.69:0/905968424 >> 172.21.15.69:0/905968424 conn(0x7fd13c0fcdf0 msgr2=0x7fd13c0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 -- 172.21.15.69:0/905968424 shutdown_connections 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 -- 172.21.15.69:0/905968424 wait complete. 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.385+0000 7fd142a12640 1 Processor -- start 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 -- start start 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 0x7fd13c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c10d570 con 0x7fd13c103450 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c10d3f0 con 0x7fd13c1018a0 2024-05-09T09:29:25.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd142a12640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c10d6f0 con 0x7fd13c102a90 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 0x7fd13c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:49202/0 (socket says 172.21.15.69:49202) 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2966565147 learned_addr learned my addr 172.21.15.69:0/2966565147 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd140faf640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2966565147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 msgr2=0x7fd13c197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 0x7fd13c197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 msgr2=0x7fd13c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.386+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2966565147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd13c19f6c0 con 0x7fd13c102a90 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9f0 tx=0x7fd13000edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd1397fa640 1 -- 172.21.15.69:0/2966565147 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13000cd80 con 0x7fd13c102a90 2024-05-09T09:29:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd1397fa640 1 -- 172.21.15.69:0/2966565147 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd130005500 con 0x7fd13c102a90 2024-05-09T09:29:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd1397fa640 1 -- 172.21.15.69:0/2966565147 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd130010690 con 0x7fd13c102a90 2024-05-09T09:29:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd13c073ac0 con 0x7fd13c102a90 2024-05-09T09:29:25.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.387+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd13c073fc0 con 0x7fd13c102a90 2024-05-09T09:29:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.389+0000 7fd1397fa640 1 -- 172.21.15.69:0/2966565147 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd1300046c0 con 0x7fd13c102a90 2024-05-09T09:29:25.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 Processor -- start 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 -- start start 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 0x56220e6e5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1397fa640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 0x7fd1240a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6eadb0 0x56220e6ed240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56220e649340 con 0x56220e6e5c40 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56220e5f1290 con 0x56220e6eadb0 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1457f9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56220e645300 con 0x56220e6e5250 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13bfff640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 0x7fd1240a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52710/0 (socket says 172.21.15.69:52710) 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2909854106 learned_addr learned my addr 172.21.15.69:0/2909854106 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd140faf640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 0x56220e6e5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd140faf640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6eadb0 0x56220e6ed240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd1397fa640 1 -- 172.21.15.69:0/2966565147 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd1300cf7a0 con 0x7fd13c102a90 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2909854106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 msgr2=0x56220e6e5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 0x56220e6e5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2909854106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6eadb0 msgr2=0x56220e6ed240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6eadb0 0x56220e6ed240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd13b7fe640 1 -- 172.21.15.69:0/2909854106 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220e6158e0 con 0x56220e6e5c40 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.390+0000 7fd140faf640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 0x56220e6e5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9c0 tx=0x7fd130004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6819356afdd369c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd13bfff640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 0x7fd1240a50f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd13c1028f0 tx=0x7fd12c012040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd1227fc640 1 -- 172.21.15.69:0/2909854106 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13009b7c0 con 0x56220e6e5c40 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd1227fc640 1 -- 172.21.15.69:0/2909854106 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd13009c380 con 0x56220e6e5c40 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd1227fc640 1 -- 172.21.15.69:0/2909854106 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300992d0 con 0x56220e6e5c40 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd1457f9740 1 -- 172.21.15.69:0/2909854106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 msgr2=0x56220e6ea740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.391+0000 7fd1457f9740 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9c0 tx=0x7fd130004a40 comp rx=0 tx=0).stop 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- 172.21.15.69:0/2909854106 shutdown_connections 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6eadb0 0x56220e6ed240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ea740 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- 172.21.15.69:0/2909854106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5250 0x56220e6e5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- 172.21.15.69:0/2909854106 >> 172.21.15.69:0/2909854106 conn(0x56220e6e08c0 msgr2=0x56220e6e0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- 172.21.15.69:0/2909854106 shutdown_connections 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- 172.21.15.69:0/2909854106 wait complete. 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- start start 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 Processor -- start 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6ef460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6eadb0 0x56220e6f69e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56220e6edd60 con 0x56220e6e5c40 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56220e604790 con 0x56220e6e5250 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56220e6edee0 con 0x56220e6eadb0 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd1457f9740 10 client.0 _ll_register_callbacks cb 0x56220e63fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52724/0 (socket says 172.21.15.69:52724) 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd13b7fe640 1 -- 172.21.15.69:0/81291720 learned_addr learned my addr 172.21.15.69:0/81291720 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.392+0000 7fd13bfff640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6ef460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd140faf640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6eadb0 0x56220e6f69e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 -- 172.21.15.69:0/81291720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6eadb0 msgr2=0x56220e6f69e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6eadb0 0x56220e6f69e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 -- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 msgr2=0x56220e6ef460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6ef460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 -- 172.21.15.69:0/81291720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56220e35d210 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd1300980e0 tx=0x7fd130009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d75a2e9565eb651a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd11b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd1300994b0 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd130098360 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd13009c380 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd130099b60 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd1457f9740 10 client.16899 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56220e429620 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.393+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd13009c380 con 0x56220e6e5c40 2024-05-09T09:29:25.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fd13009b0d0 con 0x56220e6e5c40 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 10 client.16899 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 10 client.16899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 20 client.16899 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e6fab40 0x56220e6fcf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 4 client.16899 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56220e656a70 con 0x56220e6fab40 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd13bfff640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e6fab40 0x56220e6fcf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd11b7fe640 10 client.16899 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.394+0000 7fd13bfff640 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e6fab40 0x56220e6fcf30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f24f138881de49a4 server_cookie=557e9c5a009c9e4f in_seq=0 out_seq=0 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd11b7fe640 1 -- 172.21.15.69:0/81291720 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56220e656a70 con 0x56220e6fab40 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd11b7fe640 10 client.16899 handle_command_reply: tid=0 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 client.16899 shutdown 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 msgr2=0x56220e6ef9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7fd1300980e0 tx=0x7fd130009310 comp rx=0 tx=0).stop 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 shutdown_connections 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e6fab40 0x56220e6fcf30 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6eadb0 0x56220e6f69e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6e5c40 0x56220e6ef9a0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd11b7fe640 0 client.16899 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 --2- 172.21.15.69:0/81291720 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6ef460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 >> 172.21.15.69:0/81291720 conn(0x56220e6e08c0 msgr2=0x56220e6dfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 shutdown_connections 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 1 -- 172.21.15.69:0/81291720 wait complete. 2024-05-09T09:29:25.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.395+0000 7fd1457f9740 20 client.16899 trim_cache size 0 max 16384 2024-05-09T09:29:25.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.410+0000 7fd1457f9740 1 Processor -- start 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.410+0000 7fd1457f9740 1 -- start start 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.410+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6e3f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd1457f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1aa930 con 0x56220e6eadb0 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd1457f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1a8200 con 0x56220e6e5250 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd1457f9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1a6050 con 0x56220e6e5c40 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52736/0 (socket says 172.21.15.69:52736) 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 -- 172.21.15.69:0/3707276623 learned_addr learned my addr 172.21.15.69:0/3707276623 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd13b7fe640 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd13bfff640 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6e3f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 -- 172.21.15.69:0/3707276623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 msgr2=0x56220e6f69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 -- 172.21.15.69:0/3707276623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 msgr2=0x56220e6e3f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6e3f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 -- 172.21.15.69:0/3707276623 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56220e6def00 con 0x56220e6eadb0 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd140faf640 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd134035000 tx=0x7fd13402f370 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f857067b517ce312 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd121ffb640 1 -- 172.21.15.69:0/3707276623 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134003970 con 0x56220e6eadb0 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd121ffb640 1 -- 172.21.15.69:0/3707276623 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd134036040 con 0x56220e6eadb0 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.411+0000 7fd121ffb640 1 -- 172.21.15.69:0/3707276623 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd13402f870 con 0x56220e6eadb0 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- 172.21.15.69:0/3707276623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 msgr2=0x56220e7016f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7fd134035000 tx=0x7fd13402f370 comp rx=0 tx=0).stop 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- 172.21.15.69:0/3707276623 shutdown_connections 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e7016f0 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 --2- 172.21.15.69:0/3707276623 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6e3f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- 172.21.15.69:0/3707276623 >> 172.21.15.69:0/3707276623 conn(0x56220e6e08c0 msgr2=0x56220e6e3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- 172.21.15.69:0/3707276623 shutdown_connections 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- 172.21.15.69:0/3707276623 wait complete. 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.412+0000 7fd1457f9740 1 -- start start 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 Processor -- start 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6f4560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1a66c0 con 0x56220e6eadb0 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1a7b30 con 0x56220e6e5250 2024-05-09T09:29:25.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd13c1a6d90 con 0x56220e6e5c40 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd1457f9740 10 client.0 _ll_register_callbacks cb 0x7fd134009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd13b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52744/0 (socket says 172.21.15.69:52744) 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd13bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6f4560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 -- 172.21.15.69:0/1453457921 learned_addr learned my addr 172.21.15.69:0/1453457921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 -- 172.21.15.69:0/1453457921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 msgr2=0x56220e6f4aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f4aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 -- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 msgr2=0x56220e6f4560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6f4560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 -- 172.21.15.69:0/1453457921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56220e6f60a0 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd13bfff640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd140faf640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd13402ade0 tx=0x7fd134004400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58ed44be1ec4bc62 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.413+0000 7fd11b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd134003b70 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd134003d10 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd134036040 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fd134004850 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 10 client.16905 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x56220e7613f0 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd134036040 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fd134049070 con 0x56220e6eadb0 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 10 client.16905 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 10 client.16905 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:25.419 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 20 client.16905 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e761e70 0x56220e764290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 4 client.16905 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56220e650dc0 con 0x56220e761e70 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.414+0000 7fd13bfff640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e761e70 0x56220e764290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.415+0000 7fd13bfff640 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e761e70 0x56220e764290 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17acace4b4638322 server_cookie=cfbdfdb145c08f64 in_seq=0 out_seq=0 2024-05-09T09:29:25.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.415+0000 7fd11b7fe640 10 client.16905 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:25.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 756589 8.38 MB/sec execute 443 sec latency 84.277 ms 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd11b7fe640 1 -- 172.21.15.69:0/1453457921 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56220e650dc0 con 0x56220e761e70 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd11b7fe640 10 client.16905 handle_command_reply: tid=0 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 client.16905 shutdown 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 msgr2=0x56220e6f5ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7fd13402ade0 tx=0x7fd134004400 comp rx=0 tx=0).stop 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 shutdown_connections 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56220e6eadb0 0x56220e6f5ab0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56220e761e70 0x56220e764290 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56220e6e5c40 0x56220e6f4aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 --2- 172.21.15.69:0/1453457921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56220e6e5250 0x56220e6f4560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd11b7fe640 0 client.16905 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:25.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 >> 172.21.15.69:0/1453457921 conn(0x56220e6e08c0 msgr2=0x56220e6e3870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 shutdown_connections 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 1 -- 172.21.15.69:0/1453457921 wait complete. 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.548+0000 7fd1457f9740 20 client.16905 trim_cache size 0 max 16384 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:29:25.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 msgr2=0x7fd1240a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 0x7fd1240a50f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd13c1028f0 tx=0x7fd12c012040 comp rx=0 tx=0).stop 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 msgr2=0x7fd13c197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7fd13000e9f0 tx=0x7fd13000edf0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 shutdown_connections 2024-05-09T09:29:25.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1240a2c30 0x7fd1240a50f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd13c103450 0x7fd13c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd13c102a90 0x7fd13c197fe0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 --2- 172.21.15.69:0/2966565147 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd13c1018a0 0x7fd13c197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 >> 172.21.15.69:0/2966565147 conn(0x7fd13c0fcdf0 msgr2=0x7fd13c105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 shutdown_connections 2024-05-09T09:29:25.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:25.549+0000 7fd142a12640 1 -- 172.21.15.69:0/2966565147 wait complete. 2024-05-09T09:29:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:25 smithi069 ceph-mon[30123]: pgmap v686: 129 pgs: 129 active+clean; 145 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:29:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:25 smithi163 ceph-mon[34354]: pgmap v686: 129 pgs: 129 active+clean; 145 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:29:25.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:25 smithi103 ceph-mon[34421]: pgmap v686: 129 pgs: 129 active+clean; 145 MiB data, 2.6 GiB used, 1.0 TiB / 1.0 TiB avail; 796 KiB/s rd, 2.6 MiB/s wr, 337 op/s 2024-05-09T09:29:26.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 758409 8.37 MB/sec execute 444 sec latency 120.091 ms 2024-05-09T09:29:27.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 760459 8.38 MB/sec execute 445 sec latency 15.022 ms 2024-05-09T09:29:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:27 smithi069 ceph-mon[30123]: pgmap v687: 129 pgs: 129 active+clean; 143 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-05-09T09:29:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:27 smithi163 ceph-mon[34354]: pgmap v687: 129 pgs: 129 active+clean; 143 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-05-09T09:29:27.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:27 smithi103 ceph-mon[34421]: pgmap v687: 129 pgs: 129 active+clean; 143 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 925 KiB/s rd, 3.0 MiB/s wr, 378 op/s 2024-05-09T09:29:28.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 763040 8.39 MB/sec execute 446 sec latency 20.065 ms 2024-05-09T09:29:29.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 764142 8.39 MB/sec execute 447 sec latency 50.602 ms 2024-05-09T09:29:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:29 smithi069 ceph-mon[30123]: pgmap v688: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-05-09T09:29:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:29 smithi163 ceph-mon[34354]: pgmap v688: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-05-09T09:29:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:29 smithi103 ceph-mon[34421]: pgmap v688: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 843 KiB/s rd, 2.9 MiB/s wr, 347 op/s 2024-05-09T09:29:30.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 766582 8.40 MB/sec execute 448 sec latency 38.485 ms 2024-05-09T09:29:31.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 767925 8.40 MB/sec execute 449 sec latency 12.755 ms 2024-05-09T09:29:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:31 smithi069 ceph-mon[30123]: pgmap v689: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.7 MiB/s wr, 348 op/s 2024-05-09T09:29:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:31 smithi163 ceph-mon[34354]: pgmap v689: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.7 MiB/s wr, 348 op/s 2024-05-09T09:29:31.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:31 smithi103 ceph-mon[34421]: pgmap v689: 129 pgs: 129 active+clean; 145 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 800 KiB/s rd, 2.7 MiB/s wr, 348 op/s 2024-05-09T09:29:32.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 770711 8.42 MB/sec execute 450 sec latency 14.685 ms 2024-05-09T09:29:33.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 771654 8.40 MB/sec execute 451 sec latency 165.310 ms 2024-05-09T09:29:33.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:33 smithi069 ceph-mon[30123]: pgmap v690: 129 pgs: 129 active+clean; 146 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-05-09T09:29:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:33 smithi163 ceph-mon[34354]: pgmap v690: 129 pgs: 129 active+clean; 146 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-05-09T09:29:33.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:33 smithi103 ceph-mon[34421]: pgmap v690: 129 pgs: 129 active+clean; 146 MiB data, 2.7 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 3.1 MiB/s wr, 349 op/s 2024-05-09T09:29:34.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 774184 8.42 MB/sec execute 452 sec latency 26.362 ms 2024-05-09T09:29:35.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 775212 8.41 MB/sec execute 453 sec latency 27.607 ms 2024-05-09T09:29:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:35 smithi069 ceph-mon[30123]: pgmap v691: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-05-09T09:29:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:35 smithi163 ceph-mon[34354]: pgmap v691: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-05-09T09:29:35.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:35 smithi103 ceph-mon[34421]: pgmap v691: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 763 KiB/s rd, 2.8 MiB/s wr, 348 op/s 2024-05-09T09:29:36.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 777819 8.43 MB/sec execute 454 sec latency 24.589 ms 2024-05-09T09:29:37.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 779239 8.42 MB/sec execute 455 sec latency 13.642 ms 2024-05-09T09:29:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:37 smithi069 ceph-mon[30123]: pgmap v692: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 2.9 MiB/s wr, 338 op/s 2024-05-09T09:29:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:37 smithi163 ceph-mon[34354]: pgmap v692: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 2.9 MiB/s wr, 338 op/s 2024-05-09T09:29:37.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:37 smithi103 ceph-mon[34421]: pgmap v692: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 706 KiB/s rd, 2.9 MiB/s wr, 338 op/s 2024-05-09T09:29:38.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 781531 8.43 MB/sec execute 456 sec latency 10.888 ms 2024-05-09T09:29:38.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 -- 172.21.15.69:0/1693642942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20102370 msgr2=0x7f8a20104760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 --2- 172.21.15.69:0/1693642942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20102370 0x7f8a20104760 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009900 tx=0x7f8a1802a410 comp rx=0 tx=0).stop 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 -- 172.21.15.69:0/1693642942 shutdown_connections 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 --2- 172.21.15.69:0/1693642942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a201075d0 0x7f8a201099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 --2- 172.21.15.69:0/1693642942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20107090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 --2- 172.21.15.69:0/1693642942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a20102370 0x7f8a20104760 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 -- 172.21.15.69:0/1693642942 >> 172.21.15.69:0/1693642942 conn(0x7f8a200fbd00 msgr2=0x7f8a200fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 -- 172.21.15.69:0/1693642942 shutdown_connections 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.640+0000 7f8a24c15640 1 -- 172.21.15.69:0/1693642942 wait complete. 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 Processor -- start 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 -- start start 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 0x7f8a20197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d500 con 0x7f8a201075d0 2024-05-09T09:29:38.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d380 con 0x7f8a20104ca0 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a24c15640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a2010d680 con 0x7f8a20102370 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.641+0000 7f8a1ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 0x7f8a20197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 0x7f8a20197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:55174/0 (socket says 172.21.15.69:55174) 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20198050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:48718/0 (socket says 172.21.15.69:48718) 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39458/0 (socket says 172.21.15.69:39458) 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1e59c640 1 -- 172.21.15.69:0/2924965782 learned_addr learned my addr 172.21.15.69:0/2924965782 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:38.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 -- 172.21.15.69:0/2924965782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 msgr2=0x7f8a20197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 0x7f8a20197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 -- 172.21.15.69:0/2924965782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 msgr2=0x7f8a20198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 -- 172.21.15.69:0/2924965782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a2019f730 con 0x7f8a201075d0 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a1ed9d640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000c970 tx=0x7f8a1000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a0f7fe640 1 -- 172.21.15.69:0/2924965782 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10007bb0 con 0x7f8a201075d0 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a0f7fe640 1 -- 172.21.15.69:0/2924965782 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a10007d50 con 0x7f8a201075d0 2024-05-09T09:29:38.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a2019fa20 con 0x7f8a201075d0 2024-05-09T09:29:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a0f7fe640 1 -- 172.21.15.69:0/2924965782 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a10018770 con 0x7f8a201075d0 2024-05-09T09:29:38.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.642+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a2019ff20 con 0x7f8a201075d0 2024-05-09T09:29:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a0f7fe640 1 -- 172.21.15.69:0/2924965782 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8a1000f450 con 0x7f8a201075d0 2024-05-09T09:29:38.658 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 Processor -- start 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 -- start start 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536dbb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536dc1c0 0x5570536de650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557053636340 con 0x5570536d6660 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570535de290 con 0x5570536dc1c0 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a279fe740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557053632300 con 0x5570536d7050 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39468/0 (socket says 172.21.15.69:39468) 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.645+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/701949 learned_addr learned my addr 172.21.15.69:0/701949 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a0f7fe640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 0x7f8a080a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536dc1c0 0x5570536de650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.659 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536dbb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1ed9d640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 0x7f8a080a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/701949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 msgr2=0x5570536dbb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536dbb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/701949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536dc1c0 msgr2=0x5570536de650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536dc1c0 0x5570536de650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/701949 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570536028e0 con 0x5570536d6660 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a0f7fe640 1 -- 172.21.15.69:0/2924965782 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a100cf330 con 0x7f8a201075d0 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005c40 tx=0x7f8a14005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cd0b8aea2d6f4874 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a1ed9d640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 0x7f8a080a50f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a100cce20 tx=0x7f8a10075040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a0cff9640 1 -- 172.21.15.69:0/701949 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14003970 con 0x5570536d6660 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a0cff9640 1 -- 172.21.15.69:0/701949 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a140118a0 con 0x5570536d6660 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.646+0000 7f8a0cff9640 1 -- 172.21.15.69:0/701949 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14004630 con 0x5570536d6660 2024-05-09T09:29:38.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- 172.21.15.69:0/701949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 msgr2=0x5570536d6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7f8a14005c40 tx=0x7f8a14005bd0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- 172.21.15.69:0/701949 shutdown_connections 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536dc1c0 0x5570536de650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536dbb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 --2- 172.21.15.69:0/701949 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d6660 0x5570536d6a80 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- 172.21.15.69:0/701949 >> 172.21.15.69:0/701949 conn(0x5570536cd8c0 msgr2=0x5570536cdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- 172.21.15.69:0/701949 shutdown_connections 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- 172.21.15.69:0/701949 wait complete. 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 -- start start 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.647+0000 7f8a279fe740 1 Processor -- start 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e7cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5570536df020 con 0x5570536d7050 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5570535f1790 con 0x5570536d6660 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5570536df1a0 con 0x5570536dc1c0 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 10 client.0 _ll_register_callbacks cb 0x55705362ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39478/0 (socket says 172.21.15.69:39478) 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 -- 172.21.15.69:0/3483825950 learned_addr learned my addr 172.21.15.69:0/3483825950 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e7cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 -- 172.21.15.69:0/3483825950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 msgr2=0x5570536e7cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e7cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 -- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 msgr2=0x5570536e0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e0770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 -- 172.21.15.69:0/3483825950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55705334a210 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009a20 tx=0x7f8a18031a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d76dfc0e8dc9b528 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18035040 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a18004380 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a140118a0 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.648+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8a1804f050 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 10 client.16917 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x557053416620 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a140118a0 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f8a18004cc0 con 0x5570536d7050 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 10 client.16917 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:38.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 10 client.16917 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 20 client.16917 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5570536ec270 0x5570536ee6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 4 client.16917 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557053643a70 con 0x5570536ec270 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5570536ec270 0x5570536ee6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a05ffb640 10 client.16917 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.649+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5570536ec270 0x5570536ee6f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b4a6fd75ac1f537 server_cookie=47746b843aacf00c in_seq=0 out_seq=0 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.650+0000 7f8a05ffb640 1 -- 172.21.15.69:0/3483825950 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557053643a70 con 0x5570536ec270 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.650+0000 7f8a05ffb640 10 client.16917 handle_command_reply: tid=0 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.650+0000 7f8a279fe740 1 client.16917 shutdown 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 msgr2=0x5570536e0cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7f8a18009a20 tx=0x7f8a18031a90 comp rx=0 tx=0).stop 2024-05-09T09:29:38.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 shutdown_connections 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5570536ec270 0x5570536ee6f0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e7cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a05ffb640 0 client.16917 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e0cb0 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 --2- 172.21.15.69:0/3483825950 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 >> 172.21.15.69:0/3483825950 conn(0x5570536cd8c0 msgr2=0x5570536cca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 shutdown_connections 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 1 -- 172.21.15.69:0/3483825950 wait complete. 2024-05-09T09:29:38.665 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.651+0000 7f8a279fe740 20 client.16917 trim_cache size 0 max 16384 2024-05-09T09:29:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.665+0000 7f8a279fe740 1 Processor -- start 2024-05-09T09:29:38.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 -- start start 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536de5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536e7cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201aa970 con 0x5570536dc1c0 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201a8240 con 0x5570536d6660 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a279fe740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201a6090 con 0x5570536d7050 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536e7cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39490/0 (socket says 172.21.15.69:39490) 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/2334503467 learned_addr learned my addr 172.21.15.69:0/2334503467 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536de5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/2334503467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 msgr2=0x5570536e7cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536e7cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/2334503467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 msgr2=0x5570536de5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536de5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 -- 172.21.15.69:0/2334503467 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5570536cbf00 con 0x5570536dc1c0 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009f10 tx=0x7f8a14015040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c8421339b30fd00 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a07fff640 1 -- 172.21.15.69:0/2334503467 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14011920 con 0x5570536dc1c0 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a07fff640 1 -- 172.21.15.69:0/2334503467 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14017030 con 0x5570536dc1c0 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.666+0000 7f8a07fff640 1 -- 172.21.15.69:0/2334503467 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a14005540 con 0x5570536dc1c0 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- 172.21.15.69:0/2334503467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 msgr2=0x5570536cc170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7f8a14009f10 tx=0x7f8a14015040 comp rx=0 tx=0).stop 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- 172.21.15.69:0/2334503467 shutdown_connections 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536dc1c0 0x5570536cc170 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536d7050 0x5570536e7cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2334503467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536de5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- 172.21.15.69:0/2334503467 >> 172.21.15.69:0/2334503467 conn(0x5570536cd8c0 msgr2=0x5570536db800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- 172.21.15.69:0/2334503467 shutdown_connections 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- 172.21.15.69:0/2334503467 wait complete. 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 -- start start 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.667+0000 7f8a279fe740 1 Processor -- start 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e4fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e64f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201a6700 con 0x5570536d7050 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201a7b70 con 0x5570536d6660 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a201a6dd0 con 0x5570536dc1c0 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 10 client.0 _ll_register_callbacks cb 0x7f8a18009640 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:38.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39498/0 (socket says 172.21.15.69:39498) 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 -- 172.21.15.69:0/2225434253 learned_addr learned my addr 172.21.15.69:0/2225434253 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e4fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e64f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 -- 172.21.15.69:0/2225434253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 msgr2=0x5570536e64f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e64f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 -- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 msgr2=0x5570536e4fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e4fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 -- 172.21.15.69:0/2225434253 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5570536e6ae0 con 0x5570536d7050 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a1e59c640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006010 tx=0x7f8a18003fc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aec461c882d09dea server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:38.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a05ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a18035070 con 0x5570536d7050 2024-05-09T09:29:38.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a1804c040 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a14017030 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f8a18061050 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 10 client.16923 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.668+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5570536e7e20 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a14017030 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f8a18005450 con 0x5570536d7050 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 10 client.16923 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 10 client.16923 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 20 client.16923 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:38.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557053752e00 0x557053755220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 4 client.16923 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:29:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55705363ddc0 con 0x557053752e00 2024-05-09T09:29:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557053752e00 0x557053755220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a1dd9b640 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557053752e00 0x557053755220 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e5c0b236050d0be server_cookie=dcb506236e7b63a4 in_seq=0 out_seq=0 2024-05-09T09:29:38.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.669+0000 7f8a05ffb640 10 client.16923 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a05ffb640 1 -- 172.21.15.69:0/2225434253 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55705363ddc0 con 0x557053752e00 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a05ffb640 10 client.16923 handle_command_reply: tid=0 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 client.16923 shutdown 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 msgr2=0x5570536e54e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7f8a18006010 tx=0x7f8a18003fc0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 shutdown_connections 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557053752e00 0x557053755220 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5570536dc1c0 0x5570536e64f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5570536d7050 0x5570536e54e0 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a05ffb640 0 client.16923 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 --2- 172.21.15.69:0/2225434253 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5570536d6660 0x5570536e4fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 >> 172.21.15.69:0/2225434253 conn(0x5570536cd8c0 msgr2=0x5570536d0640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 shutdown_connections 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 1 -- 172.21.15.69:0/2225434253 wait complete. 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.850+0000 7f8a279fe740 20 client.16923 trim_cache size 0 max 16384 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:29:38.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:29:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:29:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 msgr2=0x7f8a080a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 0x7f8a080a50f0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f8a100cce20 tx=0x7f8a10075040 comp rx=0 tx=0).stop 2024-05-09T09:29:38.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 msgr2=0x7f8a2019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7f8a1000c970 tx=0x7f8a1000ce30 comp rx=0 tx=0).stop 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 shutdown_connections 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a080a2c30 0x7f8a080a50f0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a201075d0 0x7f8a2019f0c0 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a20104ca0 0x7f8a20198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 --2- 172.21.15.69:0/2924965782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a20102370 0x7f8a20197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 >> 172.21.15.69:0/2924965782 conn(0x7f8a200fbd00 msgr2=0x7f8a20102cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 shutdown_connections 2024-05-09T09:29:38.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:38.851+0000 7f8a24c15640 1 -- 172.21.15.69:0/2924965782 wait complete. 2024-05-09T09:29:39.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 782825 8.43 MB/sec execute 457 sec latency 67.762 ms 2024-05-09T09:29:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:39 smithi069 ceph-mon[30123]: pgmap v693: 129 pgs: 129 active+clean; 153 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-05-09T09:29:39.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:39 smithi163 ceph-mon[34354]: pgmap v693: 129 pgs: 129 active+clean; 153 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-05-09T09:29:39.870 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:39 smithi103 ceph-mon[34421]: pgmap v693: 129 pgs: 129 active+clean; 153 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 833 KiB/s rd, 2.9 MiB/s wr, 327 op/s 2024-05-09T09:29:40.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 785002 8.44 MB/sec execute 458 sec latency 25.678 ms 2024-05-09T09:29:41.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 786773 8.43 MB/sec execute 459 sec latency 18.813 ms 2024-05-09T09:29:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:41 smithi069 ceph-mon[30123]: pgmap v694: 129 pgs: 129 active+clean; 154 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-05-09T09:29:41.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:41 smithi163 ceph-mon[34354]: pgmap v694: 129 pgs: 129 active+clean; 154 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-05-09T09:29:41.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:41 smithi103 ceph-mon[34421]: pgmap v694: 129 pgs: 129 active+clean; 154 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 778 KiB/s rd, 2.7 MiB/s wr, 289 op/s 2024-05-09T09:29:42.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 788761 8.44 MB/sec execute 460 sec latency 10.233 ms 2024-05-09T09:29:43.502 INFO:tasks.workunit.client.0.smithi069.stdout: 1 791106 8.45 MB/sec execute 461 sec latency 18.689 ms 2024-05-09T09:29:43.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:43 smithi103 ceph-mon[34421]: pgmap v695: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:43 smithi069 ceph-mon[30123]: pgmap v695: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:43 smithi163 ceph-mon[34354]: pgmap v695: 129 pgs: 129 active+clean; 149 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 803 KiB/s rd, 2.9 MiB/s wr, 333 op/s 2024-05-09T09:29:44.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 792439 8.45 MB/sec execute 462 sec latency 61.665 ms 2024-05-09T09:29:45.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 794648 8.45 MB/sec execute 463 sec latency 37.906 ms 2024-05-09T09:29:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:45 smithi069 ceph-mon[30123]: pgmap v696: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.8 MiB/s wr, 384 op/s 2024-05-09T09:29:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:45 smithi163 ceph-mon[34354]: pgmap v696: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.8 MiB/s wr, 384 op/s 2024-05-09T09:29:45.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:45 smithi103 ceph-mon[34421]: pgmap v696: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 938 KiB/s rd, 2.8 MiB/s wr, 384 op/s 2024-05-09T09:29:46.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 796062 8.45 MB/sec execute 464 sec latency 15.262 ms 2024-05-09T09:29:47.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 798828 8.47 MB/sec execute 465 sec latency 12.725 ms 2024-05-09T09:29:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:47 smithi069 ceph-mon[30123]: pgmap v697: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:29:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:47 smithi163 ceph-mon[34354]: pgmap v697: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:29:47.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:47 smithi103 ceph-mon[34421]: pgmap v697: 129 pgs: 129 active+clean; 151 MiB data, 2.8 GiB used, 1.0 TiB / 1.0 TiB avail; 773 KiB/s rd, 2.9 MiB/s wr, 340 op/s 2024-05-09T09:29:48.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 800041 8.46 MB/sec execute 466 sec latency 12.130 ms 2024-05-09T09:29:49.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 802620 8.48 MB/sec execute 467 sec latency 73.915 ms 2024-05-09T09:29:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:49 smithi069 ceph-mon[30123]: pgmap v698: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-05-09T09:29:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:49 smithi163 ceph-mon[34354]: pgmap v698: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-05-09T09:29:49.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:49 smithi103 ceph-mon[34421]: pgmap v698: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 891 KiB/s rd, 2.7 MiB/s wr, 350 op/s 2024-05-09T09:29:50.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 803566 8.47 MB/sec execute 468 sec latency 36.064 ms 2024-05-09T09:29:51.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 806257 8.48 MB/sec execute 469 sec latency 15.274 ms 2024-05-09T09:29:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:51 smithi069 ceph-mon[30123]: pgmap v699: 129 pgs: 129 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.2 MiB/s wr, 312 op/s 2024-05-09T09:29:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:51 smithi163 ceph-mon[34354]: pgmap v699: 129 pgs: 129 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.2 MiB/s wr, 312 op/s 2024-05-09T09:29:51.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:51 smithi103 ceph-mon[34421]: pgmap v699: 129 pgs: 129 active+clean; 154 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 2.2 MiB/s wr, 312 op/s 2024-05-09T09:29:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.944+0000 7fc975b14640 1 -- 172.21.15.69:0/1380816804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970100790 msgr2=0x7fc970100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.944+0000 7fc975b14640 1 --2- 172.21.15.69:0/1380816804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970100790 0x7fc970100bb0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fc960009900 tx=0x7fc96002a410 comp rx=0 tx=0).stop 2024-05-09T09:29:51.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- 172.21.15.69:0/1380816804 shutdown_connections 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- 172.21.15.69:0/1380816804 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970102340 0x7fc970108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- 172.21.15.69:0/1380816804 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 0x7fc970101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- 172.21.15.69:0/1380816804 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970100790 0x7fc970100bb0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- 172.21.15.69:0/1380816804 >> 172.21.15.69:0/1380816804 conn(0x7fc9700fbd00 msgr2=0x7fc9700fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- 172.21.15.69:0/1380816804 shutdown_connections 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- 172.21.15.69:0/1380816804 wait complete. 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 Processor -- start 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- start start 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970100790 0x7fc970197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 0x7fc970198040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc97010d530 con 0x7fc970102340 2024-05-09T09:29:51.952 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc97010d3b0 con 0x7fc970101980 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.945+0000 7fc975b14640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc97010d6b0 con 0x7fc970100790 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36248/0 (socket says 172.21.15.69:36248) 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 -- 172.21.15.69:0/3269936477 learned_addr learned my addr 172.21.15.69:0/3269936477 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96effd640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 0x7fc970198040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970100790 0x7fc970197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 -- 172.21.15.69:0/3269936477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970100790 msgr2=0x7fc970197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970100790 0x7fc970197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 -- 172.21.15.69:0/3269936477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 msgr2=0x7fc970198040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 0x7fc970198040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 -- 172.21.15.69:0/3269936477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc97019f720 con 0x7fc970102340 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96ffff640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc96800eb10 tx=0x7fc96800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.953 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96cff9640 1 -- 172.21.15.69:0/3269936477 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968004b20 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96cff9640 1 -- 172.21.15.69:0/3269936477 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc968004cc0 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc97019fa10 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc96cff9640 1 -- 172.21.15.69:0/3269936477 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc968010760 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.946+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc97019ff10 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.948+0000 7fc96cff9640 1 -- 172.21.15.69:0/3269936477 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc968005020 con 0x7fc970102340 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 Processor -- start 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 -- start start 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba641740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba644240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5583ba5a0340 con 0x5583ba63c250 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5583ba548290 con 0x5583ba63cc40 2024-05-09T09:29:51.954 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc978823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5583ba59c300 con 0x5583ba641db0 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36264/0 (socket says 172.21.15.69:36264) 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96effd640 1 -- 172.21.15.69:0/2575708415 learned_addr learned my addr 172.21.15.69:0/2575708415 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba641740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96effd640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba644240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96cff9640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 0x7fc9580a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96ffff640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 0x7fc9580a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 -- 172.21.15.69:0/2575708415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 msgr2=0x5583ba644240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba644240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 -- 172.21.15.69:0/2575708415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 msgr2=0x5583ba63c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96f7fe640 1 -- 172.21.15.69:0/2575708415 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583ba56c8e0 con 0x5583ba63cc40 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.949+0000 7fc96effd640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:51.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc96cff9640 1 -- 172.21.15.69:0/3269936477 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc9680cfa20 con 0x7fc970102340 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba641740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc960009230 tx=0x7fc960005e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=337bd33f3ee546f7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc96ffff640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 0x7fc9580a50f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc9680a4640 tx=0x7fc96806bdb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc955ffb640 1 -- 172.21.15.69:0/2575708415 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc96003f070 con 0x5583ba63cc40 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc955ffb640 1 -- 172.21.15.69:0/2575708415 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc960004030 con 0x5583ba63cc40 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.950+0000 7fc955ffb640 1 -- 172.21.15.69:0/2575708415 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960035070 con 0x5583ba63cc40 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- 172.21.15.69:0/2575708415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 msgr2=0x5583ba641740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba641740 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fc960009230 tx=0x7fc960005e80 comp rx=0 tx=0).stop 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- 172.21.15.69:0/2575708415 shutdown_connections 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba644240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba641740 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 --2- 172.21.15.69:0/2575708415 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- 172.21.15.69:0/2575708415 >> 172.21.15.69:0/2575708415 conn(0x5583ba6378c0 msgr2=0x5583ba637cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- 172.21.15.69:0/2575708415 shutdown_connections 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- 172.21.15.69:0/2575708415 wait complete. 2024-05-09T09:29:51.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 -- start start 2024-05-09T09:29:51.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.951+0000 7fc978823740 1 Processor -- start 2024-05-09T09:29:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba646470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64d9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5583ba644d00 con 0x5583ba63cc40 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5583ba55b790 con 0x5583ba63c250 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5583ba644e80 con 0x5583ba641db0 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 10 client.0 _ll_register_callbacks cb 0x5583ba596a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36274/0 (socket says 172.21.15.69:36274) 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3140595024 learned_addr learned my addr 172.21.15.69:0/3140595024 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96effd640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba646470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96effd640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64d9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3140595024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 msgr2=0x5583ba64d9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64d9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 msgr2=0x5583ba646470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba646470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3140595024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ba2b4210 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc960009900 tx=0x7fc9600041e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=776acf63ad01e1a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc94effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960051060 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc9600318b0 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.952+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc960004030 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc960058050 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 10 client.16935 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5583ba380620 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc960004030 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fc96004b030 con 0x5583ba63cc40 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 10 client.16935 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 10 client.16935 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 20 client.16935 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:51.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba651b50 0x5583ba653f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 4 client.16935 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5583ba5ada70 con 0x5583ba651b50 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.953+0000 7fc96effd640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba651b50 0x5583ba653f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc94effd640 10 client.16935 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc96effd640 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba651b50 0x5583ba653f40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f16e93d6dfa18aef server_cookie=734d67c8f07d999e in_seq=0 out_seq=0 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc94effd640 1 -- 172.21.15.69:0/3140595024 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5583ba5ada70 con 0x5583ba651b50 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc94effd640 10 client.16935 handle_command_reply: tid=0 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc978823740 1 client.16935 shutdown 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 msgr2=0x5583ba6469b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.954+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7fc960009900 tx=0x7fc9600041e0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 shutdown_connections 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba651b50 0x5583ba653f40 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64d9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba6469b0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 --2- 172.21.15.69:0/3140595024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba646470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 >> 172.21.15.69:0/3140595024 conn(0x5583ba6378c0 msgr2=0x5583ba636a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc94effd640 0 client.16935 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 shutdown_connections 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 1 -- 172.21.15.69:0/3140595024 wait complete. 2024-05-09T09:29:51.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.955+0000 7fc978823740 20 client.16935 trim_cache size 0 max 16384 2024-05-09T09:29:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 Processor -- start 2024-05-09T09:29:51.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 -- start start 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba653b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701acc20 con 0x5583ba63c250 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701aa4f0 con 0x5583ba63cc40 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc978823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701a8340 con 0x5583ba641db0 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc96effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40896/0 (socket says 172.21.15.69:40896) 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3183488746 learned_addr learned my addr 172.21.15.69:0/3183488746 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.974+0000 7fc96effd640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36276/0 (socket says 172.21.15.69:36276) 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96effd640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba653b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3183488746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 msgr2=0x5583ba653b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba653b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3183488746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 msgr2=0x5583ba63de80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3183488746 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5583ba635f00 con 0x5583ba63cc40 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96effd640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc960031400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cfbfddb5bbed5c8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc9557fa640 1 -- 172.21.15.69:0/3183488746 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960051070 con 0x5583ba63cc40 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc9557fa640 1 -- 172.21.15.69:0/3183488746 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc96003b430 con 0x5583ba63cc40 2024-05-09T09:29:51.977 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.975+0000 7fc9557fa640 1 -- 172.21.15.69:0/3183488746 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960035070 con 0x5583ba63cc40 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- 172.21.15.69:0/3183488746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 msgr2=0x5583ba63e3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fc960000c00 tx=0x7fc960031400 comp rx=0 tx=0).stop 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- 172.21.15.69:0/3183488746 shutdown_connections 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba653b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63cc40 0x5583ba63e3c0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 --2- 172.21.15.69:0/3183488746 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63c250 0x5583ba63de80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- 172.21.15.69:0/3183488746 >> 172.21.15.69:0/3183488746 conn(0x5583ba6378c0 msgr2=0x5583ba6414d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- 172.21.15.69:0/3183488746 shutdown_connections 2024-05-09T09:29:51.981 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- 172.21.15.69:0/3183488746 wait complete. 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.976+0000 7fc978823740 1 -- start start 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 Processor -- start 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba64a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701a89b0 con 0x5583ba63cc40 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701a9e20 con 0x5583ba63c250 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc9701a9080 con 0x5583ba641db0 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc978823740 10 client.0 _ll_register_callbacks cb 0x7fc964009b00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36284/0 (socket says 172.21.15.69:36284) 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3442783537 learned_addr learned my addr 172.21.15.69:0/3442783537 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96effd640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba64a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.982 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96effd640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3442783537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 msgr2=0x5583ba64b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 msgr2=0x5583ba64a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba64a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96f7fe640 1 -- 172.21.15.69:0/3442783537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5583ba64bc60 con 0x5583ba63cc40 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.977+0000 7fc96effd640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba64a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc96f7fe640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc96002a930 tx=0x7fc96003ba60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=769223478e4d8b09 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc960035070 con 0x5583ba63cc40 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc960003970 con 0x5583ba63cc40 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x5583ba5a7dc0 con 0x5583ba63cc40 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7fc96006a050 con 0x5583ba63cc40 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 10 client.16941 fetch_fsmap learned FSMap version 39 2024-05-09T09:29:51.983 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5583ba64cfa0 con 0x5583ba63cc40 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x5583ba5a7dc0 con 0x5583ba63cc40 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7fc96004b070 con 0x5583ba63cc40 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 10 client.16941 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 10 client.16941 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 20 client.16941 populate_metadata read hostname 'smithi069' 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba6bc8a0 0x5583ba6becc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 4 client.16941 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5583ba5258e0 con 0x5583ba6bc8a0 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.978+0000 7fc96effd640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba6bc8a0 0x5583ba6becc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.979+0000 7fc96effd640 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba6bc8a0 0x5583ba6becc0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=48dbc506855a98b6 server_cookie=97f1b7b21ce196da in_seq=0 out_seq=0 2024-05-09T09:29:51.984 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:51.979+0000 7fc94effd640 10 client.16941 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.109+0000 7fc94effd640 1 -- 172.21.15.69:0/3442783537 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5583ba5258e0 con 0x5583ba6bc8a0 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc94effd640 10 client.16941 handle_command_reply: tid=0 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 client.16941 shutdown 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 msgr2=0x5583ba64a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7fc96002a930 tx=0x7fc96003ba60 comp rx=0 tx=0).stop 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 shutdown_connections 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5583ba6bc8a0 0x5583ba6becc0 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5583ba641db0 0x5583ba64b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5583ba63cc40 0x5583ba64a6b0 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc94effd640 0 client.16941 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 --2- 172.21.15.69:0/3442783537 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5583ba63c250 0x5583ba64a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 >> 172.21.15.69:0/3442783537 conn(0x5583ba6378c0 msgr2=0x5583ba646450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:52.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 shutdown_connections 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 1 -- 172.21.15.69:0/3442783537 wait complete. 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.110+0000 7fc978823740 20 client.16941 trim_cache size 0 max 16384 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:29:52.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 msgr2=0x7fc9580a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 0x7fc9580a50f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc9680a4640 tx=0x7fc96806bdb0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 msgr2=0x7fc97019f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7fc96800eb10 tx=0x7fc96800c2d0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 shutdown_connections 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc9580a2c30 0x7fc9580a50f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc970102340 0x7fc97019f0b0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc970101980 0x7fc970198040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 --2- 172.21.15.69:0/3269936477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc970100790 0x7fc970197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:29:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 >> 172.21.15.69:0/3269936477 conn(0x7fc9700fbd00 msgr2=0x7fc970104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:29:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 shutdown_connections 2024-05-09T09:29:52.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:29:52.111+0000 7fc975b14640 1 -- 172.21.15.69:0/3269936477 wait complete. 2024-05-09T09:29:52.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 807780 8.48 MB/sec execute 470 sec latency 108.568 ms 2024-05-09T09:29:53.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 810044 8.49 MB/sec execute 471 sec latency 9.377 ms 2024-05-09T09:29:53.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:53 smithi069 ceph-mon[30123]: pgmap v700: 129 pgs: 129 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:29:53.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:53 smithi163 ceph-mon[34354]: pgmap v700: 129 pgs: 129 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:29:53.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:53 smithi103 ceph-mon[34421]: pgmap v700: 129 pgs: 129 active+clean; 155 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:29:54.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 812524 8.50 MB/sec execute 472 sec latency 42.745 ms 2024-05-09T09:29:55.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 813766 8.49 MB/sec execute 473 sec latency 55.783 ms 2024-05-09T09:29:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:55 smithi069 ceph-mon[30123]: pgmap v701: 129 pgs: 129 active+clean; 147 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:29:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:55 smithi163 ceph-mon[34354]: pgmap v701: 129 pgs: 129 active+clean; 147 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:29:55.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:55 smithi103 ceph-mon[34421]: pgmap v701: 129 pgs: 129 active+clean; 147 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 869 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:29:56.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 816661 8.51 MB/sec execute 474 sec latency 15.818 ms 2024-05-09T09:29:57.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 817844 8.50 MB/sec execute 475 sec latency 25.024 ms 2024-05-09T09:29:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:57 smithi069 ceph-mon[30123]: pgmap v702: 129 pgs: 129 active+clean; 150 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:29:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:57 smithi163 ceph-mon[34354]: pgmap v702: 129 pgs: 129 active+clean; 150 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:29:57.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:57 smithi103 ceph-mon[34421]: pgmap v702: 129 pgs: 129 active+clean; 150 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 834 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:29:58.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 820457 8.52 MB/sec execute 476 sec latency 14.484 ms 2024-05-09T09:29:59.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 822553 8.52 MB/sec execute 477 sec latency 14.612 ms 2024-05-09T09:29:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:29:59 smithi069 ceph-mon[30123]: pgmap v703: 129 pgs: 129 active+clean; 151 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:29:59.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:29:59 smithi163 ceph-mon[34354]: pgmap v703: 129 pgs: 129 active+clean; 151 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:29:59.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:29:59 smithi103 ceph-mon[34421]: pgmap v703: 129 pgs: 129 active+clean; 151 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 948 KiB/s rd, 2.8 MiB/s wr, 353 op/s 2024-05-09T09:30:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:29:59.998+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:30:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:29:59.998+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-05-09T09:30:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:29:59.998+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : mds.h(mds.2): Metadata damage detected 2024-05-09T09:30:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:29:59.998+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : mds.i(mds.1): Metadata damage detected 2024-05-09T09:30:00.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 824152 8.52 MB/sec execute 478 sec latency 43.383 ms 2024-05-09T09:30:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-mon[30123]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:30:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-mon[30123]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-05-09T09:30:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-mon[30123]: mds.h(mds.2): Metadata damage detected 2024-05-09T09:30:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:00 smithi069 ceph-mon[30123]: mds.i(mds.1): Metadata damage detected 2024-05-09T09:30:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:00 smithi163 ceph-mon[34354]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:30:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:00 smithi163 ceph-mon[34354]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-05-09T09:30:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:00 smithi163 ceph-mon[34354]: mds.h(mds.2): Metadata damage detected 2024-05-09T09:30:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:00 smithi163 ceph-mon[34354]: mds.i(mds.1): Metadata damage detected 2024-05-09T09:30:00.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:00 smithi103 ceph-mon[34421]: Health detail: HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:30:00.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:00 smithi103 ceph-mon[34421]: [ERR] MDS_DAMAGE: 2 MDSs report damaged metadata 2024-05-09T09:30:00.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:00 smithi103 ceph-mon[34421]: mds.h(mds.2): Metadata damage detected 2024-05-09T09:30:00.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:00 smithi103 ceph-mon[34421]: mds.i(mds.1): Metadata damage detected 2024-05-09T09:30:01.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 826599 8.53 MB/sec execute 479 sec latency 14.664 ms 2024-05-09T09:30:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:01 smithi103 ceph-mon[34421]: pgmap v704: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.7 MiB/s wr, 339 op/s 2024-05-09T09:30:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:01 smithi069 ceph-mon[30123]: pgmap v704: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.7 MiB/s wr, 339 op/s 2024-05-09T09:30:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:01 smithi163 ceph-mon[34354]: pgmap v704: 129 pgs: 129 active+clean; 152 MiB data, 2.9 GiB used, 1.0 TiB / 1.0 TiB avail; 856 KiB/s rd, 2.7 MiB/s wr, 339 op/s 2024-05-09T09:30:02.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 828141 8.53 MB/sec execute 480 sec latency 13.353 ms 2024-05-09T09:30:03.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 830905 8.55 MB/sec execute 481 sec latency 13.974 ms 2024-05-09T09:30:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:03 smithi069 ceph-mon[30123]: pgmap v705: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-05-09T09:30:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:03 smithi163 ceph-mon[34354]: pgmap v705: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-05-09T09:30:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:03 smithi103 ceph-mon[34421]: pgmap v705: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 879 KiB/s rd, 3.1 MiB/s wr, 347 op/s 2024-05-09T09:30:04.503 INFO:tasks.workunit.client.0.smithi069.stdout: 1 832305 8.54 MB/sec execute 482 sec latency 15.577 ms 2024-05-09T09:30:05.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.208+0000 7f4a5e796640 1 -- 172.21.15.69:0/2577788327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58100790 msgr2=0x7f4a58100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.208+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2577788327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58100790 0x7f4a58100bb0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f4a48009900 tx=0x7f4a4802a410 comp rx=0 tx=0).stop 2024-05-09T09:30:05.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.208+0000 7f4a5e796640 1 -- 172.21.15.69:0/2577788327 shutdown_connections 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.208+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2577788327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58102340 0x7f4a58108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2577788327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2577788327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58100790 0x7f4a58100bb0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 -- 172.21.15.69:0/2577788327 >> 172.21.15.69:0/2577788327 conn(0x7f4a580fbd00 msgr2=0x7f4a580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 -- 172.21.15.69:0/2577788327 shutdown_connections 2024-05-09T09:30:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 -- 172.21.15.69:0/2577788327 wait complete. 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.209+0000 7f4a5e796640 1 Processor -- start 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 -- start start 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 0x7f4a5819e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5810c460 con 0x7f4a58102340 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5810c2e0 con 0x7f4a58101980 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5e796640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a5810c5e0 con 0x7f4a58100790 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60042/0 (socket says 172.21.15.69:60042) 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:38154/0 (socket says 172.21.15.69:38154) 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a577fe640 1 -- 172.21.15.69:0/2231988938 learned_addr learned my addr 172.21.15.69:0/2231988938 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 0x7f4a5819e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.210+0000 7f4a577fe640 1 -- 172.21.15.69:0/2231988938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 msgr2=0x7f4a58196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a577fe640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a577fe640 1 -- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 msgr2=0x7f4a5819e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a577fe640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 0x7f4a5819e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a577fe640 1 -- 172.21.15.69:0/2231988938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4a5819e6a0 con 0x7f4a58101980 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a57fff640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:05.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 0x7f4a5819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a577fe640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009900 tx=0x7f4a4c009dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a557fa640 1 -- 172.21.15.69:0/2231988938 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c00d030 con 0x7f4a58101980 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4a58068f40 con 0x7f4a58101980 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a557fa640 1 -- 172.21.15.69:0/2231988938 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a4c00d1d0 con 0x7f4a58101980 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4a580694e0 con 0x7f4a58101980 2024-05-09T09:30:05.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.211+0000 7f4a557fa640 1 -- 172.21.15.69:0/2231988938 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a4c005280 con 0x7f4a58101980 2024-05-09T09:30:05.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 Processor -- start 2024-05-09T09:30:05.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 -- start start 2024-05-09T09:30:05.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b4db50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b50650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5587f2aa8340 con 0x5587f2b4e1c0 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5587f2a50290 con 0x5587f2b49050 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a61576740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5587f2aa4300 con 0x5587f2b48660 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a557fa640 1 -- 172.21.15.69:0/2231988938 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4a4c0054e0 con 0x7f4a58101980 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:38170/0 (socket says 172.21.15.69:38170) 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b4db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1531076867 learned_addr learned my addr 172.21.15.69:0/1531076867 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a577fe640 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b50650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.224 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1531076867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 msgr2=0x5587f2b4db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b4db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1531076867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 msgr2=0x5587f2b50650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b50650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.214+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1531076867 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f2a748e0 con 0x5587f2b48660 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000a870 tx=0x7f4a50009650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd626bbc9b97ab16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a3e7fc640 1 -- 172.21.15.69:0/1531076867 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50018020 con 0x5587f2b48660 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a557fa640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 0x7f4a400a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a3e7fc640 1 -- 172.21.15.69:0/1531076867 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a50014400 con 0x5587f2b48660 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a3e7fc640 1 -- 172.21.15.69:0/1531076867 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a500135c0 con 0x5587f2b48660 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a57fff640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 0x7f4a400a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.215+0000 7f4a557fa640 1 -- 172.21.15.69:0/2231988938 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4a4c0ce540 con 0x7f4a58101980 2024-05-09T09:30:05.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- 172.21.15.69:0/1531076867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 msgr2=0x5587f2b48a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a57fff640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 0x7f4a400a50f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4a48005ab0 tx=0x7f4a48005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000a870 tx=0x7f4a50009650 comp rx=0 tx=0).stop 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- 172.21.15.69:0/1531076867 shutdown_connections 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b50650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b4db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- 172.21.15.69:0/1531076867 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b48a80 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- 172.21.15.69:0/1531076867 >> 172.21.15.69:0/1531076867 conn(0x5587f2b3f8c0 msgr2=0x5587f2b3fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- 172.21.15.69:0/1531076867 shutdown_connections 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- 172.21.15.69:0/1531076867 wait complete. 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 -- start start 2024-05-09T09:30:05.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 Processor -- start 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.216+0000 7f4a61576740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b52e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b59e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5587f2b51080 con 0x5587f2b4e1c0 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5587f2a63790 con 0x5587f2b49050 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5587f2b51200 con 0x5587f2b48660 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 10 client.0 _ll_register_callbacks cb 0x5587f2a9ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b59e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b52e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:38172/0 (socket says 172.21.15.69:38172) 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a57fff640 1 -- 172.21.15.69:0/1075171649 learned_addr learned my addr 172.21.15.69:0/1075171649 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1075171649 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 msgr2=0x5587f2b528c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 msgr2=0x5587f2b52e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b52e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 -- 172.21.15.69:0/1075171649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587f27bc210 con 0x5587f2b4e1c0 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a57fff640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:05.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a577fe640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b52e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b59e40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000a790 tx=0x7f4a5000a400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50a434a8566acb86 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a377fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a50018040 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4a500149c0 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a50014400 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4a5001d420 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 10 client.16953 fetch_fsmap learned FSMap version 39 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.217+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5587f2888620 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a50014400 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f4a5001c540 con 0x5587f2b4e1c0 2024-05-09T09:30:05.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 10 client.16953 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 10 client.16953 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 20 client.16953 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2b5e070 0x5587f2b60460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 4 client.16953 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5587f2ab5a70 con 0x5587f2b5e070 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a57fff640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2b5e070 0x5587f2b60460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a377fe640 10 client.16953 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.218+0000 7f4a57fff640 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2b5e070 0x5587f2b60460 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e95499193ccfcdb3 server_cookie=8e07d64f10f38f4 in_seq=0 out_seq=0 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a377fe640 1 -- 172.21.15.69:0/1075171649 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5587f2ab5a70 con 0x5587f2b5e070 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a377fe640 10 client.16953 handle_command_reply: tid=0 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 client.16953 shutdown 2024-05-09T09:30:05.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 msgr2=0x5587f2b59e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b59e40 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f4a5000a790 tx=0x7f4a5000a400 comp rx=0 tx=0).stop 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 shutdown_connections 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b59e40 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b52e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2b5e070 0x5587f2b60460 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 --2- 172.21.15.69:0/1075171649 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b48660 0x5587f2b528c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a377fe640 0 client.16953 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 >> 172.21.15.69:0/1075171649 conn(0x5587f2b3f8c0 msgr2=0x5587f2b3ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 shutdown_connections 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.219+0000 7f4a61576740 1 -- 172.21.15.69:0/1075171649 wait complete. 2024-05-09T09:30:05.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.220+0000 7f4a61576740 20 client.16953 trim_cache size 0 max 16384 2024-05-09T09:30:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 Processor -- start 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 -- start start 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b59e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b4e1c0 0x5587f2b42840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581aab70 con 0x5587f2b48660 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581a8440 con 0x5587f2b49050 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a61576740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581a6290 con 0x5587f2b4e1c0 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54314/0 (socket says 172.21.15.69:54314) 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 -- 172.21.15.69:0/3750364448 learned_addr learned my addr 172.21.15.69:0/3750364448 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a577fe640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b59e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a5cd33640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b4e1c0 0x5587f2b42840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 -- 172.21.15.69:0/3750364448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b4e1c0 msgr2=0x5587f2b42840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b4e1c0 0x5587f2b42840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 -- 172.21.15.69:0/3750364448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 msgr2=0x5587f2b59e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b59e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 -- 172.21.15.69:0/3750364448 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5587f2b40630 con 0x5587f2b48660 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a577fe640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b59e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.237+0000 7f4a57fff640 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4a48031610 tx=0x7f4a4803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4499915469f06d6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.238+0000 7f4a3dffb640 1 -- 172.21.15.69:0/3750364448 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48035040 con 0x5587f2b48660 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.238+0000 7f4a3dffb640 1 -- 172.21.15.69:0/3750364448 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a480533a0 con 0x5587f2b48660 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.238+0000 7f4a3dffb640 1 -- 172.21.15.69:0/3750364448 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48004c60 con 0x5587f2b48660 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- 172.21.15.69:0/3750364448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 msgr2=0x5587f2b3fca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f4a48031610 tx=0x7f4a4803f000 comp rx=0 tx=0).stop 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- 172.21.15.69:0/3750364448 shutdown_connections 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b4e1c0 0x5587f2b42840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b49050 0x5587f2b59e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- 172.21.15.69:0/3750364448 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b48660 0x5587f2b3fca0 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- 172.21.15.69:0/3750364448 >> 172.21.15.69:0/3750364448 conn(0x5587f2b3f8c0 msgr2=0x5587f2b4d940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- 172.21.15.69:0/3750364448 shutdown_connections 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- 172.21.15.69:0/3750364448 wait complete. 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- start start 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 Processor -- start 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 0x5587f2b574b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b584c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581a6900 con 0x5587f2b4e1c0 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581a7d70 con 0x5587f2b48660 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4a581a6fd0 con 0x5587f2b49050 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.239+0000 7f4a61576740 10 client.0 _ll_register_callbacks cb 0x7f4a48009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a5cd33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b584c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60064/0 (socket says 172.21.15.69:60064) 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a5cd33640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b584c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54328/0 (socket says 172.21.15.69:54328) 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 -- 172.21.15.69:0/4273820563 learned_addr learned my addr 172.21.15.69:0/4273820563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a577fe640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 0x5587f2b574b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 -- 172.21.15.69:0/4273820563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 msgr2=0x5587f2b574b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 0x5587f2b574b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 -- 172.21.15.69:0/4273820563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 msgr2=0x5587f2b584c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b584c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 -- 172.21.15.69:0/4273820563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5587f2b58ab0 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a577fe640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 0x5587f2b574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a57fff640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a48004230 tx=0x7f4a480042e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df213ff6837e380 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a377fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4a48061020 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4a480533a0 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.240+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4a48005240 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=39) ==== 24+0+0 (secure 0 0 0) 0x7f4a4806c050 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 10 client.26270 fetch_fsmap learned FSMap version 39 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=39,monmap=4+}) -- 0x5587f2b59dc0 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4a480533a0 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 39) ==== 8346+0+0 (secure 0 0 0) 0x7f4a480039f0 con 0x5587f2b48660 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 10 client.26270 fetch_fsmap finished waiting for FSMap version 39 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 10 client.26270 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:05.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 20 client.26270 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2bc4a00 0x5587f2bc6e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 4 client.26270 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5587f2aafdc0 con 0x5587f2bc4a00 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a577fe640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2bc4a00 0x5587f2bc6e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a377fe640 10 client.26270 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:05.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.241+0000 7f4a577fe640 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2bc4a00 0x5587f2bc6e20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fadea30c71d795f server_cookie=94ce1800f9ddd778 in_seq=0 out_seq=0 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.390+0000 7f4a377fe640 1 -- 172.21.15.69:0/4273820563 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5587f2aafdc0 con 0x5587f2bc4a00 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.390+0000 7f4a377fe640 10 client.26270 handle_command_reply: tid=0 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.390+0000 7f4a61576740 1 client.26270 shutdown 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.390+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 msgr2=0x5587f2b56f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.390+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4a48004230 tx=0x7f4a480042e0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 shutdown_connections 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5587f2b4e1c0 0x5587f2b584c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5587f2bc4a00 0x5587f2bc6e20 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a377fe640 0 client.26270 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5587f2b49050 0x5587f2b574b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 --2- 172.21.15.69:0/4273820563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5587f2b48660 0x5587f2b56f70 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 >> 172.21.15.69:0/4273820563 conn(0x5587f2b3f8c0 msgr2=0x5587f2b4d7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 shutdown_connections 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 1 -- 172.21.15.69:0/4273820563 wait complete. 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a61576740 20 client.26270 trim_cache size 0 max 16384 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:30:05.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:30:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:30:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 msgr2=0x7f4a400a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 0x7f4a400a50f0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f4a48005ab0 tx=0x7f4a48005fd0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 msgr2=0x7f4a58196fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f4a4c009900 tx=0x7f4a4c009dc0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 shutdown_connections 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4a400a2c30 0x7f4a400a50f0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4a58102340 0x7f4a5819e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4a58101980 0x7f4a58196fc0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.391+0000 7f4a5e796640 1 --2- 172.21.15.69:0/2231988938 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4a58100790 0x7f4a58196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.392+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 >> 172.21.15.69:0/2231988938 conn(0x7f4a580fbd00 msgr2=0x7f4a58104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.392+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 shutdown_connections 2024-05-09T09:30:05.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:05.392+0000 7f4a5e796640 1 -- 172.21.15.69:0/2231988938 wait complete. 2024-05-09T09:30:05.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 834538 8.55 MB/sec execute 483 sec latency 112.832 ms 2024-05-09T09:30:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:05 smithi069 ceph-mon[30123]: pgmap v706: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.9 MiB/s wr, 339 op/s 2024-05-09T09:30:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:05 smithi163 ceph-mon[34354]: pgmap v706: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.9 MiB/s wr, 339 op/s 2024-05-09T09:30:05.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:05 smithi103 ceph-mon[34421]: pgmap v706: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 900 KiB/s rd, 2.9 MiB/s wr, 339 op/s 2024-05-09T09:30:06.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 835728 8.55 MB/sec execute 484 sec latency 77.073 ms 2024-05-09T09:30:07.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 838206 8.56 MB/sec execute 485 sec latency 29.651 ms 2024-05-09T09:30:08.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:07 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:08.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:07 smithi103 ceph-mon[34421]: pgmap v707: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-05-09T09:30:08.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:07 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:08.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:07 smithi069 ceph-mon[30123]: pgmap v707: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-05-09T09:30:08.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:07 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:08.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:07 smithi163 ceph-mon[34354]: pgmap v707: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 883 KiB/s rd, 3.2 MiB/s wr, 355 op/s 2024-05-09T09:30:08.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 839864 8.55 MB/sec execute 486 sec latency 26.532 ms 2024-05-09T09:30:09.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 841949 8.56 MB/sec execute 487 sec latency 9.669 ms 2024-05-09T09:30:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:09 smithi069 ceph-mon[30123]: pgmap v708: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-05-09T09:30:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:09 smithi163 ceph-mon[34354]: pgmap v708: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-05-09T09:30:09.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:09 smithi103 ceph-mon[34421]: pgmap v708: 129 pgs: 129 active+clean; 153 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 726 KiB/s rd, 2.7 MiB/s wr, 310 op/s 2024-05-09T09:30:10.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 844043 8.56 MB/sec execute 488 sec latency 66.779 ms 2024-05-09T09:30:10.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:10 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:30:10.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:10 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:30:11.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:10 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:30:11.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 845489 8.57 MB/sec execute 489 sec latency 48.154 ms 2024-05-09T09:30:12.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 847782 8.57 MB/sec execute 490 sec latency 14.528 ms 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: pgmap v709: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:30:12.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:30:12.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: pgmap v709: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-05-09T09:30:12.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:30:12.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:30:12.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: pgmap v709: 129 pgs: 129 active+clean; 156 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 847 KiB/s rd, 2.7 MiB/s wr, 355 op/s 2024-05-09T09:30:12.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:30:12.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:30:12.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:30:13.456 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:13 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:13.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 849286 8.57 MB/sec execute 491 sec latency 12.519 ms 2024-05-09T09:30:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:13 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:13.518 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:13 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:14.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 851736 8.58 MB/sec execute 492 sec latency 18.028 ms 2024-05-09T09:30:14.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:14 smithi069 ceph-mon[30123]: pgmap v710: 129 pgs: 129 active+clean; 155 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 2.8 MiB/s wr, 354 op/s 2024-05-09T09:30:14.535 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:14 smithi103 ceph-mon[34421]: pgmap v710: 129 pgs: 129 active+clean; 155 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 2.8 MiB/s wr, 354 op/s 2024-05-09T09:30:14.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:14 smithi163 ceph-mon[34354]: pgmap v710: 129 pgs: 129 active+clean; 155 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 859 KiB/s rd, 2.8 MiB/s wr, 354 op/s 2024-05-09T09:30:15.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 852689 8.58 MB/sec execute 493 sec latency 60.635 ms 2024-05-09T09:30:16.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 854693 8.57 MB/sec execute 494 sec latency 21.322 ms 2024-05-09T09:30:16.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:16 smithi163 ceph-mon[34354]: pgmap v711: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 2.7 MiB/s wr, 335 op/s 2024-05-09T09:30:16.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:16 smithi069 ceph-mon[30123]: pgmap v711: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 2.7 MiB/s wr, 335 op/s 2024-05-09T09:30:16.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:16 smithi103 ceph-mon[34421]: pgmap v711: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 2.7 MiB/s wr, 335 op/s 2024-05-09T09:30:17.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 856321 8.58 MB/sec execute 495 sec latency 13.120 ms 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/867184435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 msgr2=0x7f9cf4109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/867184435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf4109a00 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f9cec009ca0 tx=0x7f9cec02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/867184435 shutdown_connections 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/867184435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf4109a00 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/867184435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf4104ce0 0x7f9cf41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/867184435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf41023b0 0x7f9cf41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/867184435 >> 172.21.15.69:0/867184435 conn(0x7f9cf40fbd00 msgr2=0x7f9cf40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/867184435 shutdown_connections 2024-05-09T09:30:18.488 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.488+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/867184435 wait complete. 2024-05-09T09:30:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 Processor -- start 2024-05-09T09:30:18.489 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 -- start start 2024-05-09T09:30:18.490 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf41023b0 0x7f9cf4197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf419f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf410d530 con 0x7f9cf4107610 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf410d3b0 con 0x7f9cf4104ce0 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cfb9c1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf410d6b0 con 0x7f9cf41023b0 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cf8f5c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf419f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cf8f5c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:34600/0 (socket says 172.21.15.69:34600) 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/1835324638 learned_addr learned my addr 172.21.15.69:0/1835324638 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.489+0000 7f9cf975d640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf41023b0 0x7f9cf4197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/1835324638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf41023b0 msgr2=0x7f9cf4197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf41023b0 0x7f9cf4197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 msgr2=0x7f9cf419f0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf419f0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/1835324638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f9cf419f710 con 0x7f9cf4104ce0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400e9f0 tx=0x7f9ce400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cea7fc640 1 -- 172.21.15.69:0/1835324638 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce400ccc0 con 0x7f9cf4104ce0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cea7fc640 1 -- 172.21.15.69:0/1835324638 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce4005500 con 0x7f9cf4104ce0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f9cf419fa00 con 0x7f9cf4104ce0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cea7fc640 1 -- 172.21.15.69:0/1835324638 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4010690 con 0x7f9cf4104ce0 2024-05-09T09:30:18.492 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.490+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f9cf419ff50 con 0x7f9cf4104ce0 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.492+0000 7f9cea7fc640 1 -- 172.21.15.69:0/1835324638 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9ce40046c0 con 0x7f9cf4104ce0 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 Processor -- start 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 -- start start 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc3b620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc406f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc431f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562a6db9f340 con 0x562a6dc3b200 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a6db47290 con 0x562a6dc3bbf0 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562a6db9b300 con 0x562a6dc40d60 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.493+0000 7f9cea7fc640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 0x7f9ce00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc3b620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc406f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc3b620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42134/0 (socket says 172.21.15.69:42134) 2024-05-09T09:30:18.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf9f5e640 1 -- 172.21.15.69:0/4123726471 learned_addr learned my addr 172.21.15.69:0/4123726471 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc431f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 0x7f9ce00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cea7fc640 1 -- 172.21.15.69:0/1835324638 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f9ce40cf6e0 con 0x7f9cf4104ce0 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/4123726471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 msgr2=0x562a6dc406f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc406f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/4123726471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 msgr2=0x562a6dc3b620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc3b620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 -- 172.21.15.69:0/4123726471 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6db6b8e0 con 0x562a6dc40d60 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf975d640 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:18.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc431f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ce409b860 tx=0x7f9ce4009740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=97656951a837ce8b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 0x7f9ce00a5170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x562a6dc3c450 tx=0x7f9cec005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cdf7fe640 1 -- 172.21.15.69:0/4123726471 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce409b6a0 con 0x562a6dc40d60 2024-05-09T09:30:18.505 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cdf7fe640 1 -- 172.21.15.69:0/4123726471 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce409c260 con 0x562a6dc40d60 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.494+0000 7f9cdf7fe640 1 -- 172.21.15.69:0/4123726471 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9ce40a10a0 con 0x562a6dc40d60 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/4123726471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 msgr2=0x562a6dc431f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc431f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f9ce409b860 tx=0x7f9ce4009740 comp rx=0 tx=0).stop 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/4123726471 shutdown_connections 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc431f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc406f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/4123726471 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc3b620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/4123726471 >> 172.21.15.69:0/4123726471 conn(0x562a6dc36870 msgr2=0x562a6dc36c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/4123726471 shutdown_connections 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/4123726471 wait complete. 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 -- start start 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.495+0000 7f9cfe7ae740 1 Processor -- start 2024-05-09T09:30:18.506 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc459d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc4ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562a6dc43d10 con 0x562a6dc3b200 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562a6db5a790 con 0x562a6dc3bbf0 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562a6dc43e90 con 0x562a6dc40d60 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cfe7ae740 10 client.0 _ll_register_callbacks cb 0x562a6db95a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42138/0 (socket says 172.21.15.69:42138) 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 -- 172.21.15.69:0/3692168139 learned_addr learned my addr 172.21.15.69:0/3692168139 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc459d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc4ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 -- 172.21.15.69:0/3692168139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 msgr2=0x562a6dc4ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc4ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 -- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 msgr2=0x562a6dc459d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc459d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 -- 172.21.15.69:0/3692168139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6d8b3210 con 0x562a6dc3b200 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc4ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00126e0 tx=0x7f9cf0012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31ea0c01b1ccf36e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.496+0000 7f9cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0012190 con 0x562a6dc3b200 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f9cf000f8c0 con 0x562a6dc3b200 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9ce409c260 con 0x562a6dc3b200 2024-05-09T09:30:18.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9cf000fac0 con 0x562a6dc3b200 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 10 client.16971 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562a6d97f620 con 0x562a6dc3b200 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9ce409c260 con 0x562a6dc3b200 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7f9cf0010ca0 con 0x562a6dc3b200 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 10 client.16971 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 10 client.16971 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 20 client.16971 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dc51030 0x562a6dc534d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 4 client.16971 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562a6dbaca70 con 0x562a6dc51030 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.497+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dc51030 0x562a6dc534d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.498+0000 7f9cdcff9640 10 client.16971 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.498+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dc51030 0x562a6dc534d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5ee4ba6679a8760 server_cookie=7df732c368708077 in_seq=0 out_seq=0 2024-05-09T09:30:18.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3692168139 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562a6dbaca70 con 0x562a6dc51030 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cdcff9640 10 client.16971 handle_command_reply: tid=0 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 client.16971 shutdown 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 msgr2=0x562a6dc45490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f9cf00126e0 tx=0x7f9cf0012ba0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 shutdown_connections 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dc51030 0x562a6dc534d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dc4ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc459d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3692168139 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc45490 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 >> 172.21.15.69:0/3692168139 conn(0x562a6dc36870 msgr2=0x562a6dc359b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.510 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cdcff9640 0 client.16971 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 shutdown_connections 2024-05-09T09:30:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.499+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3692168139 wait complete. 2024-05-09T09:30:18.511 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.500+0000 7f9cfe7ae740 20 client.16971 trim_cache size 0 max 16384 2024-05-09T09:30:18.511 INFO:tasks.workunit.client.0.smithi069.stdout: 1 858651 8.59 MB/sec execute 496 sec latency 17.733 ms 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 Processor -- start 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 -- start start 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc3bbf0 0x562a6dc4c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.522 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41accf0 con 0x562a6dc40d60 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41aa560 con 0x562a6dc3b200 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.516+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41a83b0 con 0x562a6dc3bbf0 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:34638/0 (socket says 172.21.15.69:34638) 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42140/0 (socket says 172.21.15.69:42140) 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 -- 172.21.15.69:0/2100440809 learned_addr learned my addr 172.21.15.69:0/2100440809 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc3bbf0 0x562a6dc4c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 -- 172.21.15.69:0/2100440809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc3bbf0 msgr2=0x562a6dc4c9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc3bbf0 0x562a6dc4c9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.523 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 -- 172.21.15.69:0/2100440809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 msgr2=0x562a6dc39b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 -- 172.21.15.69:0/2100440809 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562a6dc3bfd0 con 0x562a6dc3b200 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf9f5e640 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cf975d640 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009e00 tx=0x7f9cf0009640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f55733708a8b3dd0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cdeffd640 1 -- 172.21.15.69:0/2100440809 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf000f790 con 0x562a6dc3b200 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cdeffd640 1 -- 172.21.15.69:0/2100440809 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cf00055c0 con 0x562a6dc3b200 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.517+0000 7f9cdeffd640 1 -- 172.21.15.69:0/2100440809 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0003850 con 0x562a6dc3b200 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/2100440809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 msgr2=0x562a6dc395f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009e00 tx=0x7f9cf0009640 comp rx=0 tx=0).stop 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/2100440809 shutdown_connections 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc40d60 0x562a6dc39b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc3bbf0 0x562a6dc4c9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/2100440809 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3b200 0x562a6dc395f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/2100440809 >> 172.21.15.69:0/2100440809 conn(0x562a6dc36870 msgr2=0x562a6dc36c50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/2100440809 shutdown_connections 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/2100440809 wait complete. 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.518+0000 7f9cfe7ae740 1 -- start start 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 Processor -- start 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc49ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dcb5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41a8a20 con 0x562a6dc3b200 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41a9e90 con 0x562a6dc3bbf0 2024-05-09T09:30:18.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9cf41a90f0 con 0x562a6dc40d60 2024-05-09T09:30:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cfe7ae740 10 client.0 _ll_register_callbacks cb 0x7f9cf0009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf9f5e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dcb5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42154/0 (socket says 172.21.15.69:42154) 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 -- 172.21.15.69:0/3350955119 learned_addr learned my addr 172.21.15.69:0/3350955119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc49ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 -- 172.21.15.69:0/3350955119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 msgr2=0x562a6dcb5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dcb5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 -- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 msgr2=0x562a6dc49ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc49ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf975d640 1 -- 172.21.15.69:0/3350955119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562a6dcb5c30 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.519+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc49ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cf975d640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fd10 tx=0x7f9cf001ca60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d06fcccaf62079f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0005320 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f9cf0018040 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9cf00055c0 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f9cf00035f0 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 10 client.16977 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x562a6dcb6f40 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9cf00055c0 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7f9cf0010550 con 0x562a6dc3b200 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 10 client.16977 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:18.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 10 client.16977 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.520+0000 7f9cfe7ae740 20 client.16977 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dcb7940 0x562a6dcb9d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cfe7ae740 4 client.16977 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562a6dba6dc0 con 0x562a6dcb7940 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dcb7940 0x562a6dcb9d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cdcff9640 10 client.16977 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:18.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.521+0000 7f9cf8f5c640 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dcb7940 0x562a6dcb9d60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50eae42396a5765c server_cookie=8e87b9a7a73ebea9 in_seq=0 out_seq=0 2024-05-09T09:30:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.669+0000 7f9cdcff9640 1 -- 172.21.15.69:0/3350955119 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562a6dba6dc0 con 0x562a6dcb7940 2024-05-09T09:30:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.669+0000 7f9cdcff9640 10 client.16977 handle_command_reply: tid=0 2024-05-09T09:30:18.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.669+0000 7f9cfe7ae740 1 client.16977 shutdown 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.669+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 msgr2=0x562a6dc499a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.669+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7f9cf000fd10 tx=0x7f9cf001ca60 comp rx=0 tx=0).stop 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 shutdown_connections 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562a6dcb7940 0x562a6dcb9d60 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cdcff9640 0 client.16977 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562a6dc40d60 0x562a6dcb5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562a6dc3bbf0 0x562a6dc49ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 --2- 172.21.15.69:0/3350955119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562a6dc3b200 0x562a6dc499a0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 >> 172.21.15.69:0/3350955119 conn(0x562a6dc36870 msgr2=0x562a6dc395f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 shutdown_connections 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 1 -- 172.21.15.69:0/3350955119 wait complete. 2024-05-09T09:30:18.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.670+0000 7f9cfe7ae740 20 client.16977 trim_cache size 0 max 16384 2024-05-09T09:30:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:30:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:30:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:30:18.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 msgr2=0x7f9ce00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 0x7f9ce00a5170 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x562a6dc3c450 tx=0x7f9cec005d80 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 msgr2=0x7f9cf4198030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9ce400e9f0 tx=0x7f9ce400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 shutdown_connections 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f9ce00a2cb0 0x7f9ce00a5170 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9cf4107610 0x7f9cf419f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9cf4104ce0 0x7f9cf4198030 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 --2- 172.21.15.69:0/1835324638 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9cf41023b0 0x7f9cf4197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 >> 172.21.15.69:0/1835324638 conn(0x7f9cf40fbd00 msgr2=0x7f9cf4102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 shutdown_connections 2024-05-09T09:30:18.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:18.671+0000 7f9cfb9c1640 1 -- 172.21.15.69:0/1835324638 wait complete. 2024-05-09T09:30:18.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:18 smithi069 ceph-mon[30123]: pgmap v712: 129 pgs: 129 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.8 MiB/s wr, 339 op/s 2024-05-09T09:30:18.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:18 smithi163 ceph-mon[34354]: pgmap v712: 129 pgs: 129 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.8 MiB/s wr, 339 op/s 2024-05-09T09:30:18.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:18 smithi103 ceph-mon[34421]: pgmap v712: 129 pgs: 129 active+clean; 158 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.8 MiB/s wr, 339 op/s 2024-05-09T09:30:19.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 859867 8.58 MB/sec execute 497 sec latency 93.513 ms 2024-05-09T09:30:20.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 861693 8.58 MB/sec execute 498 sec latency 38.662 ms 2024-05-09T09:30:20.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:20 smithi069 ceph-mon[30123]: pgmap v713: 129 pgs: 129 active+clean; 160 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-09T09:30:20.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:20 smithi163 ceph-mon[34354]: pgmap v713: 129 pgs: 129 active+clean; 160 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-09T09:30:20.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:20 smithi103 ceph-mon[34421]: pgmap v713: 129 pgs: 129 active+clean; 160 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 801 KiB/s rd, 2.6 MiB/s wr, 320 op/s 2024-05-09T09:30:21.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 863271 8.59 MB/sec execute 499 sec latency 23.649 ms 2024-05-09T09:30:22.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 865505 8.59 MB/sec execute 500 sec latency 15.782 ms 2024-05-09T09:30:22.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:22 smithi069 ceph-mon[30123]: pgmap v714: 129 pgs: 129 active+clean; 157 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 308 op/s 2024-05-09T09:30:22.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:22 smithi163 ceph-mon[34354]: pgmap v714: 129 pgs: 129 active+clean; 157 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 308 op/s 2024-05-09T09:30:22.832 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:22 smithi103 ceph-mon[34421]: pgmap v714: 129 pgs: 129 active+clean; 157 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.4 MiB/s wr, 308 op/s 2024-05-09T09:30:23.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 867014 8.59 MB/sec execute 501 sec latency 11.991 ms 2024-05-09T09:30:24.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 869564 8.60 MB/sec execute 502 sec latency 17.021 ms 2024-05-09T09:30:24.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:24 smithi069 ceph-mon[30123]: pgmap v715: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2024-05-09T09:30:24.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:24 smithi163 ceph-mon[34354]: pgmap v715: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2024-05-09T09:30:24.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:24 smithi103 ceph-mon[34421]: pgmap v715: 129 pgs: 129 active+clean; 157 MiB data, 3.0 GiB used, 1.0 TiB / 1.0 TiB avail; 660 KiB/s rd, 2.7 MiB/s wr, 293 op/s 2024-05-09T09:30:25.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 870557 8.60 MB/sec execute 503 sec latency 74.025 ms 2024-05-09T09:30:26.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 872614 8.60 MB/sec execute 504 sec latency 66.131 ms 2024-05-09T09:30:26.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:26 smithi069 ceph-mon[30123]: pgmap v716: 129 pgs: 129 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:30:26.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:26 smithi163 ceph-mon[34354]: pgmap v716: 129 pgs: 129 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:30:26.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:26 smithi103 ceph-mon[34421]: pgmap v716: 129 pgs: 129 active+clean; 159 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 737 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:30:27.504 INFO:tasks.workunit.client.0.smithi069.stdout: 1 874126 8.60 MB/sec execute 505 sec latency 15.123 ms 2024-05-09T09:30:28.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 876760 8.61 MB/sec execute 506 sec latency 14.650 ms 2024-05-09T09:30:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:28 smithi069 ceph-mon[30123]: pgmap v717: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:30:28.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:28 smithi163 ceph-mon[34354]: pgmap v717: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:30:28.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:28 smithi103 ceph-mon[34421]: pgmap v717: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 798 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:30:29.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 877996 8.61 MB/sec execute 507 sec latency 13.024 ms 2024-05-09T09:30:30.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 880479 8.62 MB/sec execute 508 sec latency 46.206 ms 2024-05-09T09:30:30.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:30 smithi069 ceph-mon[30123]: pgmap v718: 129 pgs: 129 active+clean; 162 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.7 MiB/s wr, 330 op/s 2024-05-09T09:30:30.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:30 smithi163 ceph-mon[34354]: pgmap v718: 129 pgs: 129 active+clean; 162 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.7 MiB/s wr, 330 op/s 2024-05-09T09:30:30.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:30 smithi103 ceph-mon[34421]: pgmap v718: 129 pgs: 129 active+clean; 162 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 777 KiB/s rd, 2.7 MiB/s wr, 330 op/s 2024-05-09T09:30:31.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 881454 8.61 MB/sec execute 509 sec latency 37.700 ms 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 -- 172.21.15.69:0/415717027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c102ed0 msgr2=0x7f8a5c1032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 --2- 172.21.15.69:0/415717027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c1032f0 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c009900 tx=0x7f8a4c02a410 comp rx=0 tx=0).stop 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 -- 172.21.15.69:0/415717027 shutdown_connections 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 --2- 172.21.15.69:0/415717027 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c100300 0x7f8a5c109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 --2- 172.21.15.69:0/415717027 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c0ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 --2- 172.21.15.69:0/415717027 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c1032f0 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 -- 172.21.15.69:0/415717027 >> 172.21.15.69:0/415717027 conn(0x7f8a5c0fb4a0 msgr2=0x7f8a5c0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 -- 172.21.15.69:0/415717027 shutdown_connections 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.780+0000 7f8a622ee640 1 -- 172.21.15.69:0/415717027 wait complete. 2024-05-09T09:30:31.781 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 Processor -- start 2024-05-09T09:30:31.782 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 -- start start 2024-05-09T09:30:31.783 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c100300 0x7f8a5c197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c19f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c10d500 con 0x7f8a5c0ff940 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c10d380 con 0x7f8a5c102ed0 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a622ee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c10d680 con 0x7f8a5c100300 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55110/0 (socket says 172.21.15.69:55110) 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3665270266 learned_addr learned my addr 172.21.15.69:0/3665270266 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c100300 0x7f8a5c197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a6088b640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c19f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3665270266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c100300 msgr2=0x7f8a5c197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c100300 0x7f8a5c197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.784 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3665270266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c102ed0 msgr2=0x7f8a5c19f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c19f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.781+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3665270266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8a5c19f6a0 con 0x7f8a5c0ff940 2024-05-09T09:30:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c031610 tx=0x7f8a4c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.785 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a597fa640 1 -- 172.21.15.69:0/3665270266 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c03f030 con 0x7f8a5c0ff940 2024-05-09T09:30:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a597fa640 1 -- 172.21.15.69:0/3665270266 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c003ad0 con 0x7f8a5c0ff940 2024-05-09T09:30:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8a5c19f930 con 0x7f8a5c0ff940 2024-05-09T09:30:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a597fa640 1 -- 172.21.15.69:0/3665270266 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c035070 con 0x7f8a5c0ff940 2024-05-09T09:30:31.786 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.782+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8a5c19fe20 con 0x7f8a5c0ff940 2024-05-09T09:30:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.784+0000 7f8a597fa640 1 -- 172.21.15.69:0/3665270266 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8a4c042030 con 0x7f8a5c0ff940 2024-05-09T09:30:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.784+0000 7f8a597fa640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 0x7f8a440a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 Processor -- start 2024-05-09T09:30:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a597fa640 1 -- 172.21.15.69:0/3665270266 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a4c0efa90 con 0x7f8a5c0ff940 2024-05-09T09:30:31.793 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 0x7f8a440a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 -- start start 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7d5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a731340 con 0x55fc2a7cd250 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a6d9290 con 0x55fc2a7d2db0 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a72d300 con 0x55fc2a7cdc40 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55112/0 (socket says 172.21.15.69:55112) 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 -- 172.21.15.69:0/1447816426 learned_addr learned my addr 172.21.15.69:0/1447816426 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a6088b640 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7d5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.794 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 0x7f8a440a50f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c198f90 tx=0x7f8a5000a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 -- 172.21.15.69:0/1447816426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 msgr2=0x55fc2a7d2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 -- 172.21.15.69:0/1447816426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 msgr2=0x55fc2a7d5240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7d5240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 -- 172.21.15.69:0/1447816426 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc2a6fd8e0 con 0x55fc2a7cd250 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.785+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0bbf00 tx=0x7f8a4c003f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e2c80105e659d6d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a427fc640 1 -- 172.21.15.69:0/1447816426 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c049040 con 0x55fc2a7cd250 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a427fc640 1 -- 172.21.15.69:0/1447816426 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c0bbf70 con 0x55fc2a7cd250 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a427fc640 1 -- 172.21.15.69:0/1447816426 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0b85e0 con 0x55fc2a7cd250 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1447816426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 msgr2=0x55fc2a7cd670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0bbf00 tx=0x7f8a4c003f60 comp rx=0 tx=0).stop 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1447816426 shutdown_connections 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7d5240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1447816426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cd670 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1447816426 >> 172.21.15.69:0/1447816426 conn(0x55fc2a7c88c0 msgr2=0x55fc2a7c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1447816426 shutdown_connections 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.786+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1447816426 wait complete. 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 -- start start 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 Processor -- start 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7de630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a731340 con 0x55fc2a7cd250 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a6ec790 con 0x55fc2a7d2db0 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fc2a7d5b20 con 0x55fc2a7cdc40 2024-05-09T09:30:31.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55120/0 (socket says 172.21.15.69:55120) 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3201765542 learned_addr learned my addr 172.21.15.69:0/3201765542 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a6088b640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7de630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.787+0000 7f8a64ff7740 10 client.0 _ll_register_callbacks cb 0x55fc2a727a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3201765542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 msgr2=0x55fc2a7d75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 msgr2=0x55fc2a7de630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7de630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 -- 172.21.15.69:0/3201765542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc2a445210 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0b8940 tx=0x7f8a4c0ba480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b231052397cf6abb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a4c0d6070 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a4c0b85e0 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a4c0bbf70 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8a4c049070 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a64ff7740 10 client.16989 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fc2a511620 con 0x55fc2a7cd250 2024-05-09T09:30:31.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a4c0bbf70 con 0x55fc2a7cd250 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7f8a4c0c4830 con 0x55fc2a7cd250 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.788+0000 7f8a64ff7740 10 client.16989 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a64ff7740 10 client.16989 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a64ff7740 20 client.16989 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a7e2880 0x55fc2a7e4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a64ff7740 4 client.16989 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fc2a73ea70 con 0x55fc2a7e2880 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a7e2880 0x55fc2a7e4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a3b7fe640 10 client.16989 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.789+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a7e2880 0x55fc2a7e4cf0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c573e9b7e41dd091 server_cookie=86ba97b2ed92c87c in_seq=0 out_seq=0 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/3201765542 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fc2a73ea70 con 0x55fc2a7e2880 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a3b7fe640 10 client.16989 handle_command_reply: tid=0 2024-05-09T09:30:31.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 client.16989 shutdown 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 msgr2=0x55fc2a7d70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c0b8940 tx=0x7f8a4c0ba480 comp rx=0 tx=0).stop 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 shutdown_connections 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7de630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a7e2880 0x55fc2a7e4cf0 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7d75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a3b7fe640 0 client.16989 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/3201765542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7cd250 0x55fc2a7d70b0 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 >> 172.21.15.69:0/3201765542 conn(0x55fc2a7c88c0 msgr2=0x55fc2a7c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 shutdown_connections 2024-05-09T09:30:31.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 1 -- 172.21.15.69:0/3201765542 wait complete. 2024-05-09T09:30:31.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.790+0000 7f8a64ff7740 20 client.16989 trim_cache size 0 max 16384 2024-05-09T09:30:31.809 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.807+0000 7f8a64ff7740 1 Processor -- start 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.807+0000 7f8a64ff7740 1 -- start start 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cbf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7de5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1acc60 con 0x55fc2a7d2db0 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1aa4d0 con 0x55fc2a7cdc40 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1a8320 con 0x55fc2a7cd250 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a5bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cbf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55128/0 (socket says 172.21.15.69:55128) 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 -- 172.21.15.69:0/858589673 learned_addr learned my addr 172.21.15.69:0/858589673 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7de5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.810 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 -- 172.21.15.69:0/858589673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 msgr2=0x55fc2a7cbf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cbf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 -- 172.21.15.69:0/858589673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 msgr2=0x55fc2a7de5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7de5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 -- 172.21.15.69:0/858589673 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fc2a511620 con 0x55fc2a7d2db0 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a6088b640 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8a54003cf0 tx=0x7f8a54004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc5cc1e66a0cd520 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a41ffb640 1 -- 172.21.15.69:0/858589673 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54010430 con 0x55fc2a7d2db0 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a41ffb640 1 -- 172.21.15.69:0/858589673 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a54010ab0 con 0x55fc2a7d2db0 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.808+0000 7f8a41ffb640 1 -- 172.21.15.69:0/858589673 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a5401bb50 con 0x55fc2a7d2db0 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- 172.21.15.69:0/858589673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 msgr2=0x55fc2a7ea600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7f8a54003cf0 tx=0x7f8a54004410 comp rx=0 tx=0).stop 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- 172.21.15.69:0/858589673 shutdown_connections 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7ea600 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7de5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.811 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/858589673 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7cbf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- 172.21.15.69:0/858589673 >> 172.21.15.69:0/858589673 conn(0x55fc2a7c88c0 msgr2=0x55fc2a7cb870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- 172.21.15.69:0/858589673 shutdown_connections 2024-05-09T09:30:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- 172.21.15.69:0/858589673 wait complete. 2024-05-09T09:30:31.812 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.809+0000 7f8a64ff7740 1 -- start start 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 Processor -- start 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7dc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7dc670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1a8990 con 0x55fc2a7d2db0 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1a9e00 con 0x55fc2a7cdc40 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8a5c1a9060 con 0x55fc2a7cd250 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a64ff7740 10 client.0 _ll_register_callbacks cb 0x7f8a54009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:31.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55142/0 (socket says 172.21.15.69:55142) 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 -- 172.21.15.69:0/1503335889 learned_addr learned my addr 172.21.15.69:0/1503335889 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a5b7fe640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7dc670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7dc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 -- 172.21.15.69:0/1503335889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 msgr2=0x55fc2a7dc130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7dc130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 -- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 msgr2=0x55fc2a7dc670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7dc670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 -- 172.21.15.69:0/1503335889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fc2a7ddc00 con 0x55fc2a7d2db0 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a6088b640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009a50 tx=0x7f8a54009a80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3864467c1d0e90fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:31.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.810+0000 7f8a3b7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8a54010640 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f8a540107e0 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8a54010ab0 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7f8a54035050 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 10 client.16995 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55fc2a849250 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8a54010ab0 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7f8a540257e0 con 0x55fc2a7d2db0 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 10 client.16995 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 10 client.16995 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 20 client.16995 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a849c60 0x55fc2a84c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 4 client.16995 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:30:31.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fc2a738dc0 con 0x55fc2a849c60 2024-05-09T09:30:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.811+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a849c60 0x55fc2a84c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.812+0000 7f8a3b7fe640 10 client.16995 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:31.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.812+0000 7f8a5bfff640 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a849c60 0x55fc2a84c080 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed65bc412ec1736f server_cookie=db5bfe297baa25fe in_seq=0 out_seq=0 2024-05-09T09:30:31.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.932+0000 7f8a3b7fe640 1 -- 172.21.15.69:0/1503335889 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fc2a738dc0 con 0x55fc2a849c60 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.932+0000 7f8a3b7fe640 10 client.16995 handle_command_reply: tid=0 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.932+0000 7f8a64ff7740 1 client.16995 shutdown 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 msgr2=0x55fc2a7dd610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8a54009a50 tx=0x7f8a54009a80 comp rx=0 tx=0).stop 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 shutdown_connections 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fc2a7d2db0 0x55fc2a7dd610 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fc2a7cdc40 0x55fc2a7dc670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fc2a849c60 0x55fc2a84c080 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 --2- 172.21.15.69:0/1503335889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fc2a7cd250 0x55fc2a7dc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 >> 172.21.15.69:0/1503335889 conn(0x55fc2a7c88c0 msgr2=0x55fc2a7d6cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a3b7fe640 0 client.16995 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 shutdown_connections 2024-05-09T09:30:31.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 1 -- 172.21.15.69:0/1503335889 wait complete. 2024-05-09T09:30:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.933+0000 7f8a64ff7740 20 client.16995 trim_cache size 0 max 16384 2024-05-09T09:30:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:30:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:30:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:30:31.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 msgr2=0x7f8a440a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 0x7f8a440a50f0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f8a5c198f90 tx=0x7f8a5000a2d0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 msgr2=0x7f8a5c197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7f8a4c031610 tx=0x7f8a4c004270 comp rx=0 tx=0).stop 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 shutdown_connections 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8a440a2c30 0x7f8a440a50f0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8a5c102ed0 0x7f8a5c19f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8a5c100300 0x7f8a5c197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 --2- 172.21.15.69:0/3665270266 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8a5c0ff940 0x7f8a5c197a80 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 >> 172.21.15.69:0/3665270266 conn(0x7f8a5c0fb4a0 msgr2=0x7f8a5c105e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 shutdown_connections 2024-05-09T09:30:31.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:31.934+0000 7f8a622ee640 1 -- 172.21.15.69:0/3665270266 wait complete. 2024-05-09T09:30:32.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 883971 8.62 MB/sec execute 510 sec latency 116.630 ms 2024-05-09T09:30:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:32 smithi069 ceph-mon[30123]: pgmap v719: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 276 op/s 2024-05-09T09:30:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:32 smithi163 ceph-mon[34354]: pgmap v719: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 276 op/s 2024-05-09T09:30:33.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:32 smithi103 ceph-mon[34421]: pgmap v719: 129 pgs: 129 active+clean; 161 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 592 KiB/s rd, 2.4 MiB/s wr, 276 op/s 2024-05-09T09:30:33.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 885089 8.61 MB/sec execute 511 sec latency 12.220 ms 2024-05-09T09:30:34.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 887709 8.63 MB/sec execute 512 sec latency 14.203 ms 2024-05-09T09:30:35.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:34 smithi069 ceph-mon[30123]: pgmap v720: 129 pgs: 129 active+clean; 164 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:30:35.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:34 smithi163 ceph-mon[34354]: pgmap v720: 129 pgs: 129 active+clean; 164 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:30:35.120 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:34 smithi103 ceph-mon[34421]: pgmap v720: 129 pgs: 129 active+clean; 164 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 835 KiB/s rd, 3.0 MiB/s wr, 344 op/s 2024-05-09T09:30:35.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 888745 8.62 MB/sec execute 513 sec latency 32.108 ms 2024-05-09T09:30:36.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 891115 8.63 MB/sec execute 514 sec latency 33.516 ms 2024-05-09T09:30:37.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:36 smithi069 ceph-mon[30123]: pgmap v721: 129 pgs: 129 active+clean; 160 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:30:37.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:36 smithi163 ceph-mon[34354]: pgmap v721: 129 pgs: 129 active+clean; 160 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:30:37.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:36 smithi103 ceph-mon[34421]: pgmap v721: 129 pgs: 129 active+clean; 160 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 719 KiB/s rd, 2.7 MiB/s wr, 311 op/s 2024-05-09T09:30:37.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 892494 8.63 MB/sec execute 515 sec latency 15.095 ms 2024-05-09T09:30:38.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 895017 8.64 MB/sec execute 516 sec latency 13.440 ms 2024-05-09T09:30:39.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:38 smithi069 ceph-mon[30123]: pgmap v722: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:30:39.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:38 smithi163 ceph-mon[34354]: pgmap v722: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:30:39.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:38 smithi103 ceph-mon[34421]: pgmap v722: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 3.0 MiB/s wr, 347 op/s 2024-05-09T09:30:39.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 897075 8.64 MB/sec execute 517 sec latency 14.715 ms 2024-05-09T09:30:40.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 898867 8.64 MB/sec execute 518 sec latency 12.487 ms 2024-05-09T09:30:41.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:40 smithi069 ceph-mon[30123]: pgmap v723: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-05-09T09:30:41.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:40 smithi163 ceph-mon[34354]: pgmap v723: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-05-09T09:30:41.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:40 smithi103 ceph-mon[34421]: pgmap v723: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 781 KiB/s rd, 2.7 MiB/s wr, 308 op/s 2024-05-09T09:30:41.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 900781 8.64 MB/sec execute 519 sec latency 71.430 ms 2024-05-09T09:30:42.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 902459 8.65 MB/sec execute 520 sec latency 32.977 ms 2024-05-09T09:30:43.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:42 smithi069 ceph-mon[30123]: pgmap v724: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-05-09T09:30:43.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:42 smithi163 ceph-mon[34354]: pgmap v724: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-05-09T09:30:43.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:42 smithi103 ceph-mon[34421]: pgmap v724: 129 pgs: 129 active+clean; 163 MiB data, 3.1 GiB used, 1.0 TiB / 1.0 TiB avail; 692 KiB/s rd, 2.4 MiB/s wr, 284 op/s 2024-05-09T09:30:43.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 905256 8.66 MB/sec execute 521 sec latency 12.882 ms 2024-05-09T09:30:44.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 906548 8.65 MB/sec execute 522 sec latency 16.439 ms 2024-05-09T09:30:44.919 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:44 smithi103 ceph-mon[34421]: pgmap v725: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 376 op/s 2024-05-09T09:30:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:44 smithi069 ceph-mon[30123]: pgmap v725: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 376 op/s 2024-05-09T09:30:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 -- 172.21.15.69:0/2055532274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41017a0 msgr2=0x7fcdf4101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/2055532274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41017a0 0x7fcdf4101c20 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7fcddc009900 tx=0x7fcddc02a410 comp rx=0 tx=0).stop 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 -- 172.21.15.69:0/2055532274 shutdown_connections 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/2055532274 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf4102160 0x7fcdf41089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/2055532274 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf41017a0 0x7fcdf4101c20 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/2055532274 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41005b0 0x7fcdf41009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 -- 172.21.15.69:0/2055532274 >> 172.21.15.69:0/2055532274 conn(0x7fcdf40fbb20 msgr2=0x7fcdf40fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 -- 172.21.15.69:0/2055532274 shutdown_connections 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.027+0000 7fcdf8da0640 1 -- 172.21.15.69:0/2055532274 wait complete. 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 Processor -- start 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 -- start start 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41017a0 0x7fcdf4197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 0x7fcdf419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d330 con 0x7fcdf4102160 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d1b0 con 0x7fcdf41005b0 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf8da0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf410d4b0 con 0x7fcdf41017a0 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 0x7fcdf419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:33874/0 (socket says 172.21.15.69:33874) 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 -- 172.21.15.69:0/1812432152 learned_addr learned my addr 172.21.15.69:0/1812432152 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdebfff640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41017a0 0x7fcdf4197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 -- 172.21.15.69:0/1812432152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41017a0 msgr2=0x7fcdf4197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41017a0 0x7fcdf4197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 -- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 msgr2=0x7fcdf419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 0x7fcdf419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf37fe640 1 -- 172.21.15.69:0/1812432152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcdf419f620 con 0x7fcdf41005b0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.028+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 0x7fcdf419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d950 tx=0x7fcde400de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf17fa640 1 -- 172.21.15.69:0/1812432152 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4017070 con 0x7fcdf41005b0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcdf419f910 con 0x7fcdf41005b0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf17fa640 1 -- 172.21.15.69:0/1812432152 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcde4009c30 con 0x7fcdf41005b0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcdf419fe30 con 0x7fcdf41005b0 2024-05-09T09:30:45.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.029+0000 7fcdf17fa640 1 -- 172.21.15.69:0/1812432152 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4021060 con 0x7fcdf41005b0 2024-05-09T09:30:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 Processor -- start 2024-05-09T09:30:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 -- start start 2024-05-09T09:30:45.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdf17fa640 1 -- 172.21.15.69:0/1812432152 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcde401a070 con 0x7fcdf41005b0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de3537f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de3588c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de35b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e7de2b7340 con 0x55e7de353dc0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7de25f290 con 0x55e7de358f30 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7de2b3300 con 0x55e7de3533d0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de3588c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de3537f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de35b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de3588c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46590/0 (socket says 172.21.15.69:46590) 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdf37fe640 1 -- 172.21.15.69:0/3714506602 learned_addr learned my addr 172.21.15.69:0/3714506602 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.032+0000 7fcdebfff640 1 -- 172.21.15.69:0/3714506602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 msgr2=0x55e7de3537f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de3537f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdebfff640 1 -- 172.21.15.69:0/3714506602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 msgr2=0x55e7de3588c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de3588c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdebfff640 1 -- 172.21.15.69:0/3714506602 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7de2838e0 con 0x55e7de358f30 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de35b3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcddc031440 tx=0x7fcddc032c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5ae6545523bd850e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdf17fa640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 0x7fcde00a5040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 0x7fcde00a5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcde9ffb640 1 -- 172.21.15.69:0/3714506602 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc040070 con 0x55e7de358f30 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcde9ffb640 1 -- 172.21.15.69:0/3714506602 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcddc032e20 con 0x55e7de358f30 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcde9ffb640 1 -- 172.21.15.69:0/3714506602 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc035040 con 0x55e7de358f30 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.033+0000 7fcdf17fa640 1 -- 172.21.15.69:0/1812432152 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcde40cee30 con 0x7fcdf41005b0 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3714506602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 msgr2=0x55e7de35b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de35b3c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fcddc031440 tx=0x7fcddc032c30 comp rx=0 tx=0).stop 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 0x7fcde00a5040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55e7de354620 tx=0x7fcdec009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3714506602 shutdown_connections 2024-05-09T09:30:45.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de35b3c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de3588c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3714506602 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de3537f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3714506602 >> 172.21.15.69:0/3714506602 conn(0x55e7de34ea40 msgr2=0x55e7de34ee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3714506602 shutdown_connections 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3714506602 wait complete. 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 -- start start 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.034+0000 7fcdfcdea740 1 Processor -- start 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35d5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de364b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e7de35be80 con 0x55e7de353dc0 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e7de272790 con 0x55e7de358f30 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e7de35c000 con 0x55e7de3533d0 2024-05-09T09:30:45.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdfcdea740 10 client.0 _ll_register_callbacks cb 0x55e7de2ada20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35d5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46598/0 (socket says 172.21.15.69:46598) 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de364b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 -- 172.21.15.69:0/3243883142 learned_addr learned my addr 172.21.15.69:0/3243883142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 -- 172.21.15.69:0/3243883142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 msgr2=0x55e7de35d5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35d5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 -- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 msgr2=0x55e7de364b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de364b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 -- 172.21.15.69:0/3243883142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7ddfcb210 con 0x55e7de353dc0 2024-05-09T09:30:45.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de364b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdebfff640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005770 tx=0x7fcddc02ab90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=87b8a7f539554294 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.035+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc051060 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcddc032e20 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcddc004040 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fcddc058050 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 10 client.17007 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e7de097620 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcddc032e20 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7fcddc004810 con 0x55e7de353dc0 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 10 client.17007 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 10 client.17007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:45.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 20 client.17007 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de368d10 0x55e7de36b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 4 client.17007 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e7de2c4a70 con 0x55e7de368d10 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.036+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de368d10 0x55e7de36b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.037+0000 7fcdd2ffd640 10 client.17007 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:45.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.037+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de368d10 0x55e7de36b120 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7443af35be3b071d server_cookie=48bef7c94000cc51 in_seq=0 out_seq=0 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/3243883142 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e7de2c4a70 con 0x55e7de368d10 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdd2ffd640 10 client.17007 handle_command_reply: tid=0 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 client.17007 shutdown 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 msgr2=0x55e7de35db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7fcddc005770 tx=0x7fcddc02ab90 comp rx=0 tx=0).stop 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 shutdown_connections 2024-05-09T09:30:45.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de358f30 0x55e7de364b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de353dc0 0x55e7de35db20 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de368d10 0x55e7de36b120 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/3243883142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35d5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdd2ffd640 0 client.17007 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 >> 172.21.15.69:0/3243883142 conn(0x55e7de34ea40 msgr2=0x55e7de34dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 shutdown_connections 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 1 -- 172.21.15.69:0/3243883142 wait complete. 2024-05-09T09:30:45.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.039+0000 7fcdfcdea740 20 client.17007 trim_cache size 0 max 16384 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 Processor -- start 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 -- start start 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41aca10 con 0x55e7de358f30 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41aa280 con 0x55e7de353dc0 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41a80d0 con 0x55e7de3533d0 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.054+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdf3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:46606/0 (socket says 172.21.15.69:46606) 2024-05-09T09:30:45.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdf3fff640 1 -- 172.21.15.69:0/1976068268 learned_addr learned my addr 172.21.15.69:0/1976068268 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:33902/0 (socket says 172.21.15.69:33902) 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 -- 172.21.15.69:0/1976068268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 msgr2=0x55e7de35c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 -- 172.21.15.69:0/1976068268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 msgr2=0x55e7de355040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 -- 172.21.15.69:0/1976068268 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e7de340330 con 0x55e7de353dc0 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:45.057 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcdebfff640 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0057c0 tx=0x7fcddc032c30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82d78080990f3866 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.058 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcde97fa640 1 -- 172.21.15.69:0/1976068268 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc051040 con 0x55e7de353dc0 2024-05-09T09:30:45.058 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcde97fa640 1 -- 172.21.15.69:0/1976068268 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcddc032d20 con 0x55e7de353dc0 2024-05-09T09:30:45.058 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.055+0000 7fcde97fa640 1 -- 172.21.15.69:0/1976068268 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcddc035040 con 0x55e7de353dc0 2024-05-09T09:30:45.058 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:44 smithi163 ceph-mon[34354]: pgmap v725: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 3.1 MiB/s wr, 376 op/s 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- 172.21.15.69:0/1976068268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 msgr2=0x55e7de364b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0057c0 tx=0x7fcddc032c30 comp rx=0 tx=0).stop 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- 172.21.15.69:0/1976068268 shutdown_connections 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de355040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de353dc0 0x55e7de364b20 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/1976068268 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de3533d0 0x55e7de35c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- 172.21.15.69:0/1976068268 >> 172.21.15.69:0/1976068268 conn(0x55e7de34ea40 msgr2=0x55e7de34ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- 172.21.15.69:0/1976068268 shutdown_connections 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- 172.21.15.69:0/1976068268 wait complete. 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.056+0000 7fcdfcdea740 1 -- start start 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 Processor -- start 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de353dc0 0x55e7de362060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de3cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41a8740 con 0x55e7de358f30 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41a9bb0 con 0x55e7de3533d0 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcdf41a8e10 con 0x55e7de353dc0 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdfcdea740 10 client.0 _ll_register_callbacks cb 0x7fcde40d27e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:45.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:33906/0 (socket says 172.21.15.69:33906) 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 -- 172.21.15.69:0/69329402 learned_addr learned my addr 172.21.15.69:0/69329402 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf3fff640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de3cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdebfff640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de353dc0 0x55e7de362060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 -- 172.21.15.69:0/69329402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de353dc0 msgr2=0x55e7de362060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de353dc0 0x55e7de362060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 -- 172.21.15.69:0/69329402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 msgr2=0x55e7de3cd240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de3cd240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.057+0000 7fcdf37fe640 1 -- 172.21.15.69:0/69329402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e7de3cd800 con 0x55e7de3533d0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdf37fe640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcde40cc600 tx=0x7fcde4010ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c3c8da6e5fd8c71c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcde4009de0 con 0x55e7de3533d0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcde40996c0 con 0x55e7de3533d0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcddc032d20 con 0x55e7de3533d0 2024-05-09T09:30:45.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=41) ==== 24+0+0 (secure 0 0 0) 0x7fcde409b3c0 con 0x55e7de3533d0 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 10 client.26324 fetch_fsmap learned FSMap version 41 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=41,monmap=4+}) -- 0x55e7de3ceb80 con 0x55e7de3533d0 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcddc032d20 con 0x55e7de3533d0 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 41) ==== 8354+0+0 (secure 0 0 0) 0x7fcde40b4160 con 0x55e7de3533d0 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 10 client.26324 fetch_fsmap finished waiting for FSMap version 41 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 10 client.26324 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 20 client.26324 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.058+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de3cf570 0x55e7de3d1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.059+0000 7fcdfcdea740 4 client.26324 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.059+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e7de2bedc0 con 0x55e7de3cf570 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.059+0000 7fcdebfff640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de3cf570 0x55e7de3d1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.059+0000 7fcdd2ffd640 10 client.26324 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:45.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.059+0000 7fcdebfff640 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de3cf570 0x55e7de3d1990 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c3d1111c964385b server_cookie=b1710c28a3e90068 in_seq=0 out_seq=0 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.196+0000 7fcdd2ffd640 1 -- 172.21.15.69:0/69329402 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e7de2bedc0 con 0x55e7de3cf570 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.196+0000 7fcdd2ffd640 10 client.26324 handle_command_reply: tid=0 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.196+0000 7fcdfcdea740 1 client.26324 shutdown 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 msgr2=0x55e7de361b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcde40cc600 tx=0x7fcde4010ce0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 shutdown_connections 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e7de358f30 0x55e7de3cd240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e7de3cf570 0x55e7de3d1990 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e7de353dc0 0x55e7de362060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 --2- 172.21.15.69:0/69329402 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e7de3533d0 0x55e7de361b20 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdd2ffd640 0 client.26324 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:45.198 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 >> 172.21.15.69:0/69329402 conn(0x55e7de34ea40 msgr2=0x55e7de35c3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 shutdown_connections 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 1 -- 172.21.15.69:0/69329402 wait complete. 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdfcdea740 20 client.26324 trim_cache size 0 max 16384 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:30:45.199 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:30:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.197+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 msgr2=0x7fcde00a5040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 0x7fcde00a5040 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55e7de354620 tx=0x7fcdec009500 comp rx=0 tx=0).stop 2024-05-09T09:30:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 msgr2=0x7fcdf4197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:45.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fcde400d950 tx=0x7fcde400de10 comp rx=0 tx=0).stop 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 shutdown_connections 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcde00a2b80 0x7fcde00a5040 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcdf4102160 0x7fcdf419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcdf41017a0 0x7fcdf4197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 --2- 172.21.15.69:0/1812432152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcdf41005b0 0x7fcdf4197a00 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 >> 172.21.15.69:0/1812432152 conn(0x7fcdf40fbb20 msgr2=0x7fcdf4104b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 shutdown_connections 2024-05-09T09:30:45.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:45.198+0000 7fcdf8da0640 1 -- 172.21.15.69:0/1812432152 wait complete. 2024-05-09T09:30:45.505 INFO:tasks.workunit.client.0.smithi069.stdout: 1 909063 8.67 MB/sec execute 523 sec latency 90.466 ms 2024-05-09T09:30:46.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 909975 8.66 MB/sec execute 524 sec latency 37.059 ms 2024-05-09T09:30:47.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:46 smithi069 ceph-mon[30123]: pgmap v726: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-05-09T09:30:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:46 smithi163 ceph-mon[34354]: pgmap v726: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-05-09T09:30:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:46 smithi103 ceph-mon[34421]: pgmap v726: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 776 KiB/s rd, 2.7 MiB/s wr, 304 op/s 2024-05-09T09:30:47.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 912629 8.67 MB/sec execute 525 sec latency 26.545 ms 2024-05-09T09:30:48.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 914228 8.67 MB/sec execute 526 sec latency 14.466 ms 2024-05-09T09:30:49.014 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:48 smithi069 ceph-mon[30123]: pgmap v727: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-05-09T09:30:49.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:48 smithi163 ceph-mon[34354]: pgmap v727: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-05-09T09:30:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:48 smithi103 ceph-mon[34421]: pgmap v727: 129 pgs: 129 active+clean; 165 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 969 KiB/s rd, 3.1 MiB/s wr, 369 op/s 2024-05-09T09:30:49.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 916380 8.68 MB/sec execute 527 sec latency 9.833 ms 2024-05-09T09:30:50.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 917482 8.70 MB/sec execute 528 sec latency 44.730 ms 2024-05-09T09:30:51.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:50 smithi069 ceph-mon[30123]: pgmap v728: 129 pgs: 129 active+clean; 165 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 323 op/s 2024-05-09T09:30:51.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:50 smithi163 ceph-mon[34354]: pgmap v728: 129 pgs: 129 active+clean; 165 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 323 op/s 2024-05-09T09:30:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:50 smithi103 ceph-mon[34421]: pgmap v728: 129 pgs: 129 active+clean; 165 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.7 MiB/s wr, 323 op/s 2024-05-09T09:30:51.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 919713 8.71 MB/sec execute 529 sec latency 65.922 ms 2024-05-09T09:30:52.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:51 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:52.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:51 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:52.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:51 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:30:52.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 920805 8.70 MB/sec execute 530 sec latency 213.913 ms 2024-05-09T09:30:53.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:52 smithi069 ceph-mon[30123]: pgmap v729: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-05-09T09:30:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:52 smithi163 ceph-mon[34354]: pgmap v729: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-05-09T09:30:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:52 smithi103 ceph-mon[34421]: pgmap v729: 129 pgs: 129 active+clean; 166 MiB data, 3.2 GiB used, 1.0 TiB / 1.0 TiB avail; 780 KiB/s rd, 2.5 MiB/s wr, 334 op/s 2024-05-09T09:30:53.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 923023 8.71 MB/sec execute 531 sec latency 17.491 ms 2024-05-09T09:30:54.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 924579 8.71 MB/sec execute 532 sec latency 17.582 ms 2024-05-09T09:30:55.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:54 smithi069 ceph-mon[30123]: pgmap v730: 129 pgs: 129 active+clean; 163 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 374 op/s 2024-05-09T09:30:55.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:54 smithi163 ceph-mon[34354]: pgmap v730: 129 pgs: 129 active+clean; 163 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 374 op/s 2024-05-09T09:30:55.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:54 smithi103 ceph-mon[34421]: pgmap v730: 129 pgs: 129 active+clean; 163 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 870 KiB/s rd, 3.0 MiB/s wr, 374 op/s 2024-05-09T09:30:55.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 927309 8.72 MB/sec execute 533 sec latency 14.421 ms 2024-05-09T09:30:56.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 928163 8.71 MB/sec execute 534 sec latency 78.102 ms 2024-05-09T09:30:57.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:56 smithi069 ceph-mon[30123]: pgmap v731: 129 pgs: 129 active+clean; 160 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.6 MiB/s wr, 342 op/s 2024-05-09T09:30:57.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:56 smithi163 ceph-mon[34354]: pgmap v731: 129 pgs: 129 active+clean; 160 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.6 MiB/s wr, 342 op/s 2024-05-09T09:30:57.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:56 smithi103 ceph-mon[34421]: pgmap v731: 129 pgs: 129 active+clean; 160 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 713 KiB/s rd, 2.6 MiB/s wr, 342 op/s 2024-05-09T09:30:57.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 930722 8.72 MB/sec execute 535 sec latency 32.408 ms 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3836433661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78100790 msgr2=0x7f4b78100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3836433661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78100790 0x7f4b78100bb0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f4b68009900 tx=0x7f4b6802a410 comp rx=0 tx=0).stop 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3836433661 shutdown_connections 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3836433661 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78102340 0x7f4b78108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3836433661 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b78101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3836433661 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78100790 0x7f4b78100bb0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3836433661 >> 172.21.15.69:0/3836433661 conn(0x7f4b780fbd00 msgr2=0x7f4b780fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3836433661 shutdown_connections 2024-05-09T09:30:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.300+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3836433661 wait complete. 2024-05-09T09:30:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 Processor -- start 2024-05-09T09:30:58.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 -- start start 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b781980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810c400 con 0x7f4b78102340 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810c280 con 0x7f4b78100790 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7cf61640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b7810c580 con 0x7f4b78101980 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b781980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.303 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60314/0 (socket says 172.21.15.69:60314) 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b781980d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:48914/0 (socket says 172.21.15.69:48914) 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34796/0 (socket says 172.21.15.69:34796) 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b6ffff640 1 -- 172.21.15.69:0/3028146782 learned_addr learned my addr 172.21.15.69:0/3028146782 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 -- 172.21.15.69:0/3028146782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 msgr2=0x7f4b781980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b781980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 -- 172.21.15.69:0/3028146782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 msgr2=0x7f4b78197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.301+0000 7f4b76d9d640 1 -- 172.21.15.69:0/3028146782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f4b7819f760 con 0x7f4b78102340 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b76d9d640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009fd0 tx=0x7f4b7000ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.304 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b6f7fe640 1 -- 172.21.15.69:0/3028146782 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b7000ce20 con 0x7f4b78102340 2024-05-09T09:30:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b6f7fe640 1 -- 172.21.15.69:0/3028146782 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b70005020 con 0x7f4b78102340 2024-05-09T09:30:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f4b7819fa50 con 0x7f4b78102340 2024-05-09T09:30:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b6f7fe640 1 -- 172.21.15.69:0/3028146782 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b70010720 con 0x7f4b78102340 2024-05-09T09:30:58.305 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.302+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f4b7819ff50 con 0x7f4b78102340 2024-05-09T09:30:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.305+0000 7f4b6f7fe640 1 -- 172.21.15.69:0/3028146782 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f4b70005540 con 0x7f4b78102340 2024-05-09T09:30:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.305+0000 7f4b7fc6f740 1 Processor -- start 2024-05-09T09:30:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 -- start start 2024-05-09T09:30:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b8f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b92240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559501aee340 con 0x559501b8fdb0 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559501a96290 con 0x559501b8a250 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559501aea300 con 0x559501b8ac40 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60330/0 (socket says 172.21.15.69:60330) 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1071812362 learned_addr learned my addr 172.21.15.69:0/1071812362 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6f7fe640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 0x7f4b640a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b92240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b7659c640 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b8f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1071812362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 msgr2=0x559501b8f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.306+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b8f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1071812362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 msgr2=0x559501b92240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b92240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1071812362 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559501aba8e0 con 0x559501b8a250 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b76d9d640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 0x7f4b640a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6f7fe640 1 -- 172.21.15.69:0/3028146782 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4b700cf9f0 con 0x7f4b78102340 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b60011910 tx=0x7f4b60009880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=998ebf6e6f3b4a7f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6cff9640 1 -- 172.21.15.69:0/1071812362 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000f7c0 con 0x559501b8a250 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6cff9640 1 -- 172.21.15.69:0/1071812362 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6000fe40 con 0x559501b8a250 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b76d9d640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 0x7f4b640a5170 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4b70098330 tx=0x7f4b70005940 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.307+0000 7f4b6cff9640 1 -- 172.21.15.69:0/1071812362 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000e3b0 con 0x559501b8a250 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1071812362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 msgr2=0x559501b8a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f4b60011910 tx=0x7f4b60009880 comp rx=0 tx=0).stop 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1071812362 shutdown_connections 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b92240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b8f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1071812362 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8a250 0x559501b8a670 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1071812362 >> 172.21.15.69:0/1071812362 conn(0x559501b858c0 msgr2=0x559501b85cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1071812362 shutdown_connections 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1071812362 wait complete. 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.308+0000 7f4b7fc6f740 1 -- start start 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 Processor -- start 2024-05-09T09:30:58.315 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b9b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559501b92d60 con 0x559501b8a250 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559501aa9790 con 0x559501b8fdb0 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559501b92ee0 con 0x559501b8ac40 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7fc6f740 10 client.0 _ll_register_callbacks cb 0x559501ae4a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34816/0 (socket says 172.21.15.69:34816) 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 -- 172.21.15.69:0/3313900299 learned_addr learned my addr 172.21.15.69:0/3313900299 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:48940/0 (socket says 172.21.15.69:48940) 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b9b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 -- 172.21.15.69:0/3313900299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 msgr2=0x559501b949a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 -- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 msgr2=0x559501b9b9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b9b9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b6ffff640 1 -- 172.21.15.69:0/3313900299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559501802210 con 0x559501b8a250 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.309+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009610 tx=0x7f4b60009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1cd818186acbdfd2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6000f790 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4b6000f930 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b6000fe40 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4b60038050 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b7fc6f740 10 client.17025 fetch_fsmap learned FSMap version 42 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5595018ce620 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.310+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b6000fe40 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f4b60033030 con 0x559501b8a250 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 10 client.17025 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 10 client.17025 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:58.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 20 client.17025 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501b9fcf0 0x559501ba2130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 4 client.17025 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x559501afba70 con 0x559501b9fcf0 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501b9fcf0 0x559501ba2130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b5dffb640 10 client.17025 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.311+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501b9fcf0 0x559501ba2130 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ef75d40083a5e28 server_cookie=f0297834a4836528 in_seq=0 out_seq=0 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b5dffb640 1 -- 172.21.15.69:0/3313900299 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x559501afba70 con 0x559501b9fcf0 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b5dffb640 10 client.17025 handle_command_reply: tid=0 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 client.17025 shutdown 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 msgr2=0x559501b94460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f4b60009610 tx=0x7f4b60009fa0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 shutdown_connections 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b9b9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501b9fcf0 0x559501ba2130 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8ac40 0x559501b949a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3313900299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8a250 0x559501b94460 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 >> 172.21.15.69:0/3313900299 conn(0x559501b858c0 msgr2=0x559501b84a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b5dffb640 0 client.17025 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 shutdown_connections 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.312+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3313900299 wait complete. 2024-05-09T09:30:58.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.313+0000 7f4b7fc6f740 20 client.17025 trim_cache size 0 max 16384 2024-05-09T09:30:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 Processor -- start 2024-05-09T09:30:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 -- start start 2024-05-09T09:30:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b8b560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b84170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781accc0 con 0x559501b8ac40 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781aa530 con 0x559501b8fdb0 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781a8380 con 0x559501b8a250 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34826/0 (socket says 172.21.15.69:34826) 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 -- 172.21.15.69:0/3129438002 learned_addr learned my addr 172.21.15.69:0/3129438002 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b8b560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b84170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 -- 172.21.15.69:0/3129438002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 msgr2=0x559501b8b560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b8b560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 -- 172.21.15.69:0/3129438002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 msgr2=0x559501b84170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.329+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b84170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.330+0000 7f4b7659c640 1 -- 172.21.15.69:0/3129438002 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x559501b89e90 con 0x559501b8ac40 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.330+0000 7f4b7659c640 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4b680314e0 tx=0x7f4b68005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2105af43f454d92e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.330+0000 7f4b5ffff640 1 -- 172.21.15.69:0/3129438002 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68048020 con 0x559501b8ac40 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.330+0000 7f4b5ffff640 1 -- 172.21.15.69:0/3129438002 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68044b50 con 0x559501b8ac40 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.330+0000 7f4b5ffff640 1 -- 172.21.15.69:0/3129438002 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b68035040 con 0x559501b8ac40 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3129438002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 msgr2=0x559501b9b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f4b680314e0 tx=0x7f4b68005fd0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3129438002 shutdown_connections 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8fdb0 0x559501b84170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8ac40 0x559501b9b9a0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/3129438002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b8b560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3129438002 >> 172.21.15.69:0/3129438002 conn(0x559501b858c0 msgr2=0x559501b94070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3129438002 shutdown_connections 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/3129438002 wait complete. 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- start start 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 Processor -- start 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b98c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 0x559501b99170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b9a180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781a89f0 con 0x559501b8fdb0 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781a9e60 con 0x559501b8ac40 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f4b781a90c0 con 0x559501b8a250 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7fc6f740 10 client.0 _ll_register_callbacks cb 0x7f4b60011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:30:58.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b9a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 0x559501b99170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b6ffff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b98c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 0x559501b99170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60366/0 (socket says 172.21.15.69:60366) 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.331+0000 7f4b7659c640 1 -- 172.21.15.69:0/1553333284 learned_addr learned my addr 172.21.15.69:0/1553333284 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1553333284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 msgr2=0x559501b98c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b98c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 msgr2=0x559501b99170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 0x559501b99170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 -- 172.21.15.69:0/1553333284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559501b9a770 con 0x559501b8fdb0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b6ffff640 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b9a180 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4b60001eb0 tx=0x7f4b60009fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5c2f178abed2d8d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:30:58.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4b6001fba0 con 0x559501b8fdb0 2024-05-09T09:30:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4b6001fd40 con 0x559501b8fdb0 2024-05-09T09:30:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f4b68044b50 con 0x559501b8fdb0 2024-05-09T09:30:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f4b60020e40 con 0x559501b8fdb0 2024-05-09T09:30:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b7fc6f740 10 client.17031 fetch_fsmap learned FSMap version 42 2024-05-09T09:30:58.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559501b9ba80 con 0x559501b8fdb0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f4b68044b50 con 0x559501b8fdb0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f4b60033040 con 0x559501b8fdb0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.332+0000 7f4b7fc6f740 10 client.17031 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7fc6f740 10 client.17031 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7fc6f740 20 client.17031 populate_metadata read hostname 'smithi069' 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501c066b0 0x559501c08ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7fc6f740 4 client.17031 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559501af5dc0 con 0x559501c066b0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7659c640 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501c066b0 0x559501c08ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b7659c640 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501c066b0 0x559501c08ad0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8487458c07b311f4 server_cookie=c50dd37c893b321b in_seq=0 out_seq=0 2024-05-09T09:30:58.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.333+0000 7f4b5dffb640 10 client.17031 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b5dffb640 1 -- 172.21.15.69:0/1553333284 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559501af5dc0 con 0x559501c066b0 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b5dffb640 10 client.17031 handle_command_reply: tid=0 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 client.17031 shutdown 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 msgr2=0x559501b9a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b9a180 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f4b60001eb0 tx=0x7f4b60009fa0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 shutdown_connections 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559501b8fdb0 0x559501b9a180 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559501b8ac40 0x559501b99170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559501c066b0 0x559501c08ad0 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 --2- 172.21.15.69:0/1553333284 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559501b8a250 0x559501b98c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 >> 172.21.15.69:0/1553333284 conn(0x559501b858c0 msgr2=0x559501b88640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b5dffb640 0 client.17031 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 shutdown_connections 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 1 -- 172.21.15.69:0/1553333284 wait complete. 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.456+0000 7f4b7fc6f740 20 client.17031 trim_cache size 0 max 16384 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:30:58.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:30:58.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:30:58.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 msgr2=0x7f4b640a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 0x7f4b640a5170 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f4b70098330 tx=0x7f4b70005940 comp rx=0 tx=0).stop 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 msgr2=0x7f4b7819f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f4b70009fd0 tx=0x7f4b7000ef10 comp rx=0 tx=0).stop 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 shutdown_connections 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f4b640a2cb0 0x7f4b640a5170 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4b78102340 0x7f4b7819f0f0 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.457+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4b78101980 0x7f4b781980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.458+0000 7f4b7cf61640 1 --2- 172.21.15.69:0/3028146782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f4b78100790 0x7f4b78197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:30:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.458+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 >> 172.21.15.69:0/3028146782 conn(0x7f4b780fbd00 msgr2=0x7f4b780fd780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:30:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.458+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 shutdown_connections 2024-05-09T09:30:58.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:30:58.458+0000 7f4b7cf61640 1 -- 172.21.15.69:0/3028146782 wait complete. 2024-05-09T09:30:58.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 931767 8.71 MB/sec execute 536 sec latency 117.501 ms 2024-05-09T09:30:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:30:58 smithi069 ceph-mon[30123]: pgmap v732: 129 pgs: 129 active+clean; 157 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:30:59.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:30:58 smithi163 ceph-mon[34354]: pgmap v732: 129 pgs: 129 active+clean; 157 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:30:59.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:30:58 smithi103 ceph-mon[34421]: pgmap v732: 129 pgs: 129 active+clean; 157 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 684 KiB/s rd, 2.7 MiB/s wr, 338 op/s 2024-05-09T09:30:59.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 934490 8.73 MB/sec execute 537 sec latency 17.371 ms 2024-05-09T09:31:00.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 935817 8.72 MB/sec execute 538 sec latency 14.247 ms 2024-05-09T09:31:01.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:00 smithi069 ceph-mon[30123]: pgmap v733: 129 pgs: 129 active+clean; 159 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2024-05-09T09:31:01.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:00 smithi163 ceph-mon[34354]: pgmap v733: 129 pgs: 129 active+clean; 159 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2024-05-09T09:31:01.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:00 smithi103 ceph-mon[34421]: pgmap v733: 129 pgs: 129 active+clean; 159 MiB data, 3.3 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.5 MiB/s wr, 301 op/s 2024-05-09T09:31:01.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 938132 8.73 MB/sec execute 539 sec latency 64.357 ms 2024-05-09T09:31:02.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 939167 8.72 MB/sec execute 540 sec latency 35.160 ms 2024-05-09T09:31:03.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:02 smithi069 ceph-mon[30123]: pgmap v734: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 284 op/s 2024-05-09T09:31:03.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:02 smithi163 ceph-mon[34354]: pgmap v734: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 284 op/s 2024-05-09T09:31:03.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:02 smithi103 ceph-mon[34421]: pgmap v734: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 284 op/s 2024-05-09T09:31:03.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 941782 8.74 MB/sec execute 541 sec latency 30.892 ms 2024-05-09T09:31:04.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 943010 8.73 MB/sec execute 542 sec latency 20.178 ms 2024-05-09T09:31:05.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:04 smithi069 ceph-mon[30123]: pgmap v735: 129 pgs: 129 active+clean; 157 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-09T09:31:05.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:04 smithi163 ceph-mon[34354]: pgmap v735: 129 pgs: 129 active+clean; 157 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-09T09:31:05.120 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:04 smithi103 ceph-mon[34421]: pgmap v735: 129 pgs: 129 active+clean; 157 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 608 KiB/s rd, 2.9 MiB/s wr, 332 op/s 2024-05-09T09:31:05.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 945399 8.74 MB/sec execute 543 sec latency 10.899 ms 2024-05-09T09:31:06.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 946624 8.73 MB/sec execute 544 sec latency 83.230 ms 2024-05-09T09:31:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:06 smithi069 ceph-mon[30123]: pgmap v736: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-05-09T09:31:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:06 smithi163 ceph-mon[34354]: pgmap v736: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-05-09T09:31:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:06 smithi103 ceph-mon[34421]: pgmap v736: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 636 KiB/s rd, 2.5 MiB/s wr, 324 op/s 2024-05-09T09:31:07.506 INFO:tasks.workunit.client.0.smithi069.stdout: 1 948898 8.74 MB/sec execute 545 sec latency 24.238 ms 2024-05-09T09:31:08.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 950292 8.74 MB/sec execute 546 sec latency 15.641 ms 2024-05-09T09:31:09.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:08 smithi069 ceph-mon[30123]: pgmap v737: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-05-09T09:31:09.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:08 smithi163 ceph-mon[34354]: pgmap v737: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-05-09T09:31:09.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:08 smithi103 ceph-mon[34421]: pgmap v737: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 2.8 MiB/s wr, 323 op/s 2024-05-09T09:31:09.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 952517 8.75 MB/sec execute 547 sec latency 21.178 ms 2024-05-09T09:31:10.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 954047 8.74 MB/sec execute 548 sec latency 15.207 ms 2024-05-09T09:31:11.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:10 smithi069 ceph-mon[30123]: pgmap v738: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-09T09:31:11.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:10 smithi163 ceph-mon[34354]: pgmap v738: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-09T09:31:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:10 smithi103 ceph-mon[34421]: pgmap v738: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 590 KiB/s rd, 2.6 MiB/s wr, 313 op/s 2024-05-09T09:31:11.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 956080 8.75 MB/sec execute 549 sec latency 76.318 ms 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 -- 172.21.15.69:0/3715426265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c0102360 msgr2=0x7f28c0108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 --2- 172.21.15.69:0/3715426265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c0102360 0x7f28c0108bf0 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f28b8009ca0 tx=0x7f28b802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 -- 172.21.15.69:0/3715426265 shutdown_connections 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 --2- 172.21.15.69:0/3715426265 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c0102360 0x7f28c0108bf0 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 --2- 172.21.15.69:0/3715426265 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01019a0 0x7f28c0101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 --2- 172.21.15.69:0/3715426265 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c01007b0 0x7f28c0100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 -- 172.21.15.69:0/3715426265 >> 172.21.15.69:0/3715426265 conn(0x7f28c00fbd20 msgr2=0x7f28c00fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 -- 172.21.15.69:0/3715426265 shutdown_connections 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.553+0000 7f28c5679640 1 -- 172.21.15.69:0/3715426265 wait complete. 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 Processor -- start 2024-05-09T09:31:11.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 -- start start 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01007b0 0x7f28c006ab70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c0102360 0x7f28c01a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010c480 con 0x7f28c01019a0 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010c300 con 0x7f28c01007b0 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28c5679640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c010c600 con 0x7f28c0102360 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28beffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01007b0 0x7f28c006ab70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37812/0 (socket says 172.21.15.69:37812) 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 -- 172.21.15.69:0/2222200227 learned_addr learned my addr 172.21.15.69:0/2222200227 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c0102360 0x7f28c01a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 -- 172.21.15.69:0/2222200227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c0102360 msgr2=0x7f28c01a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c0102360 0x7f28c01a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 -- 172.21.15.69:0/2222200227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01007b0 msgr2=0x7f28c006ab70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.554+0000 7f28be7fc640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01007b0 0x7f28c006ab70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28be7fc640 1 -- 172.21.15.69:0/2222200227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f28c01a1930 con 0x7f28c01019a0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28be7fc640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f28b400e9f0 tx=0x7f28b400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28a7fff640 1 -- 172.21.15.69:0/2222200227 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b400cd80 con 0x7f28c01019a0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f28c01a1bc0 con 0x7f28c01019a0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28a7fff640 1 -- 172.21.15.69:0/2222200227 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b4010910 con 0x7f28c01019a0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28a7fff640 1 -- 172.21.15.69:0/2222200227 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b40188f0 con 0x7f28c01019a0 2024-05-09T09:31:11.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.555+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f28c01a2160 con 0x7f28c01019a0 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.557+0000 7f28a7fff640 1 -- 172.21.15.69:0/2222200227 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f28b4010ac0 con 0x7f28c01019a0 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 Processor -- start 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 -- start start 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28a7fff640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 0x7f28a80a5120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 0x564b9c48e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c491240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28a7fff640 1 -- 172.21.15.69:0/2222200227 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f28b40cf720 con 0x7f28c01019a0 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 0x7f28a80a5120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564b9c3ed340 con 0x564b9c48edb0 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564b9c395290 con 0x564b9c489c40 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28c8383740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564b9c3e9300 con 0x564b9c489250 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28beffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 0x564b9c48e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28beffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:56340/0 (socket says 172.21.15.69:56340) 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28beffd640 1 -- 172.21.15.69:0/991576057 learned_addr learned my addr 172.21.15.69:0/991576057 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c491240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28be7fc640 1 -- 172.21.15.69:0/991576057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 msgr2=0x564b9c489670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28be7fc640 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28be7fc640 1 -- 172.21.15.69:0/991576057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 msgr2=0x564b9c491240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.558+0000 7f28be7fc640 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c491240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28be7fc640 1 -- 172.21.15.69:0/991576057 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b9c3b98e0 con 0x564b9c489c40 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28beffd640 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:11.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 0x7f28a80a5120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f28b8005e30 tx=0x7f28b802f390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28be7fc640 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 0x564b9c48e740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f28b40cd210 tx=0x7f28b409bd70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5a06b489f45430d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28a57fa640 1 -- 172.21.15.69:0/991576057 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4099fb0 con 0x564b9c489c40 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28a57fa640 1 -- 172.21.15.69:0/991576057 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b40b2070 con 0x564b9c489c40 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.559+0000 7f28a57fa640 1 -- 172.21.15.69:0/991576057 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b409b3f0 con 0x564b9c489c40 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- 172.21.15.69:0/991576057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 msgr2=0x564b9c48e740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 0x564b9c48e740 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f28b40cd210 tx=0x7f28b409bd70 comp rx=0 tx=0).stop 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- 172.21.15.69:0/991576057 shutdown_connections 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c491240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489c40 0x564b9c48e740 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- 172.21.15.69:0/991576057 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c489670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- 172.21.15.69:0/991576057 >> 172.21.15.69:0/991576057 conn(0x564b9c4848c0 msgr2=0x564b9c484cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- 172.21.15.69:0/991576057 shutdown_connections 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- 172.21.15.69:0/991576057 wait complete. 2024-05-09T09:31:11.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- start start 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 Processor -- start 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 0x564b9c493a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c49a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x564b9c491d60 con 0x564b9c48edb0 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x564b9c3a8790 con 0x564b9c489250 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.560+0000 7f28c8383740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x564b9c491ee0 con 0x564b9c489c40 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28c8383740 10 client.0 _ll_register_callbacks cb 0x564b9c3e3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28beffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 0x564b9c493a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28beffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:39646/0 (socket says 172.21.15.69:39646) 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28beffd640 1 -- 172.21.15.69:0/2689377576 learned_addr learned my addr 172.21.15.69:0/2689377576 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c49a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 -- 172.21.15.69:0/2689377576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 msgr2=0x564b9c493a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 0x564b9c493a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 -- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 msgr2=0x564b9c4934c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 -- 172.21.15.69:0/2689377576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b9c101210 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28beffd640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28be7fc640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 0x564b9c493a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c49a9d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f28b802ade0 tx=0x7f28b802a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c299f07e8693982b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b80046a0 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f28b802f610 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b40b2070 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.561+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f28b805a050 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 10 client.17043 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564b9c1cd620 con 0x564b9c48edb0 2024-05-09T09:31:11.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b40b2070 con 0x564b9c48edb0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f28b804b070 con 0x564b9c48edb0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 10 client.17043 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 10 client.17043 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 20 client.17043 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c49efc0 0x564b9c4a1420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 4 client.17043 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x564b9c3faa70 con 0x564b9c49efc0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28beffd640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c49efc0 0x564b9c4a1420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.562+0000 7f28beffd640 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c49efc0 0x564b9c4a1420 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aaf990c4213ae139 server_cookie=a94aed3fc712cdb2 in_seq=0 out_seq=0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28a2ff5640 10 client.17043 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28a2ff5640 1 -- 172.21.15.69:0/2689377576 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x564b9c3faa70 con 0x564b9c49efc0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28a2ff5640 10 client.17043 handle_command_reply: tid=0 2024-05-09T09:31:11.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 client.17043 shutdown 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 msgr2=0x564b9c49a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c49a9d0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f28b802ade0 tx=0x7f28b802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 shutdown_connections 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c48edb0 0x564b9c49a9d0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c49efc0 0x564b9c4a1420 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489c40 0x564b9c493a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 --2- 172.21.15.69:0/2689377576 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c489250 0x564b9c4934c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 >> 172.21.15.69:0/2689377576 conn(0x564b9c4848c0 msgr2=0x564b9c483a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28a2ff5640 0 client.17043 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:11.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 shutdown_connections 2024-05-09T09:31:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.563+0000 7f28c8383740 1 -- 172.21.15.69:0/2689377576 wait complete. 2024-05-09T09:31:11.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.564+0000 7f28c8383740 20 client.17043 trim_cache size 0 max 16384 2024-05-09T09:31:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 Processor -- start 2024-05-09T09:31:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 -- start start 2024-05-09T09:31:11.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c4a6cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c49a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01aeec0 con 0x564b9c489c40 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01ac730 con 0x564b9c48edb0 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.579+0000 7f28c8383740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01aa580 con 0x564b9c489250 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37832/0 (socket says 172.21.15.69:37832) 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 -- 172.21.15.69:0/1134601955 learned_addr learned my addr 172.21.15.69:0/1134601955 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c49a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28beffd640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c4a6cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 -- 172.21.15.69:0/1134601955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 msgr2=0x564b9c4a6cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c4a6cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 -- 172.21.15.69:0/1134601955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 msgr2=0x564b9c49a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c49a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 -- 172.21.15.69:0/1134601955 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x564b9c48e130 con 0x564b9c489c40 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28beffd640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c4a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28be7fc640 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f28b4098680 tx=0x7f28b40a0130 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21bcc8b6db480e28 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28a4ff9640 1 -- 172.21.15.69:0/1134601955 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4099d50 con 0x564b9c489c40 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28a4ff9640 1 -- 172.21.15.69:0/1134601955 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b40b2070 con 0x564b9c489c40 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.580+0000 7f28a4ff9640 1 -- 172.21.15.69:0/1134601955 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b4097a00 con 0x564b9c489c40 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- 172.21.15.69:0/1134601955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 msgr2=0x564b9c4a7230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f28b4098680 tx=0x7f28b40a0130 comp rx=0 tx=0).stop 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- 172.21.15.69:0/1134601955 shutdown_connections 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c49a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c489c40 0x564b9c4a7230 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 --2- 172.21.15.69:0/1134601955 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c4a6cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- 172.21.15.69:0/1134601955 >> 172.21.15.69:0/1134601955 conn(0x564b9c4848c0 msgr2=0x564b9c484ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- 172.21.15.69:0/1134601955 shutdown_connections 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- 172.21.15.69:0/1134601955 wait complete. 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.581+0000 7f28c8383740 1 -- start start 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 Processor -- start 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c497ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c497ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01aabf0 con 0x564b9c499000 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01ac060 con 0x564b9c48edb0 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f28c01ab2c0 con 0x564b9c489250 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28c8383740 10 client.0 _ll_register_callbacks cb 0x7f28b8009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28bf7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37838/0 (socket says 172.21.15.69:37838) 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28bf7fe640 1 -- 172.21.15.69:0/509506946 learned_addr learned my addr 172.21.15.69:0/509506946 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:11.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c497ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28beffd640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c497ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 -- 172.21.15.69:0/509506946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 msgr2=0x564b9c497ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c497ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 -- 172.21.15.69:0/509506946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 msgr2=0x564b9c499480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 -- 172.21.15.69:0/509506946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x564b9c4999c0 con 0x564b9c48edb0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28bf7fe640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28be7fc640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c497ff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x564b9c49ba50 tx=0x7f28b40a49f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6c192cc4a131356d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.582+0000 7f28a2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f28b40cf470 con 0x564b9c48edb0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f28b40b4e00 con 0x564b9c48edb0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f28b40b2070 con 0x564b9c48edb0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f28b40106b0 con 0x564b9c48edb0 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 10 client.26360 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:11.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x564b9c49acd0 con 0x564b9c48edb0 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f28b40b2070 con 0x564b9c48edb0 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f28b409ad00 con 0x564b9c48edb0 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 10 client.26360 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 10 client.26360 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 20 client.26360 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c5059b0 0x564b9c507da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 4 client.26360 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x564b9c3f4dc0 con 0x564b9c5059b0 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.583+0000 7f28beffd640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c5059b0 0x564b9c507da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.584+0000 7f28a2ff5640 10 client.26360 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:11.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.584+0000 7f28beffd640 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c5059b0 0x564b9c507da0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2632bd4f01bc0ce server_cookie=b9c4cc31e7636832 in_seq=0 out_seq=0 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28a2ff5640 1 -- 172.21.15.69:0/509506946 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x564b9c3f4dc0 con 0x564b9c5059b0 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28a2ff5640 10 client.26360 handle_command_reply: tid=0 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 client.26360 shutdown 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 msgr2=0x564b9c497ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c497ff0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x564b9c49ba50 tx=0x7f28b40a49f0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.732 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 shutdown_connections 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x564b9c499000 0x564b9c499480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x564b9c48edb0 0x564b9c497ff0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x564b9c5059b0 0x564b9c507da0 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 --2- 172.21.15.69:0/509506946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x564b9c489250 0x564b9c497ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28a2ff5640 0 client.26360 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 >> 172.21.15.69:0/509506946 conn(0x564b9c4848c0 msgr2=0x564b9c4934c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 shutdown_connections 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 1 -- 172.21.15.69:0/509506946 wait complete. 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.732+0000 7f28c8383740 20 client.26360 trim_cache size 0 max 16384 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:31:11.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:31:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:31:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:31:11.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 msgr2=0x7f28a80a5120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 0x7f28a80a5120 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f28b8005e30 tx=0x7f28b802f390 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 msgr2=0x7f28c019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f28b400e9f0 tx=0x7f28b400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 shutdown_connections 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f28a80a2c60 0x7f28a80a5120 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f28c0102360 0x7f28c01a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f28c01019a0 0x7f28c019ebf0 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 --2- 172.21.15.69:0/2222200227 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f28c01007b0 0x7f28c006ab70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 >> 172.21.15.69:0/2222200227 conn(0x7f28c00fbd20 msgr2=0x7f28c0104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 shutdown_connections 2024-05-09T09:31:11.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:11.733+0000 7f28c5679640 1 -- 172.21.15.69:0/2222200227 wait complete. 2024-05-09T09:31:12.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 957129 8.74 MB/sec execute 550 sec latency 88.003 ms 2024-05-09T09:31:13.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:12 smithi069 ceph-mon[30123]: pgmap v739: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.3 MiB/s wr, 275 op/s 2024-05-09T09:31:13.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:31:13.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:12 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:12 smithi163 ceph-mon[34354]: pgmap v739: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.3 MiB/s wr, 275 op/s 2024-05-09T09:31:13.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:31:13.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:12 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:12 smithi103 ceph-mon[34421]: pgmap v739: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 517 KiB/s rd, 2.3 MiB/s wr, 275 op/s 2024-05-09T09:31:13.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:31:13.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:12 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:13.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 959535 8.75 MB/sec execute 551 sec latency 29.225 ms 2024-05-09T09:31:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:31:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:31:14.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:14.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:31:14.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:31:14.018 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:31:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:31:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:31:14.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 960815 8.75 MB/sec execute 552 sec latency 22.849 ms 2024-05-09T09:31:14.957 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:14 smithi163 ceph-mon[34354]: pgmap v740: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:31:15.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:14 smithi069 ceph-mon[30123]: pgmap v740: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:31:15.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:14 smithi103 ceph-mon[34421]: pgmap v740: 129 pgs: 129 active+clean; 159 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 2.9 MiB/s wr, 350 op/s 2024-05-09T09:31:15.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 963178 8.76 MB/sec execute 553 sec latency 10.857 ms 2024-05-09T09:31:16.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 964599 8.75 MB/sec execute 554 sec latency 62.757 ms 2024-05-09T09:31:17.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:16 smithi069 ceph-mon[30123]: pgmap v741: 129 pgs: 129 active+clean; 156 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.4 MiB/s wr, 319 op/s 2024-05-09T09:31:17.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:16 smithi163 ceph-mon[34354]: pgmap v741: 129 pgs: 129 active+clean; 156 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.4 MiB/s wr, 319 op/s 2024-05-09T09:31:17.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:16 smithi103 ceph-mon[34421]: pgmap v741: 129 pgs: 129 active+clean; 156 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.4 MiB/s wr, 319 op/s 2024-05-09T09:31:17.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 966693 8.76 MB/sec execute 555 sec latency 26.626 ms 2024-05-09T09:31:18.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 968439 8.75 MB/sec execute 556 sec latency 14.782 ms 2024-05-09T09:31:19.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:18 smithi069 ceph-mon[30123]: pgmap v742: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.8 MiB/s wr, 342 op/s 2024-05-09T09:31:19.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:18 smithi163 ceph-mon[34354]: pgmap v742: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.8 MiB/s wr, 342 op/s 2024-05-09T09:31:19.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:18 smithi103 ceph-mon[34421]: pgmap v742: 129 pgs: 129 active+clean; 158 MiB data, 3.4 GiB used, 1.0 TiB / 1.0 TiB avail; 651 KiB/s rd, 2.8 MiB/s wr, 342 op/s 2024-05-09T09:31:19.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 970371 8.76 MB/sec execute 557 sec latency 12.598 ms 2024-05-09T09:31:20.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 972373 8.76 MB/sec execute 558 sec latency 23.444 ms 2024-05-09T09:31:21.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:20 smithi069 ceph-mon[30123]: pgmap v743: 129 pgs: 129 active+clean; 158 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:31:21.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:20 smithi163 ceph-mon[34354]: pgmap v743: 129 pgs: 129 active+clean; 158 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:31:21.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:20 smithi103 ceph-mon[34421]: pgmap v743: 129 pgs: 129 active+clean; 158 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 512 KiB/s rd, 2.4 MiB/s wr, 296 op/s 2024-05-09T09:31:21.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 974013 8.76 MB/sec execute 559 sec latency 22.331 ms 2024-05-09T09:31:22.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 975672 8.76 MB/sec execute 560 sec latency 99.634 ms 2024-05-09T09:31:23.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:22 smithi069 ceph-mon[30123]: pgmap v744: 129 pgs: 129 active+clean; 161 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-05-09T09:31:23.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:22 smithi163 ceph-mon[34354]: pgmap v744: 129 pgs: 129 active+clean; 161 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-05-09T09:31:23.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:22 smithi103 ceph-mon[34421]: pgmap v744: 129 pgs: 129 active+clean; 161 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 730 KiB/s rd, 2.6 MiB/s wr, 345 op/s 2024-05-09T09:31:23.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 977462 8.77 MB/sec execute 561 sec latency 14.138 ms 2024-05-09T09:31:24.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 979508 8.77 MB/sec execute 562 sec latency 15.539 ms 2024-05-09T09:31:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 -- 172.21.15.69:0/711419126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0100790 msgr2=0x7f31b0100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 --2- 172.21.15.69:0/711419126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0100790 0x7f31b0100bb0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009900 tx=0x7f31a002a410 comp rx=0 tx=0).stop 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 -- 172.21.15.69:0/711419126 shutdown_connections 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 --2- 172.21.15.69:0/711419126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 0x7f31b0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 --2- 172.21.15.69:0/711419126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0101980 0x7f31b0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 --2- 172.21.15.69:0/711419126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0100790 0x7f31b0100bb0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 -- 172.21.15.69:0/711419126 >> 172.21.15.69:0/711419126 conn(0x7f31b00fbd20 msgr2=0x7f31b00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 -- 172.21.15.69:0/711419126 shutdown_connections 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.826+0000 7f31b48ca640 1 -- 172.21.15.69:0/711419126 wait complete. 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 Processor -- start 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 -- start start 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0100790 0x7f31b0197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 0x7f31b019f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31b006afc0 con 0x7f31b0101980 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31b006ae40 con 0x7f31b0102340 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31b48ca640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f31b010d6b0 con 0x7f31b0100790 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39662/0 (socket says 172.21.15.69:39662) 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 -- 172.21.15.69:0/3018778529 learned_addr learned my addr 172.21.15.69:0/3018778529 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31aed9d640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 0x7f31b019f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31ae59c640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0100790 0x7f31b0197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 -- 172.21.15.69:0/3018778529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0100790 msgr2=0x7f31b0197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0100790 0x7f31b0197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 -- 172.21.15.69:0/3018778529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 msgr2=0x7f31b019f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 0x7f31b019f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.827+0000 7f31add9b640 1 -- 172.21.15.69:0/3018778529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f31b019f6e0 con 0x7f31b0101980 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f31add9b640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f31a400d770 tx=0x7f31a400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f319f7fe640 1 -- 172.21.15.69:0/3018778529 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4017070 con 0x7f31b0101980 2024-05-09T09:31:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f319f7fe640 1 -- 172.21.15.69:0/3018778529 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a4005500 con 0x7f31b0101980 2024-05-09T09:31:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f31b019f9d0 con 0x7f31b0101980 2024-05-09T09:31:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f319f7fe640 1 -- 172.21.15.69:0/3018778529 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4004e00 con 0x7f31b0101980 2024-05-09T09:31:24.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.828+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f31b019fed0 con 0x7f31b0101980 2024-05-09T09:31:24.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f319f7fe640 1 -- 172.21.15.69:0/3018778529 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f31a4004fa0 con 0x7f31b0101980 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 Processor -- start 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 -- start start 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x56136611cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 0x56136611f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561366077340 con 0x561366117660 2024-05-09T09:31:24.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56136601f290 con 0x56136611d1c0 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31b75d6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561366073300 con 0x561366118050 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31aed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31aed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39678/0 (socket says 172.21.15.69:39678) 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31aed9d640 1 -- 172.21.15.69:0/1306541407 learned_addr learned my addr 172.21.15.69:0/1306541407 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31ae59c640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x56136611cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.831+0000 7f31add9b640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 0x56136611f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 -- 172.21.15.69:0/1306541407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 msgr2=0x56136611cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x56136611cb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 -- 172.21.15.69:0/1306541407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 msgr2=0x56136611f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 0x56136611f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 -- 172.21.15.69:0/1306541407 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5613660438e0 con 0x561366117660 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31add9b640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 0x56136611f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:24.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31ae59c640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x56136611cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f319f7fe640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 0x7f31980a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31ae59c640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 0x7f31980a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f31aed9d640 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009fd0 tx=0x7f31a80118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f254790d467ad94 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f319f7fe640 1 -- 172.21.15.69:0/3018778529 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f31a40ced40 con 0x7f31b0101980 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f319cff9640 1 -- 172.21.15.69:0/1306541407 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8004d20 con 0x561366117660 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f319cff9640 1 -- 172.21.15.69:0/1306541407 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a8004050 con 0x561366117660 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.832+0000 7f319cff9640 1 -- 172.21.15.69:0/1306541407 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8005240 con 0x561366117660 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- 172.21.15.69:0/1306541407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 msgr2=0x561366117a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31ae59c640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 0x7f31980a50f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f31a0005ab0 tx=0x7f31a0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009fd0 tx=0x7f31a80118d0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- 172.21.15.69:0/1306541407 shutdown_connections 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56136611d1c0 0x56136611f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x56136611cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 --2- 172.21.15.69:0/1306541407 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366117660 0x561366117a80 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- 172.21.15.69:0/1306541407 >> 172.21.15.69:0/1306541407 conn(0x56136610e8c0 msgr2=0x56136610ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- 172.21.15.69:0/1306541407 shutdown_connections 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- 172.21.15.69:0/1306541407 wait complete. 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.833+0000 7f31b75d6740 1 -- start start 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 Processor -- start 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366121dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x561366128e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x561366120080 con 0x561366118050 2024-05-09T09:31:24.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x561366032790 con 0x561366117660 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x561366120200 con 0x56136611d1c0 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31b75d6740 10 client.0 _ll_register_callbacks cb 0x56136606da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31ae59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366121dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31ae59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:34962/0 (socket says 172.21.15.69:34962) 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31ae59c640 1 -- 172.21.15.69:0/518260435 learned_addr learned my addr 172.21.15.69:0/518260435 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31aed9d640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x561366128e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 -- 172.21.15.69:0/518260435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 msgr2=0x561366128e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x561366128e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 -- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 msgr2=0x561366121880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 -- 172.21.15.69:0/518260435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561365d8b210 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31ae59c640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f31add9b640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366121dc0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f31a40cc510 tx=0x7f31a40048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82ee2620365bb718 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f3195ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.834+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4020050 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a8004050 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f31a4009cc0 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f31a40b9e60 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 10 client.17061 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x561365e57620 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a8004050 con 0x561366118050 2024-05-09T09:31:24.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f31a4097940 con 0x561366118050 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 10 client.17061 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 10 client.17061 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 20 client.17061 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56136612d440 0x56136612f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 4 client.17061 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x561366084a70 con 0x56136612d440 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.835+0000 7f31ae59c640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56136612d440 0x56136612f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.836+0000 7f3195ffb640 10 client.17061 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.836+0000 7f31ae59c640 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56136612d440 0x56136612f830 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=89cb5a71e412d3d7 server_cookie=b7fd8d216e85ba1a in_seq=0 out_seq=0 2024-05-09T09:31:24.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.836+0000 7f3195ffb640 1 -- 172.21.15.69:0/518260435 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x561366084a70 con 0x56136612d440 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.836+0000 7f3195ffb640 10 client.17061 handle_command_reply: tid=0 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.836+0000 7f31b75d6740 1 client.17061 shutdown 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 msgr2=0x561366121dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366121dc0 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f31a40cc510 tx=0x7f31a40048a0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 shutdown_connections 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56136612d440 0x56136612f830 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x561366128e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366121dc0 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 --2- 172.21.15.69:0/518260435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366121880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f3195ffb640 0 client.17061 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 >> 172.21.15.69:0/518260435 conn(0x56136610e8c0 msgr2=0x56136610da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:24.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 shutdown_connections 2024-05-09T09:31:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 1 -- 172.21.15.69:0/518260435 wait complete. 2024-05-09T09:31:24.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.837+0000 7f31b75d6740 20 client.17061 trim_cache size 0 max 16384 2024-05-09T09:31:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 Processor -- start 2024-05-09T09:31:24.857 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 -- start start 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366109620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x56136611c8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01aaa50 con 0x561366118050 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01a82c0 con 0x561366117660 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31b75d6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01a6110 con 0x56136611d1c0 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31ae59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366109620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39694/0 (socket says 172.21.15.69:39694) 2024-05-09T09:31:24.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 -- 172.21.15.69:0/2557838715 learned_addr learned my addr 172.21.15.69:0/2557838715 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31aed9d640 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x56136611c8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 -- 172.21.15.69:0/2557838715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 msgr2=0x56136611c8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x56136611c8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 -- 172.21.15.69:0/2557838715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 msgr2=0x561366109620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366109620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 -- 172.21.15.69:0/2557838715 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56136612fba0 con 0x561366118050 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.852+0000 7f31add9b640 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f31a4097ca0 tx=0x7f31a409f160 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb44e09b40dc0144 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f3197fff640 1 -- 172.21.15.69:0/2557838715 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a4020070 con 0x561366118050 2024-05-09T09:31:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f3197fff640 1 -- 172.21.15.69:0/2557838715 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a40d2510 con 0x561366118050 2024-05-09T09:31:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f3197fff640 1 -- 172.21.15.69:0/2557838715 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a409b2d0 con 0x561366118050 2024-05-09T09:31:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 -- 172.21.15.69:0/2557838715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 msgr2=0x561366128dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7f31a4097ca0 tx=0x7f31a409f160 comp rx=0 tx=0).stop 2024-05-09T09:31:24.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 -- 172.21.15.69:0/2557838715 shutdown_connections 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56136611d1c0 0x56136611c8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x561366118050 0x561366128dc0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 --2- 172.21.15.69:0/2557838715 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 -- 172.21.15.69:0/2557838715 >> 172.21.15.69:0/2557838715 conn(0x56136610e8c0 msgr2=0x5613661206b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 -- 172.21.15.69:0/2557838715 shutdown_connections 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.853+0000 7f31b75d6740 1 -- 172.21.15.69:0/2557838715 wait complete. 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 -- start start 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 Processor -- start 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366125de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x561366126320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01a6780 con 0x56136611d1c0 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01a7bf0 con 0x561366117660 2024-05-09T09:31:24.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f31b01a6e50 con 0x561366118050 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31b75d6740 10 client.0 _ll_register_callbacks cb 0x7f31a400d6c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39700/0 (socket says 172.21.15.69:39700) 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 -- 172.21.15.69:0/126371319 learned_addr learned my addr 172.21.15.69:0/126371319 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31ae59c640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366125de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31add9b640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x561366126320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 -- 172.21.15.69:0/126371319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 msgr2=0x561366126320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x561366126320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 -- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 msgr2=0x561366125de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:24.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366125de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.854+0000 7f31aed9d640 1 -- 172.21.15.69:0/126371319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x561366191ee0 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31aed9d640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f31a8006010 tx=0x7f31a8009d60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c3aacd1ef8ff88 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f31a8004d30 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f31a8004050 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f31a40d2510 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f31a8037050 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 10 client.17067 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x5613661931a0 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f31a40d2510 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f31a802c020 con 0x56136611d1c0 2024-05-09T09:31:24.863 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 10 client.17067 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 10 client.17067 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 20 client.17067 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5613661939e0 0x561366195e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 4 client.17067 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56136607edc0 con 0x5613661939e0 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.855+0000 7f31ae59c640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5613661939e0 0x561366195e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.856+0000 7f3195ffb640 10 client.17067 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:24.864 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:24.856+0000 7f31ae59c640 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5613661939e0 0x561366195e00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99f72af1fc6ea96d server_cookie=519685989be101de in_seq=0 out_seq=0 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.008+0000 7f3195ffb640 1 -- 172.21.15.69:0/126371319 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56136607edc0 con 0x5613661939e0 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.008+0000 7f3195ffb640 10 client.17067 handle_command_reply: tid=0 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.008+0000 7f31b75d6740 1 client.17067 shutdown 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.008+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 msgr2=0x5613661919a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.008+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7f31a8006010 tx=0x7f31a8009d60 comp rx=0 tx=0).stop 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 shutdown_connections 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56136611d1c0 0x5613661919a0 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5613661939e0 0x561366195e00 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x561366118050 0x561366126320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 --2- 172.21.15.69:0/126371319 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x561366117660 0x561366125de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 >> 172.21.15.69:0/126371319 conn(0x56136610e8c0 msgr2=0x5613661206b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f3195ffb640 0 client.17067 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 shutdown_connections 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 1 -- 172.21.15.69:0/126371319 wait complete. 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b75d6740 20 client.17067 trim_cache size 0 max 16384 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:31:25.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 msgr2=0x7f31980a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 0x7f31980a50f0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f31a0005ab0 tx=0x7f31a0005fd0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.009+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 msgr2=0x7f31b0198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f31a400d770 tx=0x7f31a400dc30 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 shutdown_connections 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31980a2c30 0x7f31980a50f0 secure :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f31a0005ab0 tx=0x7f31a0005fd0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f31b0102340 0x7f31b019f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f31b0101980 0x7f31b0198000 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 --2- 172.21.15.69:0/3018778529 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f31b0100790 0x7f31b0197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:25.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 >> 172.21.15.69:0/3018778529 conn(0x7f31b00fbd20 msgr2=0x7f31b0104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:25.013 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:24 smithi069 ceph-mon[30123]: pgmap v745: 129 pgs: 129 active+clean; 174 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-05-09T09:31:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 shutdown_connections 2024-05-09T09:31:25.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:25.010+0000 7f31b48ca640 1 -- 172.21.15.69:0/3018778529 wait complete. 2024-05-09T09:31:25.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:24 smithi163 ceph-mon[34354]: pgmap v745: 129 pgs: 129 active+clean; 174 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-05-09T09:31:25.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:24 smithi103 ceph-mon[34421]: pgmap v745: 129 pgs: 129 active+clean; 174 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 4.2 MiB/s wr, 359 op/s 2024-05-09T09:31:25.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 981008 8.77 MB/sec execute 563 sec latency 117.197 ms 2024-05-09T09:31:26.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 982982 8.77 MB/sec execute 564 sec latency 21.243 ms 2024-05-09T09:31:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:26 smithi069 ceph-mon[30123]: pgmap v746: 129 pgs: 129 active+clean; 180 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 4.1 MiB/s wr, 277 op/s 2024-05-09T09:31:27.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:26 smithi163 ceph-mon[34354]: pgmap v746: 129 pgs: 129 active+clean; 180 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 4.1 MiB/s wr, 277 op/s 2024-05-09T09:31:27.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:26 smithi103 ceph-mon[34421]: pgmap v746: 129 pgs: 129 active+clean; 180 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 4.1 MiB/s wr, 277 op/s 2024-05-09T09:31:27.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 984493 8.77 MB/sec execute 565 sec latency 60.489 ms 2024-05-09T09:31:28.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 986476 8.77 MB/sec execute 566 sec latency 12.868 ms 2024-05-09T09:31:29.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:28 smithi069 ceph-mon[30123]: pgmap v747: 129 pgs: 129 active+clean; 175 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-05-09T09:31:29.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:28 smithi163 ceph-mon[34354]: pgmap v747: 129 pgs: 129 active+clean; 175 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-05-09T09:31:29.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:28 smithi103 ceph-mon[34421]: pgmap v747: 129 pgs: 129 active+clean; 175 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 700 KiB/s rd, 4.5 MiB/s wr, 334 op/s 2024-05-09T09:31:29.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 988121 8.78 MB/sec execute 567 sec latency 17.451 ms 2024-05-09T09:31:30.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 990280 8.77 MB/sec execute 568 sec latency 15.615 ms 2024-05-09T09:31:31.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:30 smithi069 ceph-mon[30123]: pgmap v748: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 4.2 MiB/s wr, 323 op/s 2024-05-09T09:31:31.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:30 smithi163 ceph-mon[34354]: pgmap v748: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 4.2 MiB/s wr, 323 op/s 2024-05-09T09:31:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:30 smithi103 ceph-mon[34421]: pgmap v748: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 664 KiB/s rd, 4.2 MiB/s wr, 323 op/s 2024-05-09T09:31:31.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 991700 8.78 MB/sec execute 569 sec latency 31.165 ms 2024-05-09T09:31:32.507 INFO:tasks.workunit.client.0.smithi069.stdout: 1 993527 8.77 MB/sec execute 570 sec latency 86.285 ms 2024-05-09T09:31:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:32 smithi069 ceph-mon[30123]: pgmap v749: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-05-09T09:31:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:32 smithi163 ceph-mon[34354]: pgmap v749: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-05-09T09:31:33.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:32 smithi103 ceph-mon[34421]: pgmap v749: 129 pgs: 129 active+clean; 176 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 627 KiB/s rd, 3.9 MiB/s wr, 307 op/s 2024-05-09T09:31:33.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 995177 8.78 MB/sec execute 571 sec latency 17.436 ms 2024-05-09T09:31:34.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 997164 8.78 MB/sec execute 572 sec latency 13.334 ms 2024-05-09T09:31:35.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:34 smithi069 ceph-mon[30123]: pgmap v750: 129 pgs: 129 active+clean; 178 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-05-09T09:31:35.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:34 smithi163 ceph-mon[34354]: pgmap v750: 129 pgs: 129 active+clean; 178 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-05-09T09:31:35.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:34 smithi103 ceph-mon[34421]: pgmap v750: 129 pgs: 129 active+clean; 178 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 685 KiB/s rd, 4.4 MiB/s wr, 334 op/s 2024-05-09T09:31:35.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 998768 8.78 MB/sec execute 573 sec latency 13.649 ms 2024-05-09T09:31:36.510 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1000731 8.78 MB/sec execute 574 sec latency 21.613 ms 2024-05-09T09:31:37.568 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:36 smithi103 ceph-mon[34421]: pgmap v751: 129 pgs: 129 active+clean; 177 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-09T09:31:37.568 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:36 smithi163 ceph-mon[34354]: pgmap v751: 129 pgs: 129 active+clean; 177 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-09T09:31:37.568 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:36 smithi069 ceph-mon[30123]: pgmap v751: 129 pgs: 129 active+clean; 177 MiB data, 3.5 GiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 2.8 MiB/s wr, 306 op/s 2024-05-09T09:31:37.569 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1002311 8.79 MB/sec execute 575 sec latency 54.011 ms 2024-05-09T09:31:38.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 -- 172.21.15.69:0/730996034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 msgr2=0x7f312c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 --2- 172.21.15.69:0/730996034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c100bb0 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7f3128009900 tx=0x7f312802a410 comp rx=0 tx=0).stop 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 -- 172.21.15.69:0/730996034 shutdown_connections 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 --2- 172.21.15.69:0/730996034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c102340 0x7f312c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 --2- 172.21.15.69:0/730996034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c101980 0x7f312c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 --2- 172.21.15.69:0/730996034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c100bb0 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 -- 172.21.15.69:0/730996034 >> 172.21.15.69:0/730996034 conn(0x7f312c0fbd00 msgr2=0x7f312c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 -- 172.21.15.69:0/730996034 shutdown_connections 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.104+0000 7f3133c9e640 1 -- 172.21.15.69:0/730996034 wait complete. 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 Processor -- start 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 -- start start 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c101980 0x7f312c1980a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 0x7f312c19f110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f312c10c400 con 0x7f312c100790 2024-05-09T09:31:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f312c10c280 con 0x7f312c101980 2024-05-09T09:31:38.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3133c9e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f312c10c580 con 0x7f312c102340 2024-05-09T09:31:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52100/0 (socket says 172.21.15.69:52100) 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 -- 172.21.15.69:0/462539972 learned_addr learned my addr 172.21.15.69:0/462539972 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131239640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c101980 0x7f312c1980a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f313223b640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 0x7f312c19f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 -- 172.21.15.69:0/462539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 msgr2=0x7f312c19f110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 0x7f312c19f110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 -- 172.21.15.69:0/462539972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c101980 msgr2=0x7f312c1980a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c101980 0x7f312c1980a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 -- 172.21.15.69:0/462539972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f312c19f780 con 0x7f312c100790 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f313223b640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 0x7f312c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.105+0000 7f3131a3a640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3128031610 tx=0x7f3128004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.106+0000 7f3122ffd640 1 -- 172.21.15.69:0/462539972 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f312803f030 con 0x7f312c100790 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.106+0000 7f3122ffd640 1 -- 172.21.15.69:0/462539972 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3128003ad0 con 0x7f312c100790 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.106+0000 7f3122ffd640 1 -- 172.21.15.69:0/462539972 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3128035070 con 0x7f312c100790 2024-05-09T09:31:38.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.106+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f312c19fa10 con 0x7f312c100790 2024-05-09T09:31:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.106+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f312c19ff50 con 0x7f312c100790 2024-05-09T09:31:38.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.108+0000 7f3122ffd640 1 -- 172.21.15.69:0/462539972 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3128042030 con 0x7f312c100790 2024-05-09T09:31:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 Processor -- start 2024-05-09T09:31:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 -- start start 2024-05-09T09:31:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x55950728d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 0x559507290240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5595071ec340 con 0x559507288c40 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x559507194290 con 0x559507288250 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f31369ab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5595071e8300 con 0x55950728ddb0 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f3131a3a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x55950728d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f313223b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f313223b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37116/0 (socket says 172.21.15.69:37116) 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.109+0000 7f313223b640 1 -- 172.21.15.69:0/33254391 learned_addr learned my addr 172.21.15.69:0/33254391 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3131239640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 0x559507290240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3122ffd640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 0x7f31180a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 -- 172.21.15.69:0/33254391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 msgr2=0x559507290240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 0x559507290240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 -- 172.21.15.69:0/33254391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 msgr2=0x55950728d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x55950728d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 -- 172.21.15.69:0/33254391 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595071b88e0 con 0x559507288250 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3131239640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 0x559507290240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3131a3a640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x55950728d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 0x7f31180a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3122ffd640 1 -- 172.21.15.69:0/462539972 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f31280efa80 con 0x7f312c100790 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f313223b640 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3124010620 tx=0x7f3124010ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28a551abd8950b10 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3117fff640 1 -- 172.21.15.69:0/33254391 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124019070 con 0x559507288250 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3117fff640 1 -- 172.21.15.69:0/33254391 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3124022070 con 0x559507288250 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.110+0000 7f3117fff640 1 -- 172.21.15.69:0/33254391 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124011b80 con 0x559507288250 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 -- 172.21.15.69:0/33254391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 msgr2=0x559507288670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f3124010620 tx=0x7f3124010ae0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f313223b640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 0x7f31180a5170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3124021000 tx=0x7f312401f000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 -- 172.21.15.69:0/33254391 shutdown_connections 2024-05-09T09:31:38.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55950728ddb0 0x559507290240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x55950728d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 --2- 172.21.15.69:0/33254391 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288250 0x559507288670 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 -- 172.21.15.69:0/33254391 >> 172.21.15.69:0/33254391 conn(0x5595072838c0 msgr2=0x559507283cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 -- 172.21.15.69:0/33254391 shutdown_connections 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.111+0000 7f31369ab740 1 -- 172.21.15.69:0/33254391 wait complete. 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 -- start start 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 Processor -- start 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072924a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507299a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559507290d60 con 0x55950728ddb0 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5595071a7790 con 0x559507288c40 2024-05-09T09:31:38.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x559507290ee0 con 0x559507288250 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f31369ab740 10 client.0 _ll_register_callbacks cb 0x5595071e2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37128/0 (socket says 172.21.15.69:37128) 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f313223b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507299a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 -- 172.21.15.69:0/2727092482 learned_addr learned my addr 172.21.15.69:0/2727092482 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131a3a640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072924a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 -- 172.21.15.69:0/2727092482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 msgr2=0x5595072924a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072924a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 -- 172.21.15.69:0/2727092482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 msgr2=0x559507299a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.112+0000 7f3131239640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507299a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f3131239640 1 -- 172.21.15.69:0/2727092482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559506f00210 con 0x559507288c40 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f313223b640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507299a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:31:38.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f3131239640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f311c004510 tx=0x7f311c00ba60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e492bba3287262b6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c025060 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3124022070 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f311c003b20 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f311c003d20 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 10 client.26390 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559506fcc620 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3124022070 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f311c01f070 con 0x559507288c40 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 10 client.26390 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 10 client.26390 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 20 client.26390 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.113+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55950729de30 0x5595072a0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.114+0000 7f31369ab740 4 client.26390 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:31:38.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.114+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5595071f9a70 con 0x55950729de30 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.114+0000 7f3131a3a640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55950729de30 0x5595072a0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.114+0000 7f3131a3a640 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55950729de30 0x5595072a0270 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4fd6a979ecfb0d8 server_cookie=39728588b57072d4 in_seq=0 out_seq=0 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.114+0000 7f31157fa640 10 client.26390 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31157fa640 1 -- 172.21.15.69:0/2727092482 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5595071f9a70 con 0x55950729de30 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31157fa640 10 client.26390 handle_command_reply: tid=0 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 client.26390 shutdown 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 msgr2=0x5595072929e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f311c004510 tx=0x7f311c00ba60 comp rx=0 tx=0).stop 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 shutdown_connections 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507299a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x5595072929e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55950729de30 0x5595072a0270 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 --2- 172.21.15.69:0/2727092482 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072924a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31157fa640 0 client.26390 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:38.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 >> 172.21.15.69:0/2727092482 conn(0x5595072838c0 msgr2=0x559507282a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 shutdown_connections 2024-05-09T09:31:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 1 -- 172.21.15.69:0/2727092482 wait complete. 2024-05-09T09:31:38.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.115+0000 7f31369ab740 20 client.26390 trim_cache size 0 max 16384 2024-05-09T09:31:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 Processor -- start 2024-05-09T09:31:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 -- start start 2024-05-09T09:31:38.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x559507282170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55950728ddb0 0x5595072826b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1aaaa0 con 0x559507288c40 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1a8310 con 0x55950728ddb0 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f31369ab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1a6160 con 0x559507288250 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52140/0 (socket says 172.21.15.69:52140) 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 -- 172.21.15.69:0/2589015542 learned_addr learned my addr 172.21.15.69:0/2589015542 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f313223b640 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55950728ddb0 0x5595072826b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 -- 172.21.15.69:0/2589015542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 msgr2=0x559507282170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x559507282170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 -- 172.21.15.69:0/2589015542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55950728ddb0 msgr2=0x5595072826b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.130+0000 7f3131239640 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55950728ddb0 0x5595072826b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f3131239640 1 -- 172.21.15.69:0/2589015542 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5595072935d0 con 0x559507288c40 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f3131239640 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f311c001eb0 tx=0x7f311c01f670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8c6f6f3c9e247895 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31177fe640 1 -- 172.21.15.69:0/2589015542 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c003b20 con 0x559507288c40 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31177fe640 1 -- 172.21.15.69:0/2589015542 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f311c0054e0 con 0x559507288c40 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31177fe640 1 -- 172.21.15.69:0/2589015542 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f311c004a40 con 0x559507288c40 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 -- 172.21.15.69:0/2589015542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 msgr2=0x5595072999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7f311c001eb0 tx=0x7f311c01f670 comp rx=0 tx=0).stop 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 -- 172.21.15.69:0/2589015542 shutdown_connections 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55950728ddb0 0x5595072826b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x559507288c40 0x5595072999e0 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 --2- 172.21.15.69:0/2589015542 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x559507282170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.131+0000 7f31369ab740 1 -- 172.21.15.69:0/2589015542 >> 172.21.15.69:0/2589015542 conn(0x5595072838c0 msgr2=0x559507289ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- 172.21.15.69:0/2589015542 shutdown_connections 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- 172.21.15.69:0/2589015542 wait complete. 2024-05-09T09:31:38.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- start start 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 Processor -- start 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072971d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x559507297710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1a67d0 con 0x55950728ddb0 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1a7c40 con 0x559507288c40 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f312c1a6ea0 con 0x559507288250 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f31369ab740 10 client.0 _ll_register_callbacks cb 0x7f311c016b60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f313223b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f313223b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52148/0 (socket says 172.21.15.69:52148) 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f313223b640 1 -- 172.21.15.69:0/427648417 learned_addr learned my addr 172.21.15.69:0/427648417 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f3131239640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x559507297710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.132+0000 7f3131a3a640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072971d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 -- 172.21.15.69:0/427648417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 msgr2=0x5595072971d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072971d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 -- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 msgr2=0x559507297710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x559507297710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 -- 172.21.15.69:0/427648417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559507298d10 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f313223b640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3124010450 tx=0x7f31240116d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=129cd760238cb67e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3124035020 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f311c0054e0 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f31240118c0 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f312403d050 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 10 client.17085 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x559507304570 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f311c0054e0 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f3124031a30 con 0x55950728ddb0 2024-05-09T09:31:38.137 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 10 client.17085 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 10 client.17085 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 20 client.17085 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559507304e20 0x559507307240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 4 client.17085 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.133+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5595071f3dc0 con 0x559507304e20 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.134+0000 7f3131a3a640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559507304e20 0x559507307240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.134+0000 7f31157fa640 10 client.17085 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:38.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.134+0000 7f3131a3a640 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559507304e20 0x559507307240 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b918b27324936ea server_cookie=44da3d6d94861760 in_seq=0 out_seq=0 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31157fa640 1 -- 172.21.15.69:0/427648417 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5595071f3dc0 con 0x559507304e20 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31157fa640 10 client.17085 handle_command_reply: tid=0 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 client.17085 shutdown 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 msgr2=0x559507298720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7f3124010450 tx=0x7f31240116d0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 shutdown_connections 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55950728ddb0 0x559507298720 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x559507288c40 0x559507297710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559507304e20 0x559507307240 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 --2- 172.21.15.69:0/427648417 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x559507288250 0x5595072971d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31157fa640 0 client.17085 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 >> 172.21.15.69:0/427648417 conn(0x5595072838c0 msgr2=0x559507286640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 shutdown_connections 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 1 -- 172.21.15.69:0/427648417 wait complete. 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.259+0000 7f31369ab740 20 client.17085 trim_cache size 0 max 16384 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:31:38.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:31:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 msgr2=0x7f31180a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 0x7f31180a5170 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f3124021000 tx=0x7f312401f000 comp rx=0 tx=0).stop 2024-05-09T09:31:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 msgr2=0x7f312c197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7f3128031610 tx=0x7f3128004270 comp rx=0 tx=0).stop 2024-05-09T09:31:38.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 shutdown_connections 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31180a2cb0 0x7f31180a5170 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f312c102340 0x7f312c19f110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f312c101980 0x7f312c1980a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 --2- 172.21.15.69:0/462539972 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f312c100790 0x7f312c197b60 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 >> 172.21.15.69:0/462539972 conn(0x7f312c0fbd00 msgr2=0x7f312c0fd7b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 shutdown_connections 2024-05-09T09:31:38.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:38.260+0000 7f3133c9e640 1 -- 172.21.15.69:0/462539972 wait complete. 2024-05-09T09:31:38.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1004004 8.78 MB/sec execute 576 sec latency 67.012 ms 2024-05-09T09:31:39.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:38 smithi069 ceph-mon[30123]: pgmap v752: 129 pgs: 129 active+clean; 180 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-05-09T09:31:39.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:38 smithi163 ceph-mon[34354]: pgmap v752: 129 pgs: 129 active+clean; 180 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-05-09T09:31:39.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:38 smithi103 ceph-mon[34421]: pgmap v752: 129 pgs: 129 active+clean; 180 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 817 KiB/s rd, 2.7 MiB/s wr, 347 op/s 2024-05-09T09:31:39.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1005867 8.79 MB/sec execute 577 sec latency 12.186 ms 2024-05-09T09:31:40.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1007973 8.79 MB/sec execute 578 sec latency 12.438 ms 2024-05-09T09:31:41.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:40 smithi069 ceph-mon[30123]: pgmap v753: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.3 MiB/s wr, 302 op/s 2024-05-09T09:31:41.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:40 smithi163 ceph-mon[34354]: pgmap v753: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.3 MiB/s wr, 302 op/s 2024-05-09T09:31:41.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:40 smithi103 ceph-mon[34421]: pgmap v753: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.3 MiB/s wr, 302 op/s 2024-05-09T09:31:41.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1009562 8.79 MB/sec execute 579 sec latency 32.461 ms 2024-05-09T09:31:42.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1011541 8.79 MB/sec execute 580 sec latency 59.288 ms 2024-05-09T09:31:43.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:42 smithi069 ceph-mon[30123]: pgmap v754: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-05-09T09:31:43.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:42 smithi163 ceph-mon[34354]: pgmap v754: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-05-09T09:31:43.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:42 smithi103 ceph-mon[34421]: pgmap v754: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 663 KiB/s rd, 2.1 MiB/s wr, 263 op/s 2024-05-09T09:31:43.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1013039 8.79 MB/sec execute 581 sec latency 38.707 ms 2024-05-09T09:31:44.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1015274 8.80 MB/sec execute 582 sec latency 12.913 ms 2024-05-09T09:31:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:44 smithi069 ceph-mon[30123]: pgmap v755: 129 pgs: 129 active+clean; 179 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:31:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:44 smithi163 ceph-mon[34354]: pgmap v755: 129 pgs: 129 active+clean; 179 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:31:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:44 smithi103 ceph-mon[34421]: pgmap v755: 129 pgs: 129 active+clean; 179 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 791 KiB/s rd, 2.7 MiB/s wr, 333 op/s 2024-05-09T09:31:45.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1016733 8.80 MB/sec execute 583 sec latency 13.091 ms 2024-05-09T09:31:46.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1019350 8.81 MB/sec execute 584 sec latency 13.698 ms 2024-05-09T09:31:47.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:46 smithi069 ceph-mon[30123]: pgmap v756: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-05-09T09:31:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:46 smithi163 ceph-mon[34354]: pgmap v756: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-05-09T09:31:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:46 smithi103 ceph-mon[34421]: pgmap v756: 129 pgs: 129 active+clean; 181 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 704 KiB/s rd, 2.4 MiB/s wr, 287 op/s 2024-05-09T09:31:47.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1020264 8.80 MB/sec execute 585 sec latency 89.144 ms 2024-05-09T09:31:48.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1022156 8.80 MB/sec execute 586 sec latency 49.830 ms 2024-05-09T09:31:49.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:48 smithi069 ceph-mon[30123]: pgmap v757: 129 pgs: 129 active+clean; 182 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:49.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:48 smithi163 ceph-mon[34354]: pgmap v757: 129 pgs: 129 active+clean; 182 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:48 smithi103 ceph-mon[34421]: pgmap v757: 129 pgs: 129 active+clean; 182 MiB data, 3.6 GiB used, 1.0 TiB / 1.0 TiB avail; 749 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:49.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1023589 8.80 MB/sec execute 587 sec latency 144.499 ms 2024-05-09T09:31:50.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1025594 8.80 MB/sec execute 588 sec latency 17.818 ms 2024-05-09T09:31:51.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:50 smithi069 ceph-mon[30123]: pgmap v758: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 299 op/s 2024-05-09T09:31:51.057 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:50 smithi163 ceph-mon[34354]: pgmap v758: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 299 op/s 2024-05-09T09:31:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:50 smithi103 ceph-mon[34421]: pgmap v758: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 785 KiB/s rd, 2.4 MiB/s wr, 299 op/s 2024-05-09T09:31:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 -- 172.21.15.69:0/1680336952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce8102360 msgr2=0x7f2ce8108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/1680336952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce8102360 0x7f2ce8108bf0 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0009ca0 tx=0x7f2ce002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 -- 172.21.15.69:0/1680336952 shutdown_connections 2024-05-09T09:31:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/1680336952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce8102360 0x7f2ce8108bf0 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/1680336952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81019a0 0x7f2ce8101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/1680336952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce81007b0 0x7f2ce8100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 -- 172.21.15.69:0/1680336952 >> 172.21.15.69:0/1680336952 conn(0x7f2ce80fbd20 msgr2=0x7f2ce80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 -- 172.21.15.69:0/1680336952 shutdown_connections 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.349+0000 7f2ced7b7640 1 -- 172.21.15.69:0/1680336952 wait complete. 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 Processor -- start 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 -- start start 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 0x7f2ce806ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 0x7f2ce819ec80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce810c410 con 0x7f2ce81019a0 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce810c290 con 0x7f2ce81007b0 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2ced7b7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce810c590 con 0x7f2ce8102360 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2cde7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2cde7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56772/0 (socket says 172.21.15.69:56772) 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.350+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3651842207 learned_addr learned my addr 172.21.15.69:0/3651842207 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 0x7f2ce819ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 0x7f2ce806ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3651842207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 msgr2=0x7f2ce819ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 0x7f2ce819ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3651842207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 msgr2=0x7f2ce806ab90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 0x7f2ce806ab90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3651842207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2ce819f540 con 0x7f2ce81019a0 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 0x7f2ce819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 0x7f2ce806ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c920 tx=0x7f2cd000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce4ff9640 1 -- 172.21.15.69:0/3651842207 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0007c70 con 0x7f2ce81019a0 2024-05-09T09:31:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce4ff9640 1 -- 172.21.15.69:0/3651842207 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd0007e10 con 0x7f2ce81019a0 2024-05-09T09:31:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2ce819f830 con 0x7f2ce81019a0 2024-05-09T09:31:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ce4ff9640 1 -- 172.21.15.69:0/3651842207 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0018890 con 0x7f2ce81019a0 2024-05-09T09:31:51.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.351+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2ce819fd80 con 0x7f2ce81019a0 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.353+0000 7f2ce4ff9640 1 -- 172.21.15.69:0/3651842207 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2cd0018a70 con 0x7f2ce81019a0 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.353+0000 7f2cf05a2740 1 Processor -- start 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 -- start start 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e185240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556d6e0e1340 con 0x556d6e17dc40 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d6e089290 con 0x556d6e17d250 2024-05-09T09:31:51.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556d6e0dd300 con 0x556d6e182db0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce4ff9640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 0x7f2cd40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56782/0 (socket says 172.21.15.69:56782) 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/4060894058 learned_addr learned my addr 172.21.15.69:0/4060894058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 0x7f2cd40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce4ff9640 1 -- 172.21.15.69:0/3651842207 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f2cd00cf5c0 con 0x7f2ce81019a0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e185240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/4060894058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 msgr2=0x556d6e185240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e185240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/4060894058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 msgr2=0x556d6e17d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.354+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/4060894058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6e0ad8e0 con 0x556d6e17dc40 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800bbf0 tx=0x7f2cd800d710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=afb7b074ca67a241 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 0x7f2cd40a50f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0005cc0 tx=0x7f2ce002a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2cdd7fa640 1 -- 172.21.15.69:0/4060894058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8004220 con 0x556d6e17dc40 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2cdd7fa640 1 -- 172.21.15.69:0/4060894058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd80043c0 con 0x556d6e17dc40 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2cdd7fa640 1 -- 172.21.15.69:0/4060894058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd800f840 con 0x556d6e17dc40 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4060894058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 msgr2=0x556d6e182740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.355+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800bbf0 tx=0x7f2cd800d710 comp rx=0 tx=0).stop 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4060894058 shutdown_connections 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e185240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e182740 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4060894058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4060894058 >> 172.21.15.69:0/4060894058 conn(0x556d6e1788c0 msgr2=0x556d6e178cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4060894058 shutdown_connections 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4060894058 wait complete. 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- start start 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 Processor -- start 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e187970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556d6e185d60 con 0x556d6e17d250 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556d6e09c790 con 0x556d6e17dc40 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556d6e185ee0 con 0x556d6e182db0 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2cf05a2740 10 client.0 _ll_register_callbacks cb 0x556d6e0d7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2ce6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2ce6ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56788/0 (socket says 172.21.15.69:56788) 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.356+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/1634033449 learned_addr learned my addr 172.21.15.69:0/1634033449 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e187970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/1634033449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 msgr2=0x556d6e18e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 msgr2=0x556d6e187970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e187970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 -- 172.21.15.69:0/1634033449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6ddf5210 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800b390 tx=0x7f2cd8004740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69c3e7b552344e0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd8024020 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f2cd80035f0 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd80043c0 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2cd802f050 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cf05a2740 10 client.17097 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x556d6dec1620 con 0x556d6e17d250 2024-05-09T09:31:51.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.357+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd80043c0 con 0x556d6e17d250 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f2cd8017430 con 0x556d6e17d250 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 10 client.17097 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 10 client.17097 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 20 client.17097 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e192bc0 0x556d6e194fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 4 client.17097 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:31:51.369 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.358+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556d6e0eea70 con 0x556d6e192bc0 2024-05-09T09:31:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.377+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e192bc0 0x556d6e194fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.377+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e192bc0 0x556d6e194fb0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a0264e2ae515093f server_cookie=acfff88ffe324845 in_seq=0 out_seq=0 2024-05-09T09:31:51.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.377+0000 7f2cce7fc640 10 client.17097 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cce7fc640 1 -- 172.21.15.69:0/1634033449 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556d6e0eea70 con 0x556d6e192bc0 2024-05-09T09:31:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cce7fc640 10 client.17097 handle_command_reply: tid=0 2024-05-09T09:31:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 client.17097 shutdown 2024-05-09T09:31:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 msgr2=0x556d6e187430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f2cd800b390 tx=0x7f2cd8004740 comp rx=0 tx=0).stop 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 shutdown_connections 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e192bc0 0x556d6e194fb0 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e187970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/1634033449 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17d250 0x556d6e187430 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cce7fc640 0 client.17097 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 >> 172.21.15.69:0/1634033449 conn(0x556d6e1788c0 msgr2=0x556d6e177a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 shutdown_connections 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 1 -- 172.21.15.69:0/1634033449 wait complete. 2024-05-09T09:31:51.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.378+0000 7f2cf05a2740 20 client.17097 trim_cache size 0 max 16384 2024-05-09T09:31:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 Processor -- start 2024-05-09T09:31:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 -- start start 2024-05-09T09:31:51.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17bd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81acbc0 con 0x556d6e17dc40 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81aa430 con 0x556d6e17d250 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81a8280 con 0x556d6e182db0 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56792/0 (socket says 172.21.15.69:56792) 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3049145765 learned_addr learned my addr 172.21.15.69:0/3049145765 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17bd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3049145765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 msgr2=0x556d6e18e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3049145765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 msgr2=0x556d6e17bd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17bd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.393+0000 7f2cde7fc640 1 -- 172.21.15.69:0/3049145765 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556d6e173bc0 con 0x556d6e17dc40 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00ccd40 tx=0x7f2cd0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8aa8a1f0df9f7c70 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cdcff9640 1 -- 172.21.15.69:0/3049145765 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd00b3070 con 0x556d6e17dc40 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cdcff9640 1 -- 172.21.15.69:0/3049145765 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd009fe50 con 0x556d6e17dc40 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cdcff9640 1 -- 172.21.15.69:0/3049145765 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2cd0098110 con 0x556d6e17dc40 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 -- 172.21.15.69:0/3049145765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 msgr2=0x556d6e17c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f2cd00ccd40 tx=0x7f2cd0000bc0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 -- 172.21.15.69:0/3049145765 shutdown_connections 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e182db0 0x556d6e18e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e17dc40 0x556d6e17c290 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/3049145765 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17d250 0x556d6e17bd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.394+0000 7f2cf05a2740 1 -- 172.21.15.69:0/3049145765 >> 172.21.15.69:0/3049145765 conn(0x556d6e1788c0 msgr2=0x556d6e17b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- 172.21.15.69:0/3049145765 shutdown_connections 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- 172.21.15.69:0/3049145765 wait complete. 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- start start 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 Processor -- start 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e17d250 0x556d6e18ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e18bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81a88f0 con 0x556d6e182db0 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81a9d60 con 0x556d6e17dc40 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2ce81a8fc0 con 0x556d6e17d250 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cf05a2740 10 client.0 _ll_register_callbacks cb 0x7f2ce0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2ce77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2ce77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56798/0 (socket says 172.21.15.69:56798) 2024-05-09T09:31:51.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2ce77fe640 1 -- 172.21.15.69:0/4065634971 learned_addr learned my addr 172.21.15.69:0/4065634971 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2cde7fc640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e18bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e17d250 0x556d6e18ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.395+0000 7f2ce77fe640 1 -- 172.21.15.69:0/4065634971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e17d250 msgr2=0x556d6e18ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e17d250 0x556d6e18ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2ce77fe640 1 -- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 msgr2=0x556d6e18bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e18bfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2ce77fe640 1 -- 172.21.15.69:0/4065634971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556d6e195320 con 0x556d6e182db0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2ce77fe640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0035000 tx=0x7f2ce00095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=17e0623e3922bbb6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2ce0004510 con 0x556d6e182db0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f2ce0046070 con 0x556d6e182db0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2cd009fe50 con 0x556d6e182db0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=42) ==== 24+0+0 (secure 0 0 0) 0x7f2ce00046b0 con 0x556d6e182db0 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 10 client.17103 fetch_fsmap learned FSMap version 42 2024-05-09T09:31:51.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=42,monmap=4+}) -- 0x556d6e18e780 con 0x556d6e182db0 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2cd009fe50 con 0x556d6e182db0 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 42) ==== 8358+0+0 (secure 0 0 0) 0x7f2ce0030890 con 0x556d6e182db0 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 10 client.17103 fetch_fsmap finished waiting for FSMap version 42 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 10 client.17103 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 20 client.17103 populate_metadata read hostname 'smithi069' 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e1f9470 0x556d6e1fb890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 4 client.17103 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.396+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x556d6e0e8dc0 con 0x556d6e1f9470 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.397+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e1f9470 0x556d6e1fb890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.397+0000 7f2ce6ffd640 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e1f9470 0x556d6e1fb890 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bd4ceff5171e8bd server_cookie=ee3579623896448 in_seq=0 out_seq=0 2024-05-09T09:31:51.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.397+0000 7f2cce7fc640 10 client.17103 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:51.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1027096 8.80 MB/sec execute 589 sec latency 64.348 ms 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cce7fc640 1 -- 172.21.15.69:0/4065634971 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x556d6e0e8dc0 con 0x556d6e1f9470 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cce7fc640 10 client.17103 handle_command_reply: tid=0 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 client.17103 shutdown 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 msgr2=0x556d6e18cfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0035000 tx=0x7f2ce00095c0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 shutdown_connections 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556d6e182db0 0x556d6e18cfe0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556d6e17dc40 0x556d6e18bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x556d6e1f9470 0x556d6e1fb890 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 --2- 172.21.15.69:0/4065634971 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556d6e17d250 0x556d6e18ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cce7fc640 0 client.17103 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 >> 172.21.15.69:0/4065634971 conn(0x556d6e1788c0 msgr2=0x556d6e17bed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 shutdown_connections 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 1 -- 172.21.15.69:0/4065634971 wait complete. 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.555+0000 7f2cf05a2740 20 client.17103 trim_cache size 0 max 16384 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:31:51.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:31:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 msgr2=0x7f2cd40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 0x7f2cd40a50f0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f2ce0005cc0 tx=0x7f2ce002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 msgr2=0x7f2ce819c640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:31:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7f2cd000c920 tx=0x7f2cd000cde0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 shutdown_connections 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2cd40a2c30 0x7f2cd40a50f0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2ce8102360 0x7f2ce819ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2ce81019a0 0x7f2ce819c640 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 --2- 172.21.15.69:0/3651842207 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2ce81007b0 0x7f2ce806ab90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 >> 172.21.15.69:0/3651842207 conn(0x7f2ce80fbd20 msgr2=0x7f2ce8104bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 shutdown_connections 2024-05-09T09:31:51.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:31:51.556+0000 7f2ced7b7640 1 -- 172.21.15.69:0/3651842207 wait complete. 2024-05-09T09:31:52.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1028383 8.80 MB/sec execute 590 sec latency 106.010 ms 2024-05-09T09:31:53.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:52 smithi069 ceph-mon[30123]: pgmap v759: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-05-09T09:31:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:52 smithi163 ceph-mon[34354]: pgmap v759: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-05-09T09:31:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:52 smithi103 ceph-mon[34421]: pgmap v759: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 645 KiB/s rd, 2.3 MiB/s wr, 246 op/s 2024-05-09T09:31:53.508 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1030500 8.81 MB/sec execute 591 sec latency 26.136 ms 2024-05-09T09:31:54.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1031840 8.80 MB/sec execute 592 sec latency 13.667 ms 2024-05-09T09:31:55.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:54 smithi069 ceph-mon[30123]: pgmap v760: 129 pgs: 129 active+clean; 184 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:31:55.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:54 smithi163 ceph-mon[34354]: pgmap v760: 129 pgs: 129 active+clean; 184 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:31:55.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:54 smithi103 ceph-mon[34421]: pgmap v760: 129 pgs: 129 active+clean; 184 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 851 KiB/s rd, 2.9 MiB/s wr, 356 op/s 2024-05-09T09:31:55.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1034055 8.81 MB/sec execute 593 sec latency 14.715 ms 2024-05-09T09:31:56.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1035308 8.80 MB/sec execute 594 sec latency 14.245 ms 2024-05-09T09:31:57.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:56 smithi069 ceph-mon[30123]: pgmap v761: 129 pgs: 129 active+clean; 180 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:57.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:56 smithi163 ceph-mon[34354]: pgmap v761: 129 pgs: 129 active+clean; 180 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:57.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:56 smithi103 ceph-mon[34421]: pgmap v761: 129 pgs: 129 active+clean; 180 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 788 KiB/s rd, 2.5 MiB/s wr, 311 op/s 2024-05-09T09:31:57.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1037550 8.81 MB/sec execute 595 sec latency 45.838 ms 2024-05-09T09:31:58.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1038300 8.80 MB/sec execute 596 sec latency 40.383 ms 2024-05-09T09:31:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:31:58 smithi069 ceph-mon[30123]: pgmap v762: 129 pgs: 129 active+clean; 181 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-05-09T09:31:59.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:31:58 smithi163 ceph-mon[34354]: pgmap v762: 129 pgs: 129 active+clean; 181 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-05-09T09:31:59.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:31:58 smithi103 ceph-mon[34421]: pgmap v762: 129 pgs: 129 active+clean; 181 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.4 MiB/s wr, 311 op/s 2024-05-09T09:31:59.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1040928 8.81 MB/sec execute 597 sec latency 19.979 ms 2024-05-09T09:32:00.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1041757 8.80 MB/sec execute 598 sec latency 18.312 ms 2024-05-09T09:32:01.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:00 smithi069 ceph-mon[30123]: pgmap v763: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-05-09T09:32:01.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:00 smithi163 ceph-mon[34354]: pgmap v763: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-05-09T09:32:01.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:00 smithi103 ceph-mon[34421]: pgmap v763: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 880 KiB/s rd, 2.5 MiB/s wr, 328 op/s 2024-05-09T09:32:01.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 1044489 8.81 MB/sec execute 599 sec latency 13.046 ms 2024-05-09T09:32:02.509 INFO:tasks.workunit.client.0.smithi069.stdout: 1 cleanup 600 sec 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: 0 cleanup 600 sec 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: Operation Count AvgLat MaxLat 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: ---------------------------------------- 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: NTCreateX 168339 1.804 4978.810 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: Close 123658 0.002 0.102 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: Rename 7129 5.011 83.219 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: Unlink 33992 1.902 144.494 2024-05-09T09:32:02.717 INFO:tasks.workunit.client.0.smithi069.stdout: Deltree 4 79.112 122.102 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Mkdir 2 0.004 0.005 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Qpathinfo 152553 0.839 117.190 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Qfileinfo 26730 0.001 0.053 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Qfsinfo 27982 0.130 30.202 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Sfileinfo 13722 0.020 2.912 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Find 58991 0.156 99.626 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: WriteX 83907 0.015 40.420 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: ReadX 263874 0.009 28.627 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: LockX 548 0.747 11.848 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: UnlockX 548 0.711 9.079 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: Flush 11812 4.089 83.844 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:32:02.718 INFO:tasks.workunit.client.0.smithi069.stdout:Throughput 8.81495 MB/sec 1 clients 1 procs max_latency=4978.819 ms 2024-05-09T09:32:02.719 INFO:tasks.workunit.client.0.smithi069.stderr:+ dbench 10 2024-05-09T09:32:02.719 INFO:tasks.workunit.client.0.smithi069.stdout:dbench version 4.00 - Copyright Andrew Tridgell 1999-2004 2024-05-09T09:32:02.719 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:32:02.719 INFO:tasks.workunit.client.0.smithi069.stdout:Running for 600 seconds with load '/usr/share/dbench/client.txt' and minimum warmup 120 secs 2024-05-09T09:32:02.720 INFO:tasks.workunit.client.0.smithi069.stdout:8 of 10 processes prepared for launch 0 sec 2024-05-09T09:32:02.818 INFO:tasks.workunit.client.0.smithi069.stdout:10 of 10 processes prepared for launch 0 sec 2024-05-09T09:32:02.819 INFO:tasks.workunit.client.0.smithi069.stdout:releasing clients 2024-05-09T09:32:03.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:02 smithi069 ceph-mon[30123]: pgmap v764: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-05-09T09:32:03.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:02 smithi163 ceph-mon[34354]: pgmap v764: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-05-09T09:32:03.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:02 smithi103 ceph-mon[34421]: pgmap v764: 129 pgs: 129 active+clean; 183 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 642 KiB/s rd, 2.3 MiB/s wr, 277 op/s 2024-05-09T09:32:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: 10 319 116.40 MB/sec warmup 1 sec latency 77.674 ms 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.658+0000 7f7f85c66640 1 -- 172.21.15.69:0/2403575090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80103ac0 msgr2=0x7f7f80103ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.658+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2403575090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80103ac0 0x7f7f80103ee0 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f7f70009900 tx=0x7f7f7002a410 comp rx=0 tx=0).stop 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 -- 172.21.15.69:0/2403575090 shutdown_connections 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2403575090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80105670 0x7f7f8010bf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2403575090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f80105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2403575090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80103ac0 0x7f7f80103ee0 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 -- 172.21.15.69:0/2403575090 >> 172.21.15.69:0/2403575090 conn(0x7f7f800ff050 msgr2=0x7f7f80101470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 -- 172.21.15.69:0/2403575090 shutdown_connections 2024-05-09T09:32:04.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.659+0000 7f7f85c66640 1 -- 172.21.15.69:0/2403575090 wait complete. 2024-05-09T09:32:04.662 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.660+0000 7f7f85c66640 1 Processor -- start 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.660+0000 7f7f85c66640 1 -- start start 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80103ac0 0x7f7f80199d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80105670 0x7f7f801a1280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010f710 con 0x7f7f80105670 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010f590 con 0x7f7f80103ac0 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f85c66640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f8010f890 con 0x7f7f80104cb0 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f7effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f7effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39802/0 (socket says 172.21.15.69:39802) 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f7effd640 1 -- 172.21.15.69:0/2121171491 learned_addr learned my addr 172.21.15.69:0/2121171491 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80103ac0 0x7f7f80199d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.661+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80105670 0x7f7f801a1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 -- 172.21.15.69:0/2121171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80103ac0 msgr2=0x7f7f80199d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80103ac0 0x7f7f80199d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 -- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80105670 msgr2=0x7f7f801a1280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80105670 0x7f7f801a1280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 -- 172.21.15.69:0/2121171491 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f801a1920 con 0x7f7f80104cb0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400e9f0 tx=0x7f7f7400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7cff9640 1 -- 172.21.15.69:0/2121171491 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7400cd80 con 0x7f7f80104cb0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7cff9640 1 -- 172.21.15.69:0/2121171491 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f74010910 con 0x7f7f80104cb0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f7cff9640 1 -- 172.21.15.69:0/2121171491 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74018990 con 0x7f7f80104cb0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.662+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f801a1c10 con 0x7f7f80104cb0 2024-05-09T09:32:04.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.663+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f801a20e0 con 0x7f7f80104cb0 2024-05-09T09:32:04.666 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.666+0000 7f7f7cff9640 1 -- 172.21.15.69:0/2121171491 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f74010ac0 con 0x7f7f80104cb0 2024-05-09T09:32:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.667+0000 7f7f7cff9640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 0x7f7f680a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.667+0000 7f7f7cff9640 1 -- 172.21.15.69:0/2121171491 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f740cf6b0 con 0x7f7f80104cb0 2024-05-09T09:32:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.667+0000 7f7f88970740 1 Processor -- start 2024-05-09T09:32:04.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.667+0000 7f7f88970740 1 -- start start 2024-05-09T09:32:04.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.667+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 0x7f7f680a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 0x7f7f680a52e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005ab0 tx=0x7f7f70000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e6d740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39814/0 (socket says 172.21.15.69:39814) 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e70240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8dcc340 con 0x55c3b8e68c40 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8d74290 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8dc8300 con 0x55c3b8e68250 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 -- 172.21.15.69:0/3591965156 learned_addr learned my addr 172.21.15.69:0/3591965156 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7effd640 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e6d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e70240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 -- 172.21.15.69:0/3591965156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 msgr2=0x55c3b8e70240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e70240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 -- 172.21.15.69:0/3591965156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 msgr2=0x55c3b8e6d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.668+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e6d740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.669+0000 7f7f7f7fe640 1 -- 172.21.15.69:0/3591965156 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b8d988e0 con 0x55c3b8e68250 2024-05-09T09:32:04.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.669+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f70033000 tx=0x7f7f7003d000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f3f8c037d0480494 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.669+0000 7f7f65ffb640 1 -- 172.21.15.69:0/3591965156 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7004d070 con 0x55c3b8e68250 2024-05-09T09:32:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.669+0000 7f7f65ffb640 1 -- 172.21.15.69:0/3591965156 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f70005520 con 0x55c3b8e68250 2024-05-09T09:32:04.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.669+0000 7f7f65ffb640 1 -- 172.21.15.69:0/3591965156 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f70035040 con 0x55c3b8e68250 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 msgr2=0x55c3b8e68670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f7f70033000 tx=0x7f7f7003d000 comp rx=0 tx=0).stop 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 shutdown_connections 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e70240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e6d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 --2- 172.21.15.69:0/3591965156 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e68670 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 >> 172.21.15.69:0/3591965156 conn(0x55c3b8e638c0 msgr2=0x55c3b8e63cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.670+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 shutdown_connections 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.671+0000 7f7f88970740 1 -- 172.21.15.69:0/3591965156 wait complete. 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.671+0000 7f7f88970740 1 -- start start 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.671+0000 7f7f88970740 1 Processor -- start 2024-05-09T09:32:04.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.671+0000 7f7f88970740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e72430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e799b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8e70d00 con 0x55c3b8e68c40 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8d87790 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55c3b8e70e80 con 0x55c3b8e68250 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f88970740 10 client.0 _ll_register_callbacks cb 0x55c3b8dc2a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e72430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52084/0 (socket says 172.21.15.69:52084) 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 -- 172.21.15.69:0/1553651889 learned_addr learned my addr 172.21.15.69:0/1553651889 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 -- 172.21.15.69:0/1553651889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 msgr2=0x55c3b8e72430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e72430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e799b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 -- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 msgr2=0x55c3b8e799b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e799b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 -- 172.21.15.69:0/1553651889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b8ae0210 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f7effd640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7f74098520 tx=0x7f7f7409bbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d42bb693b319913 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.672+0000 7f7f5effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f7409ce60 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f74018d30 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f70005520 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7f7409b030 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 10 client.17115 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c3b8bac620 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7409b1d0 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f7f7409aaa0 con 0x55c3b8e68c40 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 10 client.17115 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:04.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 10 client.17115 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 20 client.17115 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8e7e010 0x55c3b8e80470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 4 client.17115 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.673+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f7f70005520 con 0x55c3b8e7e010 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.674+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8e7e010 0x55c3b8e80470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.677 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.674+0000 7f7f5effd640 10 client.17115 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.674+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8e7e010 0x55c3b8e80470 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66c2aa2051553ee6 server_cookie=f563bb5df7b91104 in_seq=0 out_seq=0 2024-05-09T09:32:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.677+0000 7f7f5effd640 1 -- 172.21.15.69:0/1553651889 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f7f70005520 con 0x55c3b8e7e010 2024-05-09T09:32:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.677+0000 7f7f5effd640 10 client.17115 handle_command_reply: tid=0 2024-05-09T09:32:04.678 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.677+0000 7f7f88970740 1 client.17115 shutdown 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 msgr2=0x55c3b8e72970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f7f74098520 tx=0x7f7f7409bbe0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 shutdown_connections 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e799b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68c40 0x55c3b8e72970 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8e7e010 0x55c3b8e80470 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 --2- 172.21.15.69:0/1553651889 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e72430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 >> 172.21.15.69:0/1553651889 conn(0x55c3b8e638c0 msgr2=0x55c3b8e62a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:04.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f5effd640 0 client.17115 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 shutdown_connections 2024-05-09T09:32:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 1 -- 172.21.15.69:0/1553651889 wait complete. 2024-05-09T09:32:04.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.678+0000 7f7f88970740 20 client.17115 trim_cache size 0 max 16384 2024-05-09T09:32:04.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 Processor -- start 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 -- start start 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e79970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801aee60 con 0x55c3b8e68250 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801ac6a0 con 0x55c3b8e68c40 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f88970740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801aa4f0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52090/0 (socket says 172.21.15.69:52090) 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7f7fe640 1 -- 172.21.15.69:0/2885998579 learned_addr learned my addr 172.21.15.69:0/2885998579 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e79970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 -- 172.21.15.69:0/2885998579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e6ddb0 msgr2=0x55c3b8e79970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e79970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 -- 172.21.15.69:0/2885998579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 msgr2=0x55c3b8e85d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7effd640 1 -- 172.21.15.69:0/2885998579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55c3b8e6d140 con 0x55c3b8e68c40 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.695+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.696+0000 7f7f7effd640 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e862c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f7409b8e0 tx=0x7f7f74018420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=761fb448de6d3983 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.696+0000 7f7f657fa640 1 -- 172.21.15.69:0/2885998579 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f740a1410 con 0x55c3b8e68c40 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.696+0000 7f7f657fa640 1 -- 172.21.15.69:0/2885998579 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7409aaa0 con 0x55c3b8e68c40 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.696+0000 7f7f657fa640 1 -- 172.21.15.69:0/2885998579 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f74018d90 con 0x55c3b8e68c40 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- 172.21.15.69:0/2885998579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 msgr2=0x55c3b8e862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e862c0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f7f7409b8e0 tx=0x7f7f74018420 comp rx=0 tx=0).stop 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- 172.21.15.69:0/2885998579 shutdown_connections 2024-05-09T09:32:04.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e79970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e862c0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- 172.21.15.69:0/2885998579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e68250 0x55c3b8e85d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- 172.21.15.69:0/2885998579 >> 172.21.15.69:0/2885998579 conn(0x55c3b8e638c0 msgr2=0x55c3b8e72430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- 172.21.15.69:0/2885998579 shutdown_connections 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- 172.21.15.69:0/2885998579 wait complete. 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- start start 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 Processor -- start 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e77090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e775d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801aab60 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801abfd0 con 0x55c3b8e68c40 2024-05-09T09:32:04.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.697+0000 7f7f88970740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f801ab230 con 0x55c3b8e68250 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f88970740 10 client.0 _ll_register_callbacks cb 0x7f7f7409b9b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e77090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e775d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52098/0 (socket says 172.21.15.69:52098) 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e77090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39846/0 (socket says 172.21.15.69:39846) 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 -- 172.21.15.69:0/1959949463 learned_addr learned my addr 172.21.15.69:0/1959949463 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 -- 172.21.15.69:0/1959949463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 msgr2=0x55c3b8e77090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e77090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 -- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 msgr2=0x55c3b8e775d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e775d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 -- 172.21.15.69:0/1959949463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55c3b8e78bd0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f7ffff640 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7f78004070 tx=0x7f7f78004530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a75f785e3cfba4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f5effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f78010cc0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f7409aaa0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f7801c3a0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f7f78010e60 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f88970740 10 client.17121 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.698+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55c3b8ee4650 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f7409aaa0 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f7f7801d660 con 0x55c3b8e6ddb0 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 10 client.17121 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 10 client.17121 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 20 client.17121 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8ee5110 0x55c3b8ee7530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 4 client.17121 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:32:04.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55c3b8dd9a70 con 0x55c3b8ee5110 2024-05-09T09:32:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8ee5110 0x55c3b8ee7530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f5effd640 10 client.17121 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:04.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:04.699+0000 7f7f7f7fe640 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8ee5110 0x55c3b8ee7530 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdbc6251bec73184 server_cookie=40fc1dd54c25e7d3 in_seq=0 out_seq=0 2024-05-09T09:32:04.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 681 103.83 MB/sec warmup 2 sec latency 104.339 ms 2024-05-09T09:32:05.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:04 smithi069 ceph-mon[30123]: pgmap v765: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-05-09T09:32:05.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:04 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:32:05.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:04 smithi163 ceph-mon[34354]: pgmap v765: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-05-09T09:32:05.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:04 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:32:05.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:04 smithi103 ceph-mon[34421]: pgmap v765: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 863 KiB/s rd, 2.9 MiB/s wr, 346 op/s 2024-05-09T09:32:05.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:04 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:32:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f5effd640 1 -- 172.21.15.69:0/1959949463 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55c3b8dd9a70 con 0x55c3b8ee5110 2024-05-09T09:32:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f5effd640 10 client.17121 handle_command_reply: tid=0 2024-05-09T09:32:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 client.17121 shutdown 2024-05-09T09:32:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 msgr2=0x55c3b8e785e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:05.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f7f78004070 tx=0x7f7f78004530 comp rx=0 tx=0).stop 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 shutdown_connections 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55c3b8e6ddb0 0x55c3b8e785e0 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55c3b8e68c40 0x55c3b8e775d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55c3b8ee5110 0x55c3b8ee7530 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 --2- 172.21.15.69:0/1959949463 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55c3b8e68250 0x55c3b8e77090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f5effd640 0 client.17121 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 >> 172.21.15.69:0/1959949463 conn(0x55c3b8e638c0 msgr2=0x55c3b8e72340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 shutdown_connections 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 1 -- 172.21.15.69:0/1959949463 wait complete. 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.238+0000 7f7f88970740 20 client.17121 trim_cache size 0 max 16384 2024-05-09T09:32:05.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:32:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:32:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:32:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:32:05.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 msgr2=0x7f7f680a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 0x7f7f680a52e0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f7f70005ab0 tx=0x7f7f70000bc0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 msgr2=0x7f7f8019a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f7f7400e9f0 tx=0x7f7f7400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 shutdown_connections 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f680a2e20 0x7f7f680a52e0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f80105670 0x7f7f801a1280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f80104cb0 0x7f7f8019a250 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 --2- 172.21.15.69:0/2121171491 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f80103ac0 0x7f7f80199d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 >> 172.21.15.69:0/2121171491 conn(0x7f7f800ff050 msgr2=0x7f7f80107fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 shutdown_connections 2024-05-09T09:32:05.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:05.239+0000 7f7f85c66640 1 -- 172.21.15.69:0/2121171491 wait complete. 2024-05-09T09:32:05.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 920 72.51 MB/sec warmup 3 sec latency 411.653 ms 2024-05-09T09:32:06.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 1458 64.38 MB/sec warmup 4 sec latency 54.249 ms 2024-05-09T09:32:07.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:06 smithi069 ceph-mon[30123]: pgmap v766: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-09T09:32:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:06 smithi163 ceph-mon[34354]: pgmap v766: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-09T09:32:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:06 smithi103 ceph-mon[34421]: pgmap v766: 129 pgs: 129 active+clean; 185 MiB data, 3.7 GiB used, 1.0 TiB / 1.0 TiB avail; 677 KiB/s rd, 2.3 MiB/s wr, 254 op/s 2024-05-09T09:32:07.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 2167 54.35 MB/sec warmup 5 sec latency 86.170 ms 2024-05-09T09:32:08.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 2562 48.10 MB/sec warmup 6 sec latency 79.380 ms 2024-05-09T09:32:09.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:08 smithi069 ceph-mon[30123]: pgmap v767: 129 pgs: 129 active+clean; 184 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-05-09T09:32:09.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:08 smithi163 ceph-mon[34354]: pgmap v767: 129 pgs: 129 active+clean; 184 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-05-09T09:32:09.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:08 smithi103 ceph-mon[34421]: pgmap v767: 129 pgs: 129 active+clean; 184 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 2.9 MiB/s wr, 406 op/s 2024-05-09T09:32:09.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 3230 49.43 MB/sec warmup 7 sec latency 58.403 ms 2024-05-09T09:32:10.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 3679 48.29 MB/sec warmup 8 sec latency 74.967 ms 2024-05-09T09:32:11.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:10 smithi163 ceph-mon[34354]: pgmap v768: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.1 MiB/s wr, 497 op/s 2024-05-09T09:32:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:10 smithi103 ceph-mon[34421]: pgmap v768: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.1 MiB/s wr, 497 op/s 2024-05-09T09:32:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:10 smithi069 ceph-mon[30123]: pgmap v768: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 943 KiB/s rd, 3.1 MiB/s wr, 497 op/s 2024-05-09T09:32:11.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 3890 44.33 MB/sec warmup 9 sec latency 48.179 ms 2024-05-09T09:32:12.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 4006 40.04 MB/sec warmup 10 sec latency 114.553 ms 2024-05-09T09:32:13.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:12 smithi163 ceph-mon[34354]: pgmap v769: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 451 op/s 2024-05-09T09:32:13.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:12 smithi103 ceph-mon[34421]: pgmap v769: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 451 op/s 2024-05-09T09:32:13.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:12 smithi069 ceph-mon[30123]: pgmap v769: 129 pgs: 129 active+clean; 179 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 716 KiB/s rd, 2.8 MiB/s wr, 451 op/s 2024-05-09T09:32:13.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 4139 36.44 MB/sec warmup 11 sec latency 320.954 ms 2024-05-09T09:32:14.017 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:13 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:32:14.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:13 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:32:14.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:13 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:32:14.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 4422 33.97 MB/sec warmup 12 sec latency 99.017 ms 2024-05-09T09:32:15.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:14 smithi163 ceph-mon[34354]: pgmap v770: 129 pgs: 129 active+clean; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-05-09T09:32:15.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:14 smithi103 ceph-mon[34421]: pgmap v770: 129 pgs: 129 active+clean; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-05-09T09:32:15.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:14 smithi069 ceph-mon[30123]: pgmap v770: 129 pgs: 129 active+clean; 182 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 739 KiB/s rd, 3.5 MiB/s wr, 474 op/s 2024-05-09T09:32:15.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 4830 33.21 MB/sec warmup 13 sec latency 50.141 ms 2024-05-09T09:32:16.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:32:16.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:32:16.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:15 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:32:16.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:32:16.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:32:16.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:15 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:32:16.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:32:16.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:32:16.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:15 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:32:16.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 5509 32.50 MB/sec warmup 14 sec latency 48.091 ms 2024-05-09T09:32:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:16 smithi069 ceph-mon[30123]: pgmap v771: 129 pgs: 129 active+clean; 188 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-05-09T09:32:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:16 smithi163 ceph-mon[34354]: pgmap v771: 129 pgs: 129 active+clean; 188 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-05-09T09:32:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:16 smithi103 ceph-mon[34421]: pgmap v771: 129 pgs: 129 active+clean; 188 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 992 KiB/s rd, 3.6 MiB/s wr, 439 op/s 2024-05-09T09:32:17.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 6419 34.44 MB/sec warmup 15 sec latency 156.165 ms 2024-05-09T09:32:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/3515743658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 msgr2=0x7f0ce010ae90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/3515743658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce010ae90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4007380 tx=0x7f0cd402afc0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/3515743658 shutdown_connections 2024-05-09T09:32:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/3515743658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010df40 0x7f0ce0110400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/3515743658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010b460 0x7f0ce010d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/3515743658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce010ae90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.362+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/3515743658 >> 172.21.15.69:0/3515743658 conn(0x7f0ce006ea60 msgr2=0x7f0ce0070ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.363+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/3515743658 shutdown_connections 2024-05-09T09:32:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.363+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/3515743658 wait complete. 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 Processor -- start 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 -- start start 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce01a0280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010df40 0x7f0ce01a7830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.365 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.364+0000 7f0ce6f5c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce0113b20 con 0x7f0ce010b460 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0ce6f5c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01139a0 con 0x7f0ce0108aa0 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0ce6f5c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce0113ca0 con 0x7f0ce010df40 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57196/0 (socket says 172.21.15.69:57196) 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0cdffff640 1 -- 172.21.15.69:0/887752849 learned_addr learned my addr 172.21.15.69:0/887752849 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:18.366 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0ce54f9640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010df40 0x7f0ce01a7830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.365+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce01a0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 -- 172.21.15.69:0/887752849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010df40 msgr2=0x7f0ce01a7830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010df40 0x7f0ce01a7830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 -- 172.21.15.69:0/887752849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 msgr2=0x7f0ce01a0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce01a0280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 -- 172.21.15.69:0/887752849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f0ce01a7ea0 con 0x7f0ce010b460 2024-05-09T09:32:18.367 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.366+0000 7f0cdffff640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000e9c0 tx=0x7f0cd000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.367+0000 7f0cddffb640 1 -- 172.21.15.69:0/887752849 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd000cd80 con 0x7f0ce010b460 2024-05-09T09:32:18.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.367+0000 7f0cddffb640 1 -- 172.21.15.69:0/887752849 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0005500 con 0x7f0ce010b460 2024-05-09T09:32:18.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.367+0000 7f0cddffb640 1 -- 172.21.15.69:0/887752849 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0010690 con 0x7f0ce010b460 2024-05-09T09:32:18.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.367+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f0ce01a8190 con 0x7f0ce010b460 2024-05-09T09:32:18.368 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.367+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f0ce01a8690 con 0x7f0ce010b460 2024-05-09T09:32:18.371 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.371+0000 7f0cddffb640 1 -- 172.21.15.69:0/887752849 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f0cd00046c0 con 0x7f0ce010b460 2024-05-09T09:32:18.372 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.372+0000 7f0ce9c5f740 1 Processor -- start 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.372+0000 7f0ce9c5f740 1 -- start start 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.372+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe73d35f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73d66b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe73e3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7337340 con 0x55dbe73d31d0 2024-05-09T09:32:18.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dbe72df290 con 0x55dbe73d3bc0 2024-05-09T09:32:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dbe7333300 con 0x55dbe73d6d20 2024-05-09T09:32:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0cddffb640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 0x7f0cc80a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.373+0000 7f0cddffb640 1 -- 172.21.15.69:0/887752849 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f0cd00cf870 con 0x7f0ce010b460 2024-05-09T09:32:18.374 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.374+0000 7f0ce54f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe73d35f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.374+0000 7f0ce54f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe73d35f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57206/0 (socket says 172.21.15.69:57206) 2024-05-09T09:32:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.374+0000 7f0ce54f9640 1 -- 172.21.15.69:0/1771080686 learned_addr learned my addr 172.21.15.69:0/1771080686 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:18.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.374+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe73e3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.376+0000 7f0cdffff640 1 -- 172.21.15.69:0/1771080686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 msgr2=0x55dbe73d66b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.376 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.376+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73d66b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.376+0000 7f0cdffff640 1 -- 172.21.15.69:0/1771080686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 msgr2=0x55dbe73d35f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.376+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe73d35f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.376+0000 7f0cdffff640 1 -- 172.21.15.69:0/1771080686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe73038e0 con 0x55dbe73d6d20 2024-05-09T09:32:18.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.378+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe73e3b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0098150 tx=0x7f0cd009bac0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b709936fcf39b751 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.378+0000 7f0ce54f9640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 0x7f0cc80a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.380+0000 7f0cc6ffd640 1 -- 172.21.15.69:0/1771080686 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd009c3f0 con 0x55dbe73d6d20 2024-05-09T09:32:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.380+0000 7f0cc6ffd640 1 -- 172.21.15.69:0/1771080686 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd009bcf0 con 0x55dbe73d6d20 2024-05-09T09:32:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.380+0000 7f0cc6ffd640 1 -- 172.21.15.69:0/1771080686 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00a4730 con 0x55dbe73d6d20 2024-05-09T09:32:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.381+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1771080686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 msgr2=0x55dbe73e3b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.381+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe73e3b90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0098150 tx=0x7f0cd009bac0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.384+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1771080686 shutdown_connections 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.384+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe73e3b90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.384+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73d66b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.384+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1771080686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe73d35f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.385+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1771080686 >> 172.21.15.69:0/1771080686 conn(0x55dbe73ce8c0 msgr2=0x55dbe73cecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:18.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.385+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1771080686 shutdown_connections 2024-05-09T09:32:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.385+0000 7f0ce54f9640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 0x7f0cc80a5240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55dbe73d4420 tx=0x7f0cd8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.385+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1771080686 wait complete. 2024-05-09T09:32:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.385+0000 7f0ce9c5f740 1 -- start start 2024-05-09T09:32:18.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 Processor -- start 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d31d0 0x55dbe73dd430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d6d20 0x55dbe73e29a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dbe73d7760 con 0x55dbe73d6d20 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dbe72f2790 con 0x55dbe73d3bc0 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dbe73d78e0 con 0x55dbe73d31d0 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.386+0000 7f0ce9c5f740 10 client.0 _ll_register_callbacks cb 0x55dbe732da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.387+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.387+0000 7f0cdffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:57858/0 (socket says 172.21.15.69:57858) 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.387+0000 7f0cdffff640 1 -- 172.21.15.69:0/1981130405 learned_addr learned my addr 172.21.15.69:0/1981130405 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:18.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d31d0 0x55dbe73dd430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 -- 172.21.15.69:0/1981130405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d31d0 msgr2=0x55dbe73dd430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d31d0 0x55dbe73dd430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 -- 172.21.15.69:0/1981130405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d6d20 msgr2=0x55dbe73e29a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d6d20 0x55dbe73e29a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 -- 172.21.15.69:0/1981130405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe704b210 con 0x55dbe73d3bc0 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cdffff640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00a4430 tx=0x7f0cd0004b30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8cb55591f736ce14 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.388+0000 7f0cbffff640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.389+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd009c3f0 con 0x55dbe73d3bc0 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.389+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd009bcf0 con 0x55dbe73d3bc0 2024-05-09T09:32:18.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.389+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f0cd0099d90 con 0x55dbe73d3bc0 2024-05-09T09:32:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.392+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0cd00a00a0 con 0x55dbe73d3bc0 2024-05-09T09:32:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.392+0000 7f0ce9c5f740 10 client.26426 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:18.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.392+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55dbe7117620 con 0x55dbe73d3bc0 2024-05-09T09:32:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd009bcf0 con 0x55dbe73d3bc0 2024-05-09T09:32:18.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f0cd009ac80 con 0x55dbe73d3bc0 2024-05-09T09:32:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 10 client.26426 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 10 client.26426 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:18.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 20 client.26426 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe73f9be0 0x55dbe73fbfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 4 client.26426 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.393+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dbe7344a70 con 0x55dbe73f9be0 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.394+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe73f9be0 0x55dbe73fbfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.395+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe73f9be0 0x55dbe73fbfd0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd24c2467c0e11ad server_cookie=ee2205af3ec1f7fc in_seq=0 out_seq=0 2024-05-09T09:32:18.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.395+0000 7f0cbffff640 10 client.26426 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0cbffff640 1 -- 172.21.15.69:0/1981130405 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dbe7344a70 con 0x55dbe73f9be0 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0cbffff640 10 client.26426 handle_command_reply: tid=0 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 client.26426 shutdown 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 msgr2=0x55dbe73dd970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f0cd00a4430 tx=0x7f0cd0004b30 comp rx=0 tx=0).stop 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 shutdown_connections 2024-05-09T09:32:18.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d6d20 0x55dbe73e29a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe73dd970 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe73f9be0 0x55dbe73fbfd0 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/1981130405 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d31d0 0x55dbe73dd430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 >> 172.21.15.69:0/1981130405 conn(0x55dbe73ce8c0 msgr2=0x55dbe73cda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0cbffff640 0 client.26426 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.397+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 shutdown_connections 2024-05-09T09:32:18.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.398+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/1981130405 wait complete. 2024-05-09T09:32:18.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.398+0000 7f0ce9c5f740 20 client.26426 trim_cache size 0 max 16384 2024-05-09T09:32:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 Processor -- start 2024-05-09T09:32:18.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 -- start start 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d3bc0 0x55dbe7401ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d6d20 0x55dbe73fbbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01b3480 con 0x55dbe73d31d0 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01b0cc0 con 0x55dbe73d6d20 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01aeb10 con 0x55dbe73d3bc0 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57224/0 (socket says 172.21.15.69:57224) 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4219080269 learned_addr learned my addr 172.21.15.69:0/4219080269 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0cdffff640 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d3bc0 0x55dbe7401ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce54f9640 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d6d20 0x55dbe73fbbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4219080269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d3bc0 msgr2=0x55dbe7401ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d3bc0 0x55dbe7401ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4219080269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d6d20 msgr2=0x55dbe73fbbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d6d20 0x55dbe73fbbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.418+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4219080269 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dbe73d2e10 con 0x55dbe73d31d0 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4035040 tx=0x7f0cd4005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7231d75683c6d82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0cc67fc640 1 -- 172.21.15.69:0/4219080269 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd40333f0 con 0x55dbe73d31d0 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0cc67fc640 1 -- 172.21.15.69:0/4219080269 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd4033590 con 0x55dbe73d31d0 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0cc67fc640 1 -- 172.21.15.69:0/4219080269 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4033c20 con 0x55dbe73d31d0 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4219080269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 msgr2=0x55dbe7401760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4035040 tx=0x7f0cd4005bd0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4219080269 shutdown_connections 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.419+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d6d20 0x55dbe73fbbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d3bc0 0x55dbe7401ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4219080269 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe7401760 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4219080269 >> 172.21.15.69:0/4219080269 conn(0x55dbe73ce8c0 msgr2=0x55dbe73ceca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4219080269 shutdown_connections 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4219080269 wait complete. 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- start start 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 Processor -- start 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe740aad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe740b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01af180 con 0x55dbe73d31d0 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01b05f0 con 0x55dbe73d3bc0 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f0ce01af850 con 0x55dbe73d6d20 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce9c5f740 10 client.0 _ll_register_callbacks cb 0x7f0cd00985b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:18.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce4cf8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce4cf8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57240/0 (socket says 172.21.15.69:57240) 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4361905 learned_addr learned my addr 172.21.15.69:0/4361905 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.420+0000 7f0ce54f9640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe740b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cdffff640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe740aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4361905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 msgr2=0x55dbe740b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe740b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 msgr2=0x55dbe740aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe740aad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 -- 172.21.15.69:0/4361905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dbe73ddce0 con 0x55dbe73d31d0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce4cf8640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4000c00 tx=0x7f0cd40337c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d7afb057402cd2b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cbffff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4004250 con 0x55dbe73d31d0 2024-05-09T09:32:18.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f0cd4007a50 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0cd4033590 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0cd4007c50 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce9c5f740 10 client.17136 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55dbe740d190 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.421+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0cd4033590 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f0cd4005040 con 0x55dbe73d31d0 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 10 client.17136 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 10 client.17136 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 20 client.17136 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe745fe70 0x55dbe7462290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 4 client.17136 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dbe733edc0 con 0x55dbe745fe70 2024-05-09T09:32:18.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0cdffff640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe745fe70 0x55dbe7462290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0cdffff640 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe745fe70 0x55dbe7462290 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=266b4bc502ca60a7 server_cookie=6ea2d67b4ed4ebee in_seq=0 out_seq=0 2024-05-09T09:32:18.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:18.422+0000 7f0cbffff640 10 client.17136 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:18.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 6982 34.79 MB/sec warmup 16 sec latency 316.402 ms 2024-05-09T09:32:19.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.069+0000 7f0cbffff640 1 -- 172.21.15.69:0/4361905 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dbe733edc0 con 0x55dbe745fe70 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.069+0000 7f0cbffff640 10 client.17136 handle_command_reply: tid=0 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.069+0000 7f0ce9c5f740 1 client.17136 shutdown 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.069+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 msgr2=0x55dbe740a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.069+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4000c00 tx=0x7f0cd40337c0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 shutdown_connections 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dbe745fe70 0x55dbe7462290 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dbe73d6d20 0x55dbe740b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dbe73d3bc0 0x55dbe740aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 --2- 172.21.15.69:0/4361905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dbe73d31d0 0x55dbe740a590 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 >> 172.21.15.69:0/4361905 conn(0x55dbe73ce8c0 msgr2=0x55dbe73fbbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0cbffff640 0 client.17136 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 shutdown_connections 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 1 -- 172.21.15.69:0/4361905 wait complete. 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.070+0000 7f0ce9c5f740 20 client.17136 trim_cache size 0 max 16384 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:32:19.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 msgr2=0x7f0cc80a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 0x7f0cc80a5240 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x55dbe73d4420 tx=0x7f0cd8004ab0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 msgr2=0x7f0ce01a07c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f0cd000e9c0 tx=0x7f0cd000ee80 comp rx=0 tx=0).stop 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 shutdown_connections 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f0cc80a2d80 0x7f0cc80a5240 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f0ce010df40 0x7f0ce01a7830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f0ce010b460 0x7f0ce01a07c0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 --2- 172.21.15.69:0/887752849 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f0ce0108aa0 0x7f0ce01a0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 >> 172.21.15.69:0/887752849 conn(0x7f0ce006ea60 msgr2=0x7f0ce010be20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 shutdown_connections 2024-05-09T09:32:19.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:19.071+0000 7f0ce6f5c640 1 -- 172.21.15.69:0/887752849 wait complete. 2024-05-09T09:32:19.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:18 smithi069 ceph-mon[30123]: pgmap v772: 129 pgs: 129 active+clean; 190 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 462 op/s 2024-05-09T09:32:19.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:18 smithi163 ceph-mon[34354]: pgmap v772: 129 pgs: 129 active+clean; 190 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 462 op/s 2024-05-09T09:32:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:18 smithi103 ceph-mon[34421]: pgmap v772: 129 pgs: 129 active+clean; 190 MiB data, 3.8 GiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 4.0 MiB/s wr, 462 op/s 2024-05-09T09:32:19.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 7440 34.49 MB/sec warmup 17 sec latency 397.213 ms 2024-05-09T09:32:20.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 7827 33.32 MB/sec warmup 18 sec latency 179.141 ms 2024-05-09T09:32:21.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:20 smithi069 ceph-mon[30123]: pgmap v773: 129 pgs: 129 active+clean; 195 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-05-09T09:32:21.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:20 smithi163 ceph-mon[34354]: pgmap v773: 129 pgs: 129 active+clean; 195 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-05-09T09:32:21.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:20 smithi103 ceph-mon[34421]: pgmap v773: 129 pgs: 129 active+clean; 195 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 898 KiB/s rd, 3.9 MiB/s wr, 358 op/s 2024-05-09T09:32:21.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 8511 32.97 MB/sec warmup 19 sec latency 89.830 ms 2024-05-09T09:32:22.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 9466 34.27 MB/sec warmup 20 sec latency 49.799 ms 2024-05-09T09:32:23.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:22 smithi069 ceph-mon[30123]: pgmap v774: 129 pgs: 129 active+clean; 199 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 304 op/s 2024-05-09T09:32:23.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:22 smithi163 ceph-mon[34354]: pgmap v774: 129 pgs: 129 active+clean; 199 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 304 op/s 2024-05-09T09:32:23.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:22 smithi103 ceph-mon[34421]: pgmap v774: 129 pgs: 129 active+clean; 199 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 3.7 MiB/s wr, 304 op/s 2024-05-09T09:32:23.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 9729 32.98 MB/sec warmup 21 sec latency 364.429 ms 2024-05-09T09:32:24.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 10311 33.16 MB/sec warmup 22 sec latency 85.837 ms 2024-05-09T09:32:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:24 smithi069 ceph-mon[30123]: pgmap v775: 129 pgs: 129 active+clean; 234 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.3 MiB/s wr, 516 op/s 2024-05-09T09:32:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:24 smithi163 ceph-mon[34354]: pgmap v775: 129 pgs: 129 active+clean; 234 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.3 MiB/s wr, 516 op/s 2024-05-09T09:32:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:24 smithi103 ceph-mon[34421]: pgmap v775: 129 pgs: 129 active+clean; 234 MiB data, 3.9 GiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 7.3 MiB/s wr, 516 op/s 2024-05-09T09:32:25.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 11107 33.95 MB/sec warmup 23 sec latency 49.680 ms 2024-05-09T09:32:26.819 INFO:tasks.workunit.client.0.smithi069.stdout: 10 11756 33.85 MB/sec warmup 24 sec latency 44.341 ms 2024-05-09T09:32:27.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:26 smithi069 ceph-mon[30123]: pgmap v776: 129 pgs: 129 active+clean; 264 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-09T09:32:27.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:26 smithi163 ceph-mon[34354]: pgmap v776: 129 pgs: 129 active+clean; 264 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-09T09:32:27.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:26 smithi103 ceph-mon[34421]: pgmap v776: 129 pgs: 129 active+clean; 264 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 1.7 MiB/s rd, 11 MiB/s wr, 570 op/s 2024-05-09T09:32:27.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 12465 33.76 MB/sec warmup 25 sec latency 55.936 ms 2024-05-09T09:32:28.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 12971 33.63 MB/sec warmup 26 sec latency 131.807 ms 2024-05-09T09:32:29.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:28 smithi069 ceph-mon[30123]: pgmap v777: 129 pgs: 129 active+clean; 353 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 19 MiB/s wr, 700 op/s 2024-05-09T09:32:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:28 smithi163 ceph-mon[34354]: pgmap v777: 129 pgs: 129 active+clean; 353 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 19 MiB/s wr, 700 op/s 2024-05-09T09:32:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:28 smithi103 ceph-mon[34421]: pgmap v777: 129 pgs: 129 active+clean; 353 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 2.9 MiB/s rd, 19 MiB/s wr, 700 op/s 2024-05-09T09:32:29.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 13749 33.95 MB/sec warmup 27 sec latency 155.644 ms 2024-05-09T09:32:30.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 14690 34.32 MB/sec warmup 28 sec latency 44.116 ms 2024-05-09T09:32:31.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:30 smithi069 ceph-mon[30123]: pgmap v778: 129 pgs: 129 active+clean; 359 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 723 op/s 2024-05-09T09:32:31.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:30 smithi163 ceph-mon[34354]: pgmap v778: 129 pgs: 129 active+clean; 359 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 723 op/s 2024-05-09T09:32:31.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:30 smithi103 ceph-mon[34421]: pgmap v778: 129 pgs: 129 active+clean; 359 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 3.5 MiB/s rd, 19 MiB/s wr, 723 op/s 2024-05-09T09:32:31.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 15561 35.16 MB/sec warmup 29 sec latency 41.630 ms 2024-05-09T09:32:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- 172.21.15.69:0/3179087891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 msgr2=0x7f086c101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 --2- 172.21.15.69:0/3179087891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c101e00 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f085c009900 tx=0x7f085c02a410 comp rx=0 tx=0).stop 2024-05-09T09:32:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- 172.21.15.69:0/3179087891 shutdown_connections 2024-05-09T09:32:32.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 --2- 172.21.15.69:0/3179087891 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c102340 0x7f086c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 --2- 172.21.15.69:0/3179087891 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c101e00 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 --2- 172.21.15.69:0/3179087891 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086c100790 0x7f086c100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- 172.21.15.69:0/3179087891 >> 172.21.15.69:0/3179087891 conn(0x7f086c0fbd00 msgr2=0x7f086c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- 172.21.15.69:0/3179087891 shutdown_connections 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- 172.21.15.69:0/3179087891 wait complete. 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 Processor -- start 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.192+0000 7f08743ee640 1 -- start start 2024-05-09T09:32:32.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c100790 0x7f086c1958d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086c102340 0x7f086c19ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f086c10b360 con 0x7f086c101980 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f086c10b1e0 con 0x7f086c102340 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f08743ee640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f086c10b4e0 con 0x7f086c100790 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39278/0 (socket says 172.21.15.69:39278) 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 -- 172.21.15.69:0/3932695065 learned_addr learned my addr 172.21.15.69:0/3932695065 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f087218a640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c100790 0x7f086c1958d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.195 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 -- 172.21.15.69:0/3932695065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c100790 msgr2=0x7f086c1958d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c100790 0x7f086c1958d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 -- 172.21.15.69:0/3932695065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086c102340 msgr2=0x7f086c19ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086c102340 0x7f086c19ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 -- 172.21.15.69:0/3932695065 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f086c19d4f0 con 0x7f086c101980 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.193+0000 7f0871989640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f085c031440 tx=0x7f085c032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.194+0000 7f08637fe640 1 -- 172.21.15.69:0/3932695065 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c03f030 con 0x7f086c101980 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.194+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f086c1a1b90 con 0x7f086c101980 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.194+0000 7f08637fe640 1 -- 172.21.15.69:0/3932695065 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c003ce0 con 0x7f086c101980 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.194+0000 7f08637fe640 1 -- 172.21.15.69:0/3932695065 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c035070 con 0x7f086c101980 2024-05-09T09:32:32.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.194+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f086c1a20d0 con 0x7f086c101980 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.196+0000 7f08637fe640 1 -- 172.21.15.69:0/3932695065 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f085c042030 con 0x7f086c101980 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 Processor -- start 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 -- start start 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09ed670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a9e0951340 con 0x55a9e09f2db0 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9e08f9290 con 0x55a9e09edc40 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08771dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a9e094d300 con 0x55a9e09ed250 2024-05-09T09:32:32.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f08637fe640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 0x7f08580a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f0871989640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f0871989640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39286/0 (socket says 172.21.15.69:39286) 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f087298b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09ed670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f087218a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f0871989640 1 -- 172.21.15.69:0/2520382504 learned_addr learned my addr 172.21.15.69:0/2520382504 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.197+0000 7f087298b640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 0x7f08580a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f08637fe640 1 -- 172.21.15.69:0/3932695065 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f085c0ef8c0 con 0x7f086c101980 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 -- 172.21.15.69:0/2520382504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 msgr2=0x55a9e09ed670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09ed670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 -- 172.21.15.69:0/2520382504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 msgr2=0x55a9e09f2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 -- 172.21.15.69:0/2520382504 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e091d8e0 con 0x55a9e09f2db0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0871989640 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f085c005ec0 tx=0x7f085c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=364adeb6019c3d99 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f087298b640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 0x7f08580a50f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0864005c40 tx=0x7f0864005bd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0860bfe640 1 -- 172.21.15.69:0/2520382504 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c049070 con 0x55a9e09f2db0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0860bfe640 1 -- 172.21.15.69:0/2520382504 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c0bc4a0 con 0x55a9e09f2db0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.198+0000 7f0860bfe640 1 -- 172.21.15.69:0/2520382504 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f085c0b8410 con 0x55a9e09f2db0 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- 172.21.15.69:0/2520382504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 msgr2=0x55a9e09f5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f085c005ec0 tx=0x7f085c02a410 comp rx=0 tx=0).stop 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- 172.21.15.69:0/2520382504 shutdown_connections 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09f5240 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 --2- 172.21.15.69:0/2520382504 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09ed670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- 172.21.15.69:0/2520382504 >> 172.21.15.69:0/2520382504 conn(0x55a9e09e88c0 msgr2=0x55a9e09e8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- 172.21.15.69:0/2520382504 shutdown_connections 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- 172.21.15.69:0/2520382504 wait complete. 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 -- start start 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.199+0000 7f08771dd740 1 Processor -- start 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f7610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.207 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a9e0951340 con 0x55a9e09f2db0 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9e090c790 con 0x55a9e09edc40 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a9e09f5b20 con 0x55a9e09ed250 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f08771dd740 10 client.0 _ll_register_callbacks cb 0x55a9e0947a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087298b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f0871989640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f7610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:49720/0 (socket says 172.21.15.69:49720) 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087298b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39294/0 (socket says 172.21.15.69:39294) 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 -- 172.21.15.69:0/4080131899 learned_addr learned my addr 172.21.15.69:0/4080131899 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 -- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 msgr2=0x55a9e09f7610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f7610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 -- 172.21.15.69:0/4080131899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 msgr2=0x55a9e09fe650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 -- 172.21.15.69:0/4080131899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e0665210 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087298b640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f087218a640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f086800bfd0 tx=0x7f0868014960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f35cfcfeee6d9f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.200+0000 7f0855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086801b070 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f085c0bc4a0 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f0868005020 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f0868026050 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 10 client.26356 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a9e0731620 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f085c0bc4a0 con 0x55a9e09ed250 2024-05-09T09:32:32.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f08680035f0 con 0x55a9e09ed250 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 10 client.26356 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 10 client.26356 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 20 client.26356 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a03090 0x55a9e0a05530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 4 client.26356 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a9e095ea70 con 0x55a9e0a03090 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.201+0000 7f0871989640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a03090 0x55a9e0a05530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.202+0000 7f0871989640 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a03090 0x55a9e0a05530 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fde623b0a8cdbabd server_cookie=17cf5be17df54dc2 in_seq=0 out_seq=0 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.202+0000 7f0855ffb640 10 client.26356 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f0855ffb640 1 -- 172.21.15.69:0/4080131899 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a9e095ea70 con 0x55a9e0a03090 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f0855ffb640 10 client.26356 handle_command_reply: tid=0 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 client.26356 shutdown 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 msgr2=0x55a9e09f70d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f086800bfd0 tx=0x7f0868014960 comp rx=0 tx=0).stop 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 shutdown_connections 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09fe650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09edc40 0x55a9e09f7610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a03090 0x55a9e0a05530 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 --2- 172.21.15.69:0/4080131899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09ed250 0x55a9e09f70d0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 >> 172.21.15.69:0/4080131899 conn(0x55a9e09e88c0 msgr2=0x55a9e09e7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f0855ffb640 0 client.26356 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 shutdown_connections 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.203+0000 7f08771dd740 1 -- 172.21.15.69:0/4080131899 wait complete. 2024-05-09T09:32:32.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.204+0000 7f08771dd740 20 client.26356 trim_cache size 0 max 16384 2024-05-09T09:32:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.219+0000 7f08771dd740 1 Processor -- start 2024-05-09T09:32:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.219+0000 7f08771dd740 1 -- start start 2024-05-09T09:32:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 0x55a9e09ebd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fe610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1aef70 con 0x55a9e09f2db0 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ac7b0 con 0x55a9e09ed250 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f08771dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1aa600 con 0x55a9e09edc40 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087298b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 0x55a9e09ebd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087298b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39304/0 (socket says 172.21.15.69:39304) 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087298b640 1 -- 172.21.15.69:0/3882753393 learned_addr learned my addr 172.21.15.69:0/3882753393 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f0871989640 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fe610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 -- 172.21.15.69:0/3882753393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 msgr2=0x55a9e09fe610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fe610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 -- 172.21.15.69:0/3882753393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 msgr2=0x55a9e09e7170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 -- 172.21.15.69:0/3882753393 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9e09ee020 con 0x55a9e09ed250 2024-05-09T09:32:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087298b640 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:32:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.220+0000 7f087218a640 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 0x55a9e09ebd50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0868003a90 tx=0x7f0868005830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=dd47336865e6c92c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.221+0000 7f0857fff640 1 -- 172.21.15.69:0/3882753393 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868000e60 con 0x55a9e09ed250 2024-05-09T09:32:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.221+0000 7f0857fff640 1 -- 172.21.15.69:0/3882753393 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0868003c80 con 0x55a9e09ed250 2024-05-09T09:32:32.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.221+0000 7f0857fff640 1 -- 172.21.15.69:0/3882753393 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f086801b070 con 0x55a9e09ed250 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.221+0000 7f08771dd740 1 -- 172.21.15.69:0/3882753393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 msgr2=0x55a9e09ebd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.221+0000 7f08771dd740 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 0x55a9e09ebd50 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f0868003a90 tx=0x7f0868005830 comp rx=0 tx=0).stop 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- 172.21.15.69:0/3882753393 shutdown_connections 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09f2db0 0x55a9e09e7170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fe610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- 172.21.15.69:0/3882753393 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09ed250 0x55a9e09ebd50 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- 172.21.15.69:0/3882753393 >> 172.21.15.69:0/3882753393 conn(0x55a9e09e88c0 msgr2=0x55a9e09eb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- 172.21.15.69:0/3882753393 shutdown_connections 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- 172.21.15.69:0/3882753393 wait complete. 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- start start 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 Processor -- start 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09f2db0 0x55a9e0a676a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1aac70 con 0x55a9e09ed250 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ac0e0 con 0x55a9e09f2db0 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f086c1ab340 con 0x55a9e09edc40 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f08771dd740 10 client.0 _ll_register_callbacks cb 0x7f086800ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.222+0000 7f087218a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39312/0 (socket says 172.21.15.69:39312) 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 -- 172.21.15.69:0/927580478 learned_addr learned my addr 172.21.15.69:0/927580478 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0871989640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087298b640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09f2db0 0x55a9e0a676a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 -- 172.21.15.69:0/927580478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 msgr2=0x55a9e09fb830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fb830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 -- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09f2db0 msgr2=0x55a9e0a676a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09f2db0 0x55a9e0a676a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 -- 172.21.15.69:0/927580478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a9e09fe8b0 con 0x55a9e09ed250 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f087218a640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f08680058c0 tx=0x7f0868005470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e06fd9529c8dc1b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0855ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f0868037060 con 0x55a9e09ed250 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f086802a630 con 0x55a9e09ed250 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f0868003c80 con 0x55a9e09ed250 2024-05-09T09:32:32.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f086803e050 con 0x55a9e09ed250 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f08771dd740 10 client.17154 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55a9e0a68c00 con 0x55a9e09ed250 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.223+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f0868003c80 con 0x55a9e09ed250 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f08680178f0 con 0x55a9e09ed250 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 10 client.17154 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 10 client.17154 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 20 client.17154 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a69580 0x55a9e0a6b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 4 client.17154 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a9e0958dc0 con 0x55a9e0a69580 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f0871989640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a69580 0x55a9e0a6b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f0871989640 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a69580 0x55a9e0a6b9a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d57e5ae3df0c5d63 server_cookie=410c6a15a7e0f409 in_seq=0 out_seq=0 2024-05-09T09:32:32.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.224+0000 7f0855ffb640 10 client.17154 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f0855ffb640 1 -- 172.21.15.69:0/927580478 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a9e0958dc0 con 0x55a9e0a69580 2024-05-09T09:32:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f0855ffb640 10 client.17154 handle_command_reply: tid=0 2024-05-09T09:32:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 client.17154 shutdown 2024-05-09T09:32:32.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 msgr2=0x55a9e09fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f08680058c0 tx=0x7f0868005470 comp rx=0 tx=0).stop 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 shutdown_connections 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9e09f2db0 0x55a9e0a676a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a9e0a69580 0x55a9e0a6b9a0 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9e09edc40 0x55a9e09fb830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f0855ffb640 0 client.17154 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 --2- 172.21.15.69:0/927580478 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9e09ed250 0x55a9e09fb2f0 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 >> 172.21.15.69:0/927580478 conn(0x55a9e09e88c0 msgr2=0x55a9e09eb640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 shutdown_connections 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 1 -- 172.21.15.69:0/927580478 wait complete. 2024-05-09T09:32:32.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.685+0000 7f08771dd740 20 client.17154 trim_cache size 0 max 16384 2024-05-09T09:32:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:32:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:32:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:32:32.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:32:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 msgr2=0x7f08580a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 0x7f08580a50f0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f0864005c40 tx=0x7f0864005bd0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 msgr2=0x7f086c195e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f085c031440 tx=0x7f085c032ce0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 shutdown_connections 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f08580a2c30 0x7f08580a50f0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f086c102340 0x7f086c19ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f086c101980 0x7f086c195e10 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 --2- 172.21.15.69:0/3932695065 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f086c100790 0x7f086c1958d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 >> 172.21.15.69:0/3932695065 conn(0x7f086c0fbd00 msgr2=0x7f086c104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 shutdown_connections 2024-05-09T09:32:32.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:32.686+0000 7f08743ee640 1 -- 172.21.15.69:0/3932695065 wait complete. 2024-05-09T09:32:32.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 15943 34.57 MB/sec warmup 30 sec latency 385.213 ms 2024-05-09T09:32:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:32 smithi069 ceph-mon[30123]: pgmap v779: 129 pgs: 129 active+clean; 360 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 19 MiB/s wr, 668 op/s 2024-05-09T09:32:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:32 smithi163 ceph-mon[34354]: pgmap v779: 129 pgs: 129 active+clean; 360 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 19 MiB/s wr, 668 op/s 2024-05-09T09:32:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:32 smithi103 ceph-mon[34421]: pgmap v779: 129 pgs: 129 active+clean; 360 MiB data, 4.0 GiB used, 1.0 TiB / 1.0 TiB avail; 4.0 MiB/s rd, 19 MiB/s wr, 668 op/s 2024-05-09T09:32:33.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 16628 34.68 MB/sec warmup 31 sec latency 67.876 ms 2024-05-09T09:32:34.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 17194 34.23 MB/sec warmup 32 sec latency 188.051 ms 2024-05-09T09:32:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:34 smithi069 ceph-mon[30123]: pgmap v780: 129 pgs: 129 active+clean; 370 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 721 op/s 2024-05-09T09:32:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:34 smithi163 ceph-mon[34354]: pgmap v780: 129 pgs: 129 active+clean; 370 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 721 op/s 2024-05-09T09:32:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:34 smithi103 ceph-mon[34421]: pgmap v780: 129 pgs: 129 active+clean; 370 MiB data, 4.1 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 721 op/s 2024-05-09T09:32:35.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 17862 34.47 MB/sec warmup 33 sec latency 68.412 ms 2024-05-09T09:32:36.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 18663 34.40 MB/sec warmup 34 sec latency 50.199 ms 2024-05-09T09:32:37.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:36 smithi069 ceph-mon[30123]: pgmap v781: 129 pgs: 129 active+clean; 375 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 16 MiB/s wr, 599 op/s 2024-05-09T09:32:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:36 smithi163 ceph-mon[34354]: pgmap v781: 129 pgs: 129 active+clean; 375 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 16 MiB/s wr, 599 op/s 2024-05-09T09:32:37.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:36 smithi103 ceph-mon[34421]: pgmap v781: 129 pgs: 129 active+clean; 375 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 16 MiB/s wr, 599 op/s 2024-05-09T09:32:37.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 19425 34.75 MB/sec warmup 35 sec latency 49.567 ms 2024-05-09T09:32:38.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 20067 35.11 MB/sec warmup 36 sec latency 218.886 ms 2024-05-09T09:32:39.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:38 smithi069 ceph-mon[30123]: pgmap v782: 129 pgs: 129 active+clean; 374 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 707 op/s 2024-05-09T09:32:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:38 smithi163 ceph-mon[34354]: pgmap v782: 129 pgs: 129 active+clean; 374 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 707 op/s 2024-05-09T09:32:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:38 smithi103 ceph-mon[34421]: pgmap v782: 129 pgs: 129 active+clean; 374 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 13 MiB/s wr, 707 op/s 2024-05-09T09:32:39.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 20552 34.78 MB/sec warmup 37 sec latency 157.859 ms 2024-05-09T09:32:40.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 21112 34.58 MB/sec warmup 38 sec latency 128.264 ms 2024-05-09T09:32:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:40 smithi069 ceph-mon[30123]: pgmap v783: 129 pgs: 129 active+clean; 377 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2024-05-09T09:32:41.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:40 smithi163 ceph-mon[34354]: pgmap v783: 129 pgs: 129 active+clean; 377 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2024-05-09T09:32:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:40 smithi103 ceph-mon[34421]: pgmap v783: 129 pgs: 129 active+clean; 377 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.4 MiB/s wr, 583 op/s 2024-05-09T09:32:41.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 21702 34.35 MB/sec warmup 39 sec latency 56.835 ms 2024-05-09T09:32:42.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 22531 34.56 MB/sec warmup 40 sec latency 58.575 ms 2024-05-09T09:32:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:42 smithi069 ceph-mon[30123]: pgmap v784: 129 pgs: 129 active+clean; 378 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:32:43.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:42 smithi163 ceph-mon[34354]: pgmap v784: 129 pgs: 129 active+clean; 378 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:32:43.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:42 smithi103 ceph-mon[34421]: pgmap v784: 129 pgs: 129 active+clean; 378 MiB data, 4.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:32:43.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 23422 34.89 MB/sec warmup 41 sec latency 80.789 ms 2024-05-09T09:32:44.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 24085 35.29 MB/sec warmup 42 sec latency 96.991 ms 2024-05-09T09:32:45.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:44 smithi069 ceph-mon[30123]: pgmap v785: 129 pgs: 129 active+clean; 389 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 614 op/s 2024-05-09T09:32:45.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:44 smithi163 ceph-mon[34354]: pgmap v785: 129 pgs: 129 active+clean; 389 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 614 op/s 2024-05-09T09:32:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:44 smithi103 ceph-mon[34421]: pgmap v785: 129 pgs: 129 active+clean; 389 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.3 MiB/s wr, 614 op/s 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 -- 172.21.15.69:0/2509364095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054102340 msgr2=0x7f8054108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 --2- 172.21.15.69:0/2509364095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054102340 0x7f8054108bd0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f804c009ca0 tx=0x7f804c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 -- 172.21.15.69:0/2509364095 shutdown_connections 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 --2- 172.21.15.69:0/2509364095 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054102340 0x7f8054108bd0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 --2- 172.21.15.69:0/2509364095 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054101980 0x7f8054101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 --2- 172.21.15.69:0/2509364095 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054100790 0x7f8054100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 -- 172.21.15.69:0/2509364095 >> 172.21.15.69:0/2509364095 conn(0x7f80540fbd00 msgr2=0x7f80540fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 -- 172.21.15.69:0/2509364095 shutdown_connections 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.784+0000 7f805bae1640 1 -- 172.21.15.69:0/2509364095 wait complete. 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 Processor -- start 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 -- start start 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054101980 0x7f8054197fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054102340 0x7f805419f040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f805410d530 con 0x7f8054100790 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f805410d3b0 con 0x7f8054102340 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805bae1640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f805410d6b0 con 0x7f8054101980 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805987d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805987d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44094/0 (socket says 172.21.15.69:44094) 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805987d640 1 -- 172.21.15.69:0/1040888936 learned_addr learned my addr 172.21.15.69:0/1040888936 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805907c640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054101980 0x7f8054197fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.785+0000 7f805a07e640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054102340 0x7f805419f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 -- 172.21.15.69:0/1040888936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054101980 msgr2=0x7f8054197fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054101980 0x7f8054197fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 -- 172.21.15.69:0/1040888936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054102340 msgr2=0x7f805419f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054102340 0x7f805419f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 -- 172.21.15.69:0/1040888936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f805419f6b0 con 0x7f8054100790 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805987d640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f805000e9c0 tx=0x7f805000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f804affd640 1 -- 172.21.15.69:0/1040888936 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f805000cd80 con 0x7f8054100790 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f804affd640 1 -- 172.21.15.69:0/1040888936 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f80500035f0 con 0x7f8054100790 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f805419f9a0 con 0x7f8054100790 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f804affd640 1 -- 172.21.15.69:0/1040888936 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8050010690 con 0x7f8054100790 2024-05-09T09:32:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.786+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f805419fea0 con 0x7f8054100790 2024-05-09T09:32:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 Processor -- start 2024-05-09T09:32:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f804affd640 1 -- 172.21.15.69:0/1040888936 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f80500037a0 con 0x7f8054100790 2024-05-09T09:32:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 -- start start 2024-05-09T09:32:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.795 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d18740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d1b240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8c77340 con 0x55b3e8d13c40 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8c1f290 con 0x55b3e8d18db0 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805e7f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8c73300 con 0x55b3e8d13250 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805a07e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805987d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d18740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805a07e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:34458/0 (socket says 172.21.15.69:34458) 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805a07e640 1 -- 172.21.15.69:0/348206432 learned_addr learned my addr 172.21.15.69:0/348206432 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.789+0000 7f805907c640 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d1b240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 -- 172.21.15.69:0/348206432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 msgr2=0x55b3e8d1b240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d1b240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 -- 172.21.15.69:0/348206432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 msgr2=0x55b3e8d18740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d18740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 -- 172.21.15.69:0/348206432 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3e8c438e0 con 0x55b3e8d13250 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f804affd640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 0x7f80400a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805a07e640 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f804c009ca0 tx=0x7f804c009360 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b21a6524fd65ef93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f805987d640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 0x7f80400a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f804affd640 1 -- 172.21.15.69:0/1040888936 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f80500cf710 con 0x7f8054100790 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f803ffff640 1 -- 172.21.15.69:0/348206432 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c0098c0 con 0x55b3e8d13250 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f803ffff640 1 -- 172.21.15.69:0/348206432 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804c036040 con 0x55b3e8d13250 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.790+0000 7f803ffff640 1 -- 172.21.15.69:0/348206432 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c004290 con 0x55b3e8d13250 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- 172.21.15.69:0/348206432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 msgr2=0x55b3e8d13670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805987d640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 0x7f80400a50f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f80500cd200 tx=0x7f80500d5550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f804c009ca0 tx=0x7f804c009360 comp rx=0 tx=0).stop 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- 172.21.15.69:0/348206432 shutdown_connections 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d1b240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d18740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 --2- 172.21.15.69:0/348206432 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d13670 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- 172.21.15.69:0/348206432 >> 172.21.15.69:0/348206432 conn(0x55b3e8d0e8c0 msgr2=0x55b3e8d0ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- 172.21.15.69:0/348206432 shutdown_connections 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- 172.21.15.69:0/348206432 wait complete. 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 -- start start 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.791+0000 7f805e7f3740 1 Processor -- start 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8d1bb20 con 0x55b3e8d13c40 2024-05-09T09:32:45.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8c32790 con 0x55b3e8d18db0 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b3e8d1bca0 con 0x55b3e8d13250 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805e7f3740 10 client.0 _ll_register_callbacks cb 0x55b3e8c6da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44112/0 (socket says 172.21.15.69:44112) 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 -- 172.21.15.69:0/899524513 learned_addr learned my addr 172.21.15.69:0/899524513 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805a07e640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805a07e640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 -- 172.21.15.69:0/899524513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 msgr2=0x55b3e8d1d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 -- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 msgr2=0x55b3e8d24740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 -- 172.21.15.69:0/899524513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3e898b210 con 0x55b3e8d13c40 2024-05-09T09:32:45.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805a07e640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.792+0000 7f805907c640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f804400bf80 tx=0x7f80440118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8527694c2848a818 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044004020 con 0x55b3e8d13c40 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f804c036040 con 0x55b3e8d13c40 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f80440041c0 con 0x55b3e8d13c40 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f8044024050 con 0x55b3e8d13c40 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 10 client.17166 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b3e8a57620 con 0x55b3e8d13c40 2024-05-09T09:32:45.800 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f804c036040 con 0x55b3e8d13c40 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f8044004be0 con 0x55b3e8d13c40 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 10 client.17166 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 10 client.17166 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 20 client.17166 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d28dc0 0x55b3e8d2b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 4 client.17166 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.793+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b3e8c84a70 con 0x55b3e8d28dc0 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.794+0000 7f805a07e640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d28dc0 0x55b3e8d2b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.794+0000 7f803d7fa640 10 client.17166 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:45.801 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.794+0000 7f805a07e640 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d28dc0 0x55b3e8d2b1d0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0b941ca423b1b09 server_cookie=27d774dfc516d345 in_seq=0 out_seq=0 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.797+0000 7f803d7fa640 1 -- 172.21.15.69:0/899524513 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b3e8c84a70 con 0x55b3e8d28dc0 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.797+0000 7f803d7fa640 10 client.17166 handle_command_reply: tid=0 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.797+0000 7f805e7f3740 1 client.17166 shutdown 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 msgr2=0x55b3e8d1d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f804400bf80 tx=0x7f80440118d0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 shutdown_connections 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d28dc0 0x55b3e8d2b1d0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 --2- 172.21.15.69:0/899524513 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f803d7fa640 0 client.17166 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 >> 172.21.15.69:0/899524513 conn(0x55b3e8d0e8c0 msgr2=0x55b3e8d0da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 shutdown_connections 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 1 -- 172.21.15.69:0/899524513 wait complete. 2024-05-09T09:32:45.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.798+0000 7f805e7f3740 20 client.17166 trim_cache size 0 max 16384 2024-05-09T09:32:45.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.814+0000 7f805e7f3740 1 Processor -- start 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 -- start start 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80541acc20 con 0x55b3e8d13c40 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80541aa460 con 0x55b3e8d18db0 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805e7f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f80541a82b0 con 0x55b3e8d13250 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44126/0 (socket says 172.21.15.69:44126) 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 -- 172.21.15.69:0/383235860 learned_addr learned my addr 172.21.15.69:0/383235860 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805a07e640 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 -- 172.21.15.69:0/383235860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 msgr2=0x55b3e8d1d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 -- 172.21.15.69:0/383235860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 msgr2=0x55b3e8d24700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 -- 172.21.15.69:0/383235860 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b3e8d0f630 con 0x55b3e8d13c40 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805a07e640 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f805907c640 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8044001eb0 tx=0x7f8044011e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d7cb992d90543fc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.821 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f803f7fe640 1 -- 172.21.15.69:0/383235860 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044004020 con 0x55b3e8d13c40 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f803f7fe640 1 -- 172.21.15.69:0/383235860 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8044010e90 con 0x55b3e8d13c40 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.815+0000 7f803f7fe640 1 -- 172.21.15.69:0/383235860 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8044004660 con 0x55b3e8d13c40 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- 172.21.15.69:0/383235860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 msgr2=0x55b3e8d1d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f8044001eb0 tx=0x7f8044011e50 comp rx=0 tx=0).stop 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- 172.21.15.69:0/383235860 shutdown_connections 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d18db0 0x55b3e8d24700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d13c40 0x55b3e8d1d700 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 --2- 172.21.15.69:0/383235860 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d1d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- 172.21.15.69:0/383235860 >> 172.21.15.69:0/383235860 conn(0x55b3e8d0e8c0 msgr2=0x55b3e8d0eca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- 172.21.15.69:0/383235860 shutdown_connections 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- 172.21.15.69:0/383235860 wait complete. 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.816+0000 7f805e7f3740 1 -- start start 2024-05-09T09:32:45.822 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 Processor -- start 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d21b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d13c40 0x55b3e8d22090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80541a8920 con 0x55b3e8d18db0 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80541a9d90 con 0x55b3e8d13c40 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f80541a8ff0 con 0x55b3e8d13250 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805e7f3740 10 client.0 _ll_register_callbacks cb 0x7f804400bb90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d21b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805907c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d13c40 0x55b3e8d22090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44134/0 (socket says 172.21.15.69:44134) 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 -- 172.21.15.69:0/4114444296 learned_addr learned my addr 172.21.15.69:0/4114444296 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 -- 172.21.15.69:0/4114444296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 msgr2=0x55b3e8d21b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d21b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 -- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d13c40 msgr2=0x55b3e8d22090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.817+0000 7f805a07e640 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d13c40 0x55b3e8d22090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805a07e640 1 -- 172.21.15.69:0/4114444296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b3e8d23690 con 0x55b3e8d18db0 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805a07e640 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f804c006010 tx=0x7f804c0099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e295de1d2ffec5f0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f804c055070 con 0x55b3e8d18db0 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f804c041040 con 0x55b3e8d18db0 2024-05-09T09:32:45.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f8044010e90 con 0x55b3e8d18db0 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f804c031720 con 0x55b3e8d18db0 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 10 client.17172 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55b3e8d249a0 con 0x55b3e8d18db0 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8044010e90 con 0x55b3e8d18db0 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f804c02f870 con 0x55b3e8d18db0 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 10 client.17172 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 10 client.17172 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 20 client.17172 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:45.825 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d8f820 0x55b3e8d91c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:45.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 4 client.17172 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:32:45.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.818+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b3e8c7edc0 con 0x55b3e8d8f820 2024-05-09T09:32:45.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.819+0000 7f805907c640 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d8f820 0x55b3e8d91c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:45.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.819+0000 7f803d7fa640 10 client.17172 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:45.826 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:45.819+0000 7f805907c640 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d8f820 0x55b3e8d91c40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b901d0bb685e765 server_cookie=62bf2bc85eabad58 in_seq=0 out_seq=0 2024-05-09T09:32:45.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 24558 34.95 MB/sec warmup 43 sec latency 44.677 ms 2024-05-09T09:32:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f803d7fa640 1 -- 172.21.15.69:0/4114444296 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b3e8c7edc0 con 0x55b3e8d8f820 2024-05-09T09:32:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f803d7fa640 10 client.17172 handle_command_reply: tid=0 2024-05-09T09:32:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 client.17172 shutdown 2024-05-09T09:32:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 msgr2=0x55b3e8d230a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:46.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f804c006010 tx=0x7f804c0099c0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 shutdown_connections 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b3e8d18db0 0x55b3e8d230a0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b3e8d13c40 0x55b3e8d22090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b3e8d8f820 0x55b3e8d91c40 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 --2- 172.21.15.69:0/4114444296 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b3e8d13250 0x55b3e8d21b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f803d7fa640 0 client.17172 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 >> 172.21.15.69:0/4114444296 conn(0x55b3e8d0e8c0 msgr2=0x55b3e8d11840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 shutdown_connections 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.342+0000 7f805e7f3740 1 -- 172.21.15.69:0/4114444296 wait complete. 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805e7f3740 20 client.17172 trim_cache size 0 max 16384 2024-05-09T09:32:46.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 msgr2=0x7f80400a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 0x7f80400a50f0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f80500cd200 tx=0x7f80500d5550 comp rx=0 tx=0).stop 2024-05-09T09:32:46.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 msgr2=0x7f8054197a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f805000e9c0 tx=0x7f805000ee80 comp rx=0 tx=0).stop 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 shutdown_connections 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f80400a2c30 0x7f80400a50f0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8054102340 0x7f805419f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8054101980 0x7f8054197fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 --2- 172.21.15.69:0/1040888936 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8054100790 0x7f8054197a90 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.343+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 >> 172.21.15.69:0/1040888936 conn(0x7f80540fbd00 msgr2=0x7f8054104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.344+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 shutdown_connections 2024-05-09T09:32:46.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:46.344+0000 7f805bae1640 1 -- 172.21.15.69:0/1040888936 wait complete. 2024-05-09T09:32:46.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 24893 34.50 MB/sec warmup 44 sec latency 385.529 ms 2024-05-09T09:32:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:46 smithi069 ceph-mon[30123]: pgmap v786: 129 pgs: 129 active+clean; 393 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 523 op/s 2024-05-09T09:32:47.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:46 smithi163 ceph-mon[34354]: pgmap v786: 129 pgs: 129 active+clean; 393 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 523 op/s 2024-05-09T09:32:47.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:46 smithi103 ceph-mon[34421]: pgmap v786: 129 pgs: 129 active+clean; 393 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.8 MiB/s rd, 3.7 MiB/s wr, 523 op/s 2024-05-09T09:32:47.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 25622 34.41 MB/sec warmup 45 sec latency 43.146 ms 2024-05-09T09:32:48.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 26517 34.68 MB/sec warmup 46 sec latency 53.134 ms 2024-05-09T09:32:49.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:48 smithi069 ceph-mon[30123]: pgmap v787: 129 pgs: 129 active+clean; 394 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 590 op/s 2024-05-09T09:32:49.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:48 smithi163 ceph-mon[34354]: pgmap v787: 129 pgs: 129 active+clean; 394 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 590 op/s 2024-05-09T09:32:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:48 smithi103 ceph-mon[34421]: pgmap v787: 129 pgs: 129 active+clean; 394 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.1 MiB/s wr, 590 op/s 2024-05-09T09:32:49.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 27335 35.10 MB/sec warmup 47 sec latency 63.216 ms 2024-05-09T09:32:50.820 INFO:tasks.workunit.client.0.smithi069.stdout: 10 28003 35.17 MB/sec warmup 48 sec latency 33.608 ms 2024-05-09T09:32:51.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:50 smithi069 ceph-mon[30123]: pgmap v788: 129 pgs: 129 active+clean; 399 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 538 op/s 2024-05-09T09:32:51.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:50 smithi163 ceph-mon[34354]: pgmap v788: 129 pgs: 129 active+clean; 399 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 538 op/s 2024-05-09T09:32:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:50 smithi103 ceph-mon[34421]: pgmap v788: 129 pgs: 129 active+clean; 399 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.9 MiB/s rd, 3.6 MiB/s wr, 538 op/s 2024-05-09T09:32:51.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 28548 35.00 MB/sec warmup 49 sec latency 51.910 ms 2024-05-09T09:32:52.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 29378 34.94 MB/sec warmup 50 sec latency 48.909 ms 2024-05-09T09:32:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:52 smithi069 ceph-mon[30123]: pgmap v789: 129 pgs: 129 active+clean; 402 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.5 MiB/s wr, 550 op/s 2024-05-09T09:32:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:52 smithi163 ceph-mon[34354]: pgmap v789: 129 pgs: 129 active+clean; 402 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.5 MiB/s wr, 550 op/s 2024-05-09T09:32:53.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:52 smithi103 ceph-mon[34421]: pgmap v789: 129 pgs: 129 active+clean; 402 MiB data, 4.3 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.5 MiB/s wr, 550 op/s 2024-05-09T09:32:53.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 30108 35.28 MB/sec warmup 51 sec latency 164.936 ms 2024-05-09T09:32:54.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 30129 34.64 MB/sec warmup 52 sec latency 1151.924 ms 2024-05-09T09:32:55.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:55 smithi069 ceph-mon[30123]: pgmap v790: 129 pgs: 129 active+clean; 411 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 650 op/s 2024-05-09T09:32:55.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:55 smithi163 ceph-mon[34354]: pgmap v790: 129 pgs: 129 active+clean; 411 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 650 op/s 2024-05-09T09:32:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:55 smithi103 ceph-mon[34421]: pgmap v790: 129 pgs: 129 active+clean; 411 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 4.4 MiB/s wr, 650 op/s 2024-05-09T09:32:55.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 30146 34.00 MB/sec warmup 53 sec latency 1848.034 ms 2024-05-09T09:32:56.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 30330 33.55 MB/sec warmup 54 sec latency 848.706 ms 2024-05-09T09:32:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:57 smithi069 ceph-mon[30123]: pgmap v791: 129 pgs: 129 active+clean; 451 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 652 op/s 2024-05-09T09:32:57.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:57 smithi163 ceph-mon[34354]: pgmap v791: 129 pgs: 129 active+clean; 451 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 652 op/s 2024-05-09T09:32:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:57 smithi103 ceph-mon[34421]: pgmap v791: 129 pgs: 129 active+clean; 451 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 6.8 MiB/s wr, 652 op/s 2024-05-09T09:32:57.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 30728 33.25 MB/sec warmup 55 sec latency 89.824 ms 2024-05-09T09:32:58.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 31206 33.02 MB/sec warmup 56 sec latency 155.495 ms 2024-05-09T09:32:59.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:32:59 smithi163 ceph-mon[34354]: pgmap v792: 129 pgs: 129 active+clean; 520 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 831 op/s 2024-05-09T09:32:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:32:59 smithi103 ceph-mon[34421]: pgmap v792: 129 pgs: 129 active+clean; 520 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 831 op/s 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 -- 172.21.15.69:0/4018896667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 msgr2=0x7f4424109d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 --2- 172.21.15.69:0/4018896667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f4424109d00 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7f441c009ca0 tx=0x7f441c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 -- 172.21.15.69:0/4018896667 shutdown_connections 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 --2- 172.21.15.69:0/4018896667 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f4424109d00 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 --2- 172.21.15.69:0/4018896667 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 0x7f4424102f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 --2- 172.21.15.69:0/4018896667 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 0x7f4424101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 -- 172.21.15.69:0/4018896667 >> 172.21.15.69:0/4018896667 conn(0x7f44240fce50 msgr2=0x7f44240ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 -- 172.21.15.69:0/4018896667 shutdown_connections 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 -- 172.21.15.69:0/4018896667 wait complete. 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.444+0000 7f442a963640 1 Processor -- start 2024-05-09T09:32:59.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 -- start start 2024-05-09T09:32:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 0x7f4424197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 0x7f4424198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f442410d540 con 0x7f4424103470 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f442410d3c0 con 0x7f44241018c0 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f442a963640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f442410d6c0 con 0x7f4424102ab0 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38856/0 (socket says 172.21.15.69:38856) 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 -- 172.21.15.69:0/3062031472 learned_addr learned my addr 172.21.15.69:0/3062031472 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4423fff640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 0x7f4424197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f44237fe640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 0x7f4424198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 -- 172.21.15.69:0/3062031472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 msgr2=0x7f4424198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 0x7f4424198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 -- 172.21.15.69:0/3062031472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 msgr2=0x7f4424197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 0x7f4424197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 -- 172.21.15.69:0/3062031472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f442419f6e0 con 0x7f4424103470 2024-05-09T09:32:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.445+0000 7f4428f00640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f441c031ae0 tx=0x7f441c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.446+0000 7f44217fa640 1 -- 172.21.15.69:0/3062031472 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c003b70 con 0x7f4424103470 2024-05-09T09:32:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.446+0000 7f44217fa640 1 -- 172.21.15.69:0/3062031472 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441c003d10 con 0x7f4424103470 2024-05-09T09:32:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.446+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f442419f970 con 0x7f4424103470 2024-05-09T09:32:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.446+0000 7f44217fa640 1 -- 172.21.15.69:0/3062031472 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441c02fc40 con 0x7f4424103470 2024-05-09T09:32:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.446+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f442419ff10 con 0x7f4424103470 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.448+0000 7f44217fa640 1 -- 172.21.15.69:0/3062031472 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f441c004050 con 0x7f4424103470 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.448+0000 7f442d67b740 1 Processor -- start 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.448+0000 7f442d67b740 1 -- start start 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd28e670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd296240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5556bd1f2340 con 0x5556bd28ec40 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5556bd19a290 con 0x5556bd293db0 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f442d67b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5556bd1ee300 con 0x5556bd28e250 2024-05-09T09:32:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44217fa640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 0x7f440c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd296240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38860/0 (socket says 172.21.15.69:38860) 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4423fff640 1 -- 172.21.15.69:0/2652748877 learned_addr learned my addr 172.21.15.69:0/2652748877 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd28e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4428f00640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 0x7f440c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44217fa640 1 -- 172.21.15.69:0/3062031472 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f441c0f0320 con 0x7f4424103470 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 -- 172.21.15.69:0/2652748877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 msgr2=0x5556bd28e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd28e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 -- 172.21.15.69:0/2652748877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 msgr2=0x5556bd293740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 -- 172.21.15.69:0/2652748877 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bd1be8e0 con 0x5556bd293db0 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4423fff640 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:32:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f4428f00640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 0x7f440c0a5170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f441c005e30 tx=0x7f441c087760 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.449+0000 7f44237fe640 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd296240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4418009fd0 tx=0x7f44180046c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=18acde15979d8dad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f440a7fc640 1 -- 172.21.15.69:0/2652748877 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418004050 con 0x5556bd293db0 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f440a7fc640 1 -- 172.21.15.69:0/2652748877 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441800fc70 con 0x5556bd293db0 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f440a7fc640 1 -- 172.21.15.69:0/2652748877 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418003a10 con 0x5556bd293db0 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 -- 172.21.15.69:0/2652748877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 msgr2=0x5556bd296240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd296240 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f4418009fd0 tx=0x7f44180046c0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 -- 172.21.15.69:0/2652748877 shutdown_connections 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd296240 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd293740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 --2- 172.21.15.69:0/2652748877 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd28e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 -- 172.21.15.69:0/2652748877 >> 172.21.15.69:0/2652748877 conn(0x5556bd2898c0 msgr2=0x5556bd289cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 -- 172.21.15.69:0/2652748877 shutdown_connections 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.450+0000 7f442d67b740 1 -- 172.21.15.69:0/2652748877 wait complete. 2024-05-09T09:32:59.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.451+0000 7f442d67b740 1 -- start start 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.451+0000 7f442d67b740 1 Processor -- start 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.451+0000 7f442d67b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2984a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.451+0000 7f442d67b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28ec40 0x5556bd2989e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f442d67b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 0x5556bd29fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f442d67b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5556bd296d60 con 0x5556bd293db0 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f442d67b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5556bd1ad790 con 0x5556bd28e250 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f442d67b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5556bd296ee0 con 0x5556bd28ec40 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f442d67b740 10 client.0 _ll_register_callbacks cb 0x5556bd1e8a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2984a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58342/0 (socket says 172.21.15.69:58342) 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 -- 172.21.15.69:0/2767640338 learned_addr learned my addr 172.21.15.69:0/2767640338 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 0x5556bd29fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 -- 172.21.15.69:0/2767640338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28ec40 msgr2=0x5556bd2989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28ec40 0x5556bd2989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 -- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 msgr2=0x5556bd2984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 -- 172.21.15.69:0/2767640338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bcf06210 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44237fe640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 0x5556bd29fa20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f44180057c0 tx=0x7f44180046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5a9303b969a31d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44037fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4418011260 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.452+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f4418005030 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f441800fc70 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4418033050 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f442d67b740 10 client.17184 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5556bcfd2620 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441800fc70 con 0x5556bd293db0 2024-05-09T09:32:59.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f4418004cd0 con 0x5556bd293db0 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f442d67b740 10 client.17184 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.453+0000 7f442d67b740 10 client.17184 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f442d67b740 20 client.17184 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd2a42d0 0x5556bd2a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f442d67b740 4 client.17184 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5556bd1ffa70 con 0x5556bd2a42d0 2024-05-09T09:32:59.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f4423fff640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd2a42d0 0x5556bd2a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f4423fff640 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd2a42d0 0x5556bd2a66f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=65e3eb8c956cacdf server_cookie=6dd6c024a467f0f6 in_seq=0 out_seq=0 2024-05-09T09:32:59.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.454+0000 7f44037fe640 10 client.17184 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:59.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.456+0000 7f44037fe640 1 -- 172.21.15.69:0/2767640338 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5556bd1ffa70 con 0x5556bd2a42d0 2024-05-09T09:32:59.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.456+0000 7f44037fe640 10 client.17184 handle_command_reply: tid=0 2024-05-09T09:32:59.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.456+0000 7f442d67b740 1 client.17184 shutdown 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 msgr2=0x5556bd29fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 0x5556bd29fa20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7f44180057c0 tx=0x7f44180046c0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 shutdown_connections 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd293db0 0x5556bd29fa20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd2a42d0 0x5556bd2a66f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28ec40 0x5556bd2989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 --2- 172.21.15.69:0/2767640338 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 >> 172.21.15.69:0/2767640338 conn(0x5556bd2898c0 msgr2=0x5556bd288a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f44037fe640 0 client.17184 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 shutdown_connections 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 1 -- 172.21.15.69:0/2767640338 wait complete. 2024-05-09T09:32:59.461 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.457+0000 7f442d67b740 20 client.17184 trim_cache size 0 max 16384 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.476+0000 7f442d67b740 1 Processor -- start 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.476+0000 7f442d67b740 1 -- start start 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.476+0000 7f442d67b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2a00b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.476+0000 7f442d67b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.476+0000 7f442d67b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 0x5556bd289ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f442d67b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44241accf0 con 0x5556bd28ec40 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f442d67b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44241aa530 con 0x5556bd28e250 2024-05-09T09:32:59.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f442d67b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44241a8380 con 0x5556bd293db0 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38866/0 (socket says 172.21.15.69:38866) 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 -- 172.21.15.69:0/1724966294 learned_addr learned my addr 172.21.15.69:0/1724966294 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f44237fe640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2a00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f44237fe640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 0x5556bd289ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 -- 172.21.15.69:0/1724966294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 msgr2=0x5556bd289ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 0x5556bd289ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 -- 172.21.15.69:0/1724966294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 msgr2=0x5556bd2a00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2a00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 -- 172.21.15.69:0/1724966294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5556bd28f910 con 0x5556bd28ec40 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f44237fe640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 0x5556bd289ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4423fff640 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5556bd28a830 tx=0x7f4414010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a18232d94faa7095 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4409ffb640 1 -- 172.21.15.69:0/1724966294 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414004220 con 0x5556bd28ec40 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4409ffb640 1 -- 172.21.15.69:0/1724966294 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441401c930 con 0x5556bd28ec40 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.477+0000 7f4409ffb640 1 -- 172.21.15.69:0/1724966294 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f4414003e10 con 0x5556bd28ec40 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- 172.21.15.69:0/1724966294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 msgr2=0x5556bd29f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x5556bd28a830 tx=0x7f4414010800 comp rx=0 tx=0).stop 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- 172.21.15.69:0/1724966294 shutdown_connections 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd293db0 0x5556bd289ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29f940 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 --2- 172.21.15.69:0/1724966294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd28e250 0x5556bd2a00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- 172.21.15.69:0/1724966294 >> 172.21.15.69:0/1724966294 conn(0x5556bd2898c0 msgr2=0x5556bd293470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- 172.21.15.69:0/1724966294 shutdown_connections 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- 172.21.15.69:0/1724966294 wait complete. 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.478+0000 7f442d67b740 1 -- start start 2024-05-09T09:32:59.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 Processor -- start 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd29f660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd29c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44241a89f0 con 0x5556bd28ec40 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44241a9e60 con 0x5556bd293db0 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44241a90c0 con 0x5556bd28e250 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f442d67b740 10 client.0 _ll_register_callbacks cb 0x7f4418009de0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38880/0 (socket says 172.21.15.69:38880) 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 -- 172.21.15.69:0/41580829 learned_addr learned my addr 172.21.15.69:0/41580829 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f44237fe640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd29c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f44237fe640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd29f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 -- 172.21.15.69:0/41580829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 msgr2=0x5556bd29f660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd29f660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 -- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 msgr2=0x5556bd29c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd29c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 -- 172.21.15.69:0/41580829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5556bd29cd10 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f44237fe640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd29c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f4423fff640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4414009730 tx=0x7f4414009f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe4db5f6469793be server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.479+0000 7f44037fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f441401c450 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f441401c5f0 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f441401c930 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f4414036050 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 10 client.17190 fetch_fsmap learned FSMap version 43 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5556bd309870 con 0x5556bd28ec40 2024-05-09T09:32:59.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f441401c930 con 0x5556bd28ec40 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f44140046c0 con 0x5556bd28ec40 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 10 client.17190 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 10 client.17190 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 20 client.17190 populate_metadata read hostname 'smithi069' 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd30a2a0 0x5556bd30c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 4 client.17190 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5556bd1f9dc0 con 0x5556bd30a2a0 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.480+0000 7f44237fe640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd30a2a0 0x5556bd30c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.481+0000 7f44037fe640 10 client.17190 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:59.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.481+0000 7f44237fe640 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd30a2a0 0x5556bd30c6c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=977a69c3ed844594 server_cookie=f302c27f2857011d in_seq=0 out_seq=0 2024-05-09T09:32:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:32:59 smithi069 ceph-mon[30123]: pgmap v792: 129 pgs: 129 active+clean; 520 MiB data, 4.4 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 16 MiB/s wr, 831 op/s 2024-05-09T09:32:59.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 31636 32.86 MB/sec warmup 57 sec latency 312.899 ms 2024-05-09T09:32:59.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f44037fe640 1 -- 172.21.15.69:0/41580829 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5556bd1f9dc0 con 0x5556bd30a2a0 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f44037fe640 10 client.17190 handle_command_reply: tid=0 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 client.17190 shutdown 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 msgr2=0x5556bd29c290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7f4414009730 tx=0x7f4414009f40 comp rx=0 tx=0).stop 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 shutdown_connections 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5556bd293db0 0x5556bd29c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5556bd28ec40 0x5556bd29c290 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5556bd30a2a0 0x5556bd30c6c0 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 --2- 172.21.15.69:0/41580829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5556bd28e250 0x5556bd29f660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f44037fe640 0 client.17190 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 >> 172.21.15.69:0/41580829 conn(0x5556bd2898c0 msgr2=0x5556bd28fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 shutdown_connections 2024-05-09T09:32:59.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.986+0000 7f442d67b740 1 -- 172.21.15.69:0/41580829 wait complete. 2024-05-09T09:32:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.987+0000 7f442d67b740 20 client.17190 trim_cache size 0 max 16384 2024-05-09T09:32:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:32:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:32:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:32:59.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.987+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 msgr2=0x7f440c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.987+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 0x7f440c0a5170 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f441c005e30 tx=0x7f441c087760 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.987+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 msgr2=0x7f442419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.987+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7f441c031ae0 tx=0x7f441c031ee0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 shutdown_connections 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f440c0a2cb0 0x7f440c0a5170 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f4424103470 0x7f442419f070 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f4424102ab0 0x7f4424198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 --2- 172.21.15.69:0/3062031472 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44241018c0 0x7f4424197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 >> 172.21.15.69:0/3062031472 conn(0x7f44240fce50 msgr2=0x7f4424105ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 shutdown_connections 2024-05-09T09:32:59.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:32:59.988+0000 7f442a963640 1 -- 172.21.15.69:0/3062031472 wait complete. 2024-05-09T09:33:00.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 32416 33.13 MB/sec warmup 58 sec latency 364.618 ms 2024-05-09T09:33:01.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:01 smithi163 ceph-mon[34354]: pgmap v793: 129 pgs: 129 active+clean; 523 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 18 MiB/s wr, 864 op/s 2024-05-09T09:33:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:01 smithi103 ceph-mon[34421]: pgmap v793: 129 pgs: 129 active+clean; 523 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 18 MiB/s wr, 864 op/s 2024-05-09T09:33:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:01 smithi069 ceph-mon[30123]: pgmap v793: 129 pgs: 129 active+clean; 523 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 18 MiB/s wr, 864 op/s 2024-05-09T09:33:01.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 33082 33.25 MB/sec warmup 59 sec latency 53.557 ms 2024-05-09T09:33:02.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 33568 33.09 MB/sec warmup 60 sec latency 78.208 ms 2024-05-09T09:33:03.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:03 smithi163 ceph-mon[34354]: pgmap v794: 129 pgs: 129 active+clean; 490 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-05-09T09:33:03.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:03 smithi103 ceph-mon[34421]: pgmap v794: 129 pgs: 129 active+clean; 490 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-05-09T09:33:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:03 smithi069 ceph-mon[30123]: pgmap v794: 129 pgs: 129 active+clean; 490 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 18 MiB/s wr, 842 op/s 2024-05-09T09:33:03.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 34008 32.92 MB/sec warmup 61 sec latency 110.454 ms 2024-05-09T09:33:04.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 34813 32.88 MB/sec warmup 62 sec latency 57.832 ms 2024-05-09T09:33:05.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:05 smithi163 ceph-mon[34354]: pgmap v795: 129 pgs: 129 active+clean; 437 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-05-09T09:33:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:05 smithi103 ceph-mon[34421]: pgmap v795: 129 pgs: 129 active+clean; 437 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-05-09T09:33:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:05 smithi069 ceph-mon[30123]: pgmap v795: 129 pgs: 129 active+clean; 437 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 19 MiB/s wr, 1.04k op/s 2024-05-09T09:33:05.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 35571 33.17 MB/sec warmup 63 sec latency 58.248 ms 2024-05-09T09:33:06.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 36384 33.38 MB/sec warmup 64 sec latency 76.278 ms 2024-05-09T09:33:07.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:07 smithi163 ceph-mon[34354]: pgmap v796: 129 pgs: 129 active+clean; 443 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-05-09T09:33:07.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:07 smithi103 ceph-mon[34421]: pgmap v796: 129 pgs: 129 active+clean; 443 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-05-09T09:33:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:07 smithi069 ceph-mon[30123]: pgmap v796: 129 pgs: 129 active+clean; 443 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 18 MiB/s wr, 1.06k op/s 2024-05-09T09:33:07.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 36928 33.41 MB/sec warmup 65 sec latency 45.006 ms 2024-05-09T09:33:08.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 37431 33.23 MB/sec warmup 66 sec latency 64.848 ms 2024-05-09T09:33:09.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:09 smithi163 ceph-mon[34354]: pgmap v797: 129 pgs: 129 active+clean; 450 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-05-09T09:33:09.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:09 smithi103 ceph-mon[34421]: pgmap v797: 129 pgs: 129 active+clean; 450 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-05-09T09:33:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:09 smithi069 ceph-mon[30123]: pgmap v797: 129 pgs: 129 active+clean; 450 MiB data, 4.5 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 16 MiB/s wr, 1.01k op/s 2024-05-09T09:33:09.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 38083 33.13 MB/sec warmup 67 sec latency 49.275 ms 2024-05-09T09:33:10.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 39051 33.41 MB/sec warmup 68 sec latency 42.505 ms 2024-05-09T09:33:11.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:11 smithi163 ceph-mon[34354]: pgmap v798: 129 pgs: 129 active+clean; 454 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 7.0 MiB/s wr, 839 op/s 2024-05-09T09:33:11.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:11 smithi103 ceph-mon[34421]: pgmap v798: 129 pgs: 129 active+clean; 454 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 7.0 MiB/s wr, 839 op/s 2024-05-09T09:33:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:11 smithi069 ceph-mon[30123]: pgmap v798: 129 pgs: 129 active+clean; 454 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 7.0 MiB/s wr, 839 op/s 2024-05-09T09:33:11.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 39973 33.74 MB/sec warmup 69 sec latency 41.142 ms 2024-05-09T09:33:12.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 40642 33.90 MB/sec warmup 70 sec latency 45.957 ms 2024-05-09T09:33:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.100+0000 7f81d0109640 1 -- 172.21.15.69:0/1283793347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8100790 msgr2=0x7f81c8100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.100+0000 7f81d0109640 1 --2- 172.21.15.69:0/1283793347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8100790 0x7f81c8100bb0 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7f81c4009900 tx=0x7f81c402a410 comp rx=0 tx=0).stop 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- 172.21.15.69:0/1283793347 shutdown_connections 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- 172.21.15.69:0/1283793347 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8102340 0x7f81c8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- 172.21.15.69:0/1283793347 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8101980 0x7f81c8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- 172.21.15.69:0/1283793347 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8100790 0x7f81c8100bb0 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- 172.21.15.69:0/1283793347 >> 172.21.15.69:0/1283793347 conn(0x7f81c80fbd00 msgr2=0x7f81c80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- 172.21.15.69:0/1283793347 shutdown_connections 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- 172.21.15.69:0/1283793347 wait complete. 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 Processor -- start 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- start start 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8100790 0x7f81c8069a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8101980 0x7f81c819c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.103 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810c430 con 0x7f81c8102340 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810c2b0 con 0x7f81c8100790 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.101+0000 7f81d0109640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81c810c5b0 con 0x7f81c8101980 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40182/0 (socket says 172.21.15.69:40182) 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 -- 172.21.15.69:0/1795660924 learned_addr learned my addr 172.21.15.69:0/1795660924 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8101980 0x7f81c819c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81cdea5640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8100790 0x7f81c8069a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 -- 172.21.15.69:0/1795660924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8101980 msgr2=0x7f81c819c8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8101980 0x7f81c819c8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 -- 172.21.15.69:0/1795660924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8100790 msgr2=0x7f81c8069a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8100790 0x7f81c8069a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 -- 172.21.15.69:0/1795660924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f81c819f7d0 con 0x7f81c8102340 2024-05-09T09:33:13.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f81c000eaa0 tx=0x7f81c000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81beffd640 1 -- 172.21.15.69:0/1795660924 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c000cdb0 con 0x7f81c8102340 2024-05-09T09:33:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f81c819fa60 con 0x7f81c8102340 2024-05-09T09:33:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81beffd640 1 -- 172.21.15.69:0/1795660924 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c0005020 con 0x7f81c8102340 2024-05-09T09:33:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81beffd640 1 -- 172.21.15.69:0/1795660924 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c00116b0 con 0x7f81c8102340 2024-05-09T09:33:13.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.102+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f81c819ffb0 con 0x7f81c8102340 2024-05-09T09:33:13.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.105+0000 7f81beffd640 1 -- 172.21.15.69:0/1795660924 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f81c0005560 con 0x7f81c8102340 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.105+0000 7f81d2e13740 1 Processor -- start 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 -- start start 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5ee640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa5f3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562dfa547340 con 0x562dfa5f1620 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562dfa4ef290 con 0x562dfa5e2240 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81d2e13740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562dfa543300 con 0x562dfa5eeb80 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cd6a4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5ee640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:59378/0 (socket says 172.21.15.69:59378) 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 -- 172.21.15.69:0/2629420684 learned_addr learned my addr 172.21.15.69:0/2629420684 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:13.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa5f3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 -- 172.21.15.69:0/2629420684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 msgr2=0x562dfa5ee640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5ee640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 -- 172.21.15.69:0/2629420684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 msgr2=0x562dfa5f3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa5f3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81cdea5640 1 -- 172.21.15.69:0/2629420684 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dfa5138e0 con 0x562dfa5eeb80 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.106+0000 7f81beffd640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 0x7f81b40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 0x7f81b40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81cdea5640 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f81c4006010 tx=0x7f81c4004170 comp rx=0 tx=0).ready entity=mon.1 client_cookie=badff87520e4bf58 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81beffd640 1 -- 172.21.15.69:0/1795660924 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f81c00cf590 con 0x7f81c8102340 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81b3fff640 1 -- 172.21.15.69:0/2629420684 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4035040 con 0x562dfa5eeb80 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81b3fff640 1 -- 172.21.15.69:0/2629420684 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c4005520 con 0x562dfa5eeb80 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.107+0000 7f81b3fff640 1 -- 172.21.15.69:0/2629420684 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c40318b0 con 0x562dfa5eeb80 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- 172.21.15.69:0/2629420684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 msgr2=0x562dfa5f0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 0x7f81b40a50f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x562dfa5e1d70 tx=0x7f81b8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f81c4006010 tx=0x7f81c4004170 comp rx=0 tx=0).stop 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- 172.21.15.69:0/2629420684 shutdown_connections 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa5f3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f0f90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 --2- 172.21.15.69:0/2629420684 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5ee640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- 172.21.15.69:0/2629420684 >> 172.21.15.69:0/2629420684 conn(0x562dfa5e2cd0 msgr2=0x562dfa5e30d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- 172.21.15.69:0/2629420684 shutdown_connections 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- 172.21.15.69:0/2629420684 wait complete. 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 -- start start 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.108+0000 7f81d2e13740 1 Processor -- start 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f6170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5f1620 0x562dfa5fd1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562dfa5f4570 con 0x562dfa5e2240 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562dfa502790 con 0x562dfa5f1620 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40200/0 (socket says 172.21.15.69:40200) 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 -- 172.21.15.69:0/1832423499 learned_addr learned my addr 172.21.15.69:0/1832423499 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562dfa5f46f0 con 0x562dfa5eeb80 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f6170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81d2e13740 10 client.0 _ll_register_callbacks cb 0x562dfa53da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5f1620 0x562dfa5fd1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 -- 172.21.15.69:0/1832423499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 msgr2=0x562dfa5f6170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f6170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 -- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5f1620 msgr2=0x562dfa5fd1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5f1620 0x562dfa5fd1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cdea5640 1 -- 172.21.15.69:0/1832423499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dfa25b210 con 0x562dfa5e2240 2024-05-09T09:33:13.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.109+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81cdea5640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f81c402aa00 tx=0x7f81c4004510 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e812c43eb0005c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81c4035050 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f81c4004980 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81c4005520 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f81c4004df0 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81d2e13740 10 client.17202 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.110+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562dfa327620 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81c4005520 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f81c40035f0 con 0x562dfa5e2240 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 10 client.17202 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 10 client.17202 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 20 client.17202 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:13.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa601310 0x562dfa603700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 4 client.17202 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:33:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562dfa554a70 con 0x562dfa601310 2024-05-09T09:33:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.111+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa601310 0x562dfa603700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.112+0000 7f81b17fa640 10 client.17202 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:13.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.112+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa601310 0x562dfa603700 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed2acc2774daff54 server_cookie=cc5e3048c5f27093 in_seq=0 out_seq=0 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.117+0000 7f81b17fa640 1 -- 172.21.15.69:0/1832423499 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562dfa554a70 con 0x562dfa601310 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.117+0000 7f81b17fa640 10 client.17202 handle_command_reply: tid=0 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.117+0000 7f81d2e13740 1 client.17202 shutdown 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.117+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 msgr2=0x562dfa5f5c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.117+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f81c402aa00 tx=0x7f81c4004510 comp rx=0 tx=0).stop 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 shutdown_connections 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5f1620 0x562dfa5fd1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa601310 0x562dfa603700 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5f6170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81b17fa640 0 client.17202 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 --2- 172.21.15.69:0/1832423499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5e2240 0x562dfa5f5c30 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 >> 172.21.15.69:0/1832423499 conn(0x562dfa5e2cd0 msgr2=0x562dfa5ecec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 shutdown_connections 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 1 -- 172.21.15.69:0/1832423499 wait complete. 2024-05-09T09:33:13.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.118+0000 7f81d2e13740 20 client.17202 trim_cache size 0 max 16384 2024-05-09T09:33:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 Processor -- start 2024-05-09T09:33:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 -- start start 2024-05-09T09:33:13.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa607fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5fd170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81aab10 con 0x562dfa5f1620 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81a8350 con 0x562dfa5e2240 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81d2e13740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81a61a0 con 0x562dfa5eeb80 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81ce6a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5fd170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81ce6a6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40208/0 (socket says 172.21.15.69:40208) 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81ce6a6640 1 -- 172.21.15.69:0/3921333526 learned_addr learned my addr 172.21.15.69:0/3921333526 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cdea5640 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa607fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 -- 172.21.15.69:0/3921333526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 msgr2=0x562dfa607fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa607fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 -- 172.21.15.69:0/3921333526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 msgr2=0x562dfa608500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.135 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81cd6a4640 1 -- 172.21.15.69:0/3921333526 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562dfa5e3a40 con 0x562dfa5eeb80 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.134+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.135+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5fd170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x562dfa5f7110 tx=0x7f81b80046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6449e5304da57efd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.135+0000 7f81b37fe640 1 -- 172.21.15.69:0/3921333526 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004360 con 0x562dfa5eeb80 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.135+0000 7f81b37fe640 1 -- 172.21.15.69:0/3921333526 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81b802e530 con 0x562dfa5eeb80 2024-05-09T09:33:13.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.135+0000 7f81b37fe640 1 -- 172.21.15.69:0/3921333526 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b802d960 con 0x562dfa5eeb80 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- 172.21.15.69:0/3921333526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 msgr2=0x562dfa5fd170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5fd170 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x562dfa5f7110 tx=0x7f81b80046c0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- 172.21.15.69:0/3921333526 shutdown_connections 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5f1620 0x562dfa608500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5eeb80 0x562dfa5fd170 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 --2- 172.21.15.69:0/3921333526 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa607fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- 172.21.15.69:0/3921333526 >> 172.21.15.69:0/3921333526 conn(0x562dfa5e2cd0 msgr2=0x562dfa5e30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- 172.21.15.69:0/3921333526 shutdown_connections 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- 172.21.15.69:0/3921333526 wait complete. 2024-05-09T09:33:13.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 -- start start 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.136+0000 7f81d2e13740 1 Processor -- start 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5f9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 0x562dfa665830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81a6810 con 0x562dfa5eeb80 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81a7c80 con 0x562dfa5e2240 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f81c81a6ee0 con 0x562dfa5f1620 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81d2e13740 10 client.0 _ll_register_callbacks cb 0x7f81b8009e00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40214/0 (socket says 172.21.15.69:40214) 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 -- 172.21.15.69:0/76787927 learned_addr learned my addr 172.21.15.69:0/76787927 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 0x562dfa665830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cdea5640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5f9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 -- 172.21.15.69:0/76787927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 msgr2=0x562dfa665830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 0x562dfa665830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 -- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 msgr2=0x562dfa5f9e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5f9e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 -- 172.21.15.69:0/76787927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562dfa5fd420 con 0x562dfa5eeb80 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81ce6a6640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 0x562dfa665830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81cd6a4640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f81b80119c0 tx=0x7f81b8011f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ba1c8a0c66ebf89 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81b17fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.137+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f81b8004360 con 0x562dfa5eeb80 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f81b802e6e0 con 0x562dfa5eeb80 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f81b802e530 con 0x562dfa5eeb80 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f81b804a050 con 0x562dfa5eeb80 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 10 client.17208 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:13.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x562dfa666da0 con 0x562dfa5eeb80 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f81b802e530 con 0x562dfa5eeb80 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f81b80049e0 con 0x562dfa5eeb80 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 10 client.17208 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 10 client.17208 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 20 client.17208 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa667750 0x562dfa669b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 4 client.17208 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562dfa54edc0 con 0x562dfa667750 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.138+0000 7f81cdea5640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa667750 0x562dfa669b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.139+0000 7f81b17fa640 10 client.17208 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:13.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.139+0000 7f81cdea5640 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa667750 0x562dfa669b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab7a24b29218b02e server_cookie=761d6a8b3207f03a in_seq=0 out_seq=0 2024-05-09T09:33:13.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:13 smithi163 ceph-mon[34354]: pgmap v799: 129 pgs: 129 active+clean; 457 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 678 op/s 2024-05-09T09:33:13.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:13 smithi103 ceph-mon[34421]: pgmap v799: 129 pgs: 129 active+clean; 457 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 678 op/s 2024-05-09T09:33:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:13 smithi069 ceph-mon[30123]: pgmap v799: 129 pgs: 129 active+clean; 457 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 3.3 MiB/s wr, 678 op/s 2024-05-09T09:33:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81b17fa640 1 -- 172.21.15.69:0/76787927 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562dfa54edc0 con 0x562dfa667750 2024-05-09T09:33:13.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81b17fa640 10 client.17208 handle_command_reply: tid=0 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 client.17208 shutdown 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 msgr2=0x562dfa5fa350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f81b80119c0 tx=0x7f81b8011f80 comp rx=0 tx=0).stop 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 shutdown_connections 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562dfa667750 0x562dfa669b70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562dfa5f1620 0x562dfa665830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562dfa5eeb80 0x562dfa5fa350 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 --2- 172.21.15.69:0/76787927 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562dfa5e2240 0x562dfa5f9e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 >> 172.21.15.69:0/76787927 conn(0x562dfa5e2cd0 msgr2=0x562dfa5f5220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81b17fa640 0 client.17208 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 shutdown_connections 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.696+0000 7f81d2e13740 1 -- 172.21.15.69:0/76787927 wait complete. 2024-05-09T09:33:13.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d2e13740 20 client.17208 trim_cache size 0 max 16384 2024-05-09T09:33:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:33:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:33:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:33:13.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 msgr2=0x7f81b40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 0x7f81b40a50f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x562dfa5e1d70 tx=0x7f81b8005f90 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 msgr2=0x7f81c819eec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f81c000eaa0 tx=0x7f81c000ef60 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 shutdown_connections 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f81b40a2c30 0x7f81b40a50f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f81c8102340 0x7f81c819eec0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f81c8101980 0x7f81c819c8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 --2- 172.21.15.69:0/1795660924 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f81c8100790 0x7f81c8069a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:13.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 >> 172.21.15.69:0/1795660924 conn(0x7f81c80fbd00 msgr2=0x7f81c8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 shutdown_connections 2024-05-09T09:33:13.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:13.697+0000 7f81d0109640 1 -- 172.21.15.69:0/1795660924 wait complete. 2024-05-09T09:33:13.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 40993 33.60 MB/sec warmup 71 sec latency 387.970 ms 2024-05-09T09:33:14.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 41151 33.24 MB/sec warmup 72 sec latency 679.735 ms 2024-05-09T09:33:15.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:15 smithi163 ceph-mon[34354]: pgmap v800: 129 pgs: 129 active+clean; 464 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 639 op/s 2024-05-09T09:33:15.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:15 smithi103 ceph-mon[34421]: pgmap v800: 129 pgs: 129 active+clean; 464 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 639 op/s 2024-05-09T09:33:15.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:15 smithi069 ceph-mon[30123]: pgmap v800: 129 pgs: 129 active+clean; 464 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 3.7 MiB/s wr, 639 op/s 2024-05-09T09:33:15.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 41154 32.78 MB/sec warmup 73 sec latency 1679.780 ms 2024-05-09T09:33:16.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:16 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:33:16.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:16 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:33:16.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:16 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:33:16.821 INFO:tasks.workunit.client.0.smithi069.stdout: 10 41260 32.47 MB/sec warmup 74 sec latency 2081.982 ms 2024-05-09T09:33:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:17 smithi163 ceph-mon[34354]: pgmap v801: 129 pgs: 129 active+clean; 491 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 610 op/s 2024-05-09T09:33:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:33:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:33:17.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:33:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:17 smithi103 ceph-mon[34421]: pgmap v801: 129 pgs: 129 active+clean; 491 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 610 op/s 2024-05-09T09:33:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:33:17.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:33:17.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:33:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:17 smithi069 ceph-mon[30123]: pgmap v801: 129 pgs: 129 active+clean; 491 MiB data, 4.6 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 5.9 MiB/s wr, 610 op/s 2024-05-09T09:33:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:33:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:33:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:33:17.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 41466 32.14 MB/sec warmup 75 sec latency 596.488 ms 2024-05-09T09:33:18.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 41774 31.93 MB/sec warmup 76 sec latency 137.000 ms 2024-05-09T09:33:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:19 smithi103 ceph-mon[34421]: pgmap v802: 129 pgs: 129 active+clean; 572 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 746 op/s 2024-05-09T09:33:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:19 smithi069 ceph-mon[30123]: pgmap v802: 129 pgs: 129 active+clean; 572 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 746 op/s 2024-05-09T09:33:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:19 smithi163 ceph-mon[34354]: pgmap v802: 129 pgs: 129 active+clean; 572 MiB data, 4.7 GiB used, 1.0 TiB / 1.0 TiB avail; 12 MiB/s rd, 14 MiB/s wr, 746 op/s 2024-05-09T09:33:19.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 42278 31.73 MB/sec warmup 77 sec latency 68.319 ms 2024-05-09T09:33:20.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 43168 32.00 MB/sec warmup 78 sec latency 41.465 ms 2024-05-09T09:33:21.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:21 smithi103 ceph-mon[34421]: pgmap v803: 129 pgs: 129 active+clean; 578 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 17 MiB/s wr, 848 op/s 2024-05-09T09:33:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:21 smithi069 ceph-mon[30123]: pgmap v803: 129 pgs: 129 active+clean; 578 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 17 MiB/s wr, 848 op/s 2024-05-09T09:33:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:21 smithi163 ceph-mon[34354]: pgmap v803: 129 pgs: 129 active+clean; 578 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 17 MiB/s wr, 848 op/s 2024-05-09T09:33:21.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 43893 32.16 MB/sec warmup 79 sec latency 105.329 ms 2024-05-09T09:33:22.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 44578 32.34 MB/sec warmup 80 sec latency 48.204 ms 2024-05-09T09:33:23.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:23 smithi103 ceph-mon[34421]: pgmap v804: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:33:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:23 smithi069 ceph-mon[30123]: pgmap v804: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:33:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:23 smithi163 ceph-mon[34354]: pgmap v804: 129 pgs: 129 active+clean; 540 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:33:23.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 44944 32.11 MB/sec warmup 81 sec latency 180.913 ms 2024-05-09T09:33:24.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 45509 31.97 MB/sec warmup 82 sec latency 50.070 ms 2024-05-09T09:33:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:25 smithi103 ceph-mon[34421]: pgmap v805: 129 pgs: 129 active+clean; 462 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 1.07k op/s 2024-05-09T09:33:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:25 smithi069 ceph-mon[30123]: pgmap v805: 129 pgs: 129 active+clean; 462 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 1.07k op/s 2024-05-09T09:33:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:25 smithi163 ceph-mon[34354]: pgmap v805: 129 pgs: 129 active+clean; 462 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 18 MiB/s wr, 1.07k op/s 2024-05-09T09:33:25.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 46359 32.10 MB/sec warmup 83 sec latency 50.206 ms 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 -- 172.21.15.69:0/2608855756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 msgr2=0x7fe0a0108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/2608855756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a0108c80 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7fe098009cd0 tx=0x7fe09802a8e0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 -- 172.21.15.69:0/2608855756 shutdown_connections 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/2608855756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a0108c80 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/2608855756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 0x7fe0a0100ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/2608855756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 0x7fe0a01021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.803 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 -- 172.21.15.69:0/2608855756 >> 172.21.15.69:0/2608855756 conn(0x7fe0a00fb490 msgr2=0x7fe0a00fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 -- 172.21.15.69:0/2608855756 shutdown_connections 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.800+0000 7fe0a49b7640 1 -- 172.21.15.69:0/2608855756 wait complete. 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 Processor -- start 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 -- start start 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 0x7fe0a01004e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 0x7fe0a019ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c410 con 0x7fe0a01013e0 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c290 con 0x7fe0a0100a20 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe0a49b7640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a010c590 con 0x7fe0a0101d80 2024-05-09T09:33:26.804 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35640/0 (socket says 172.21.15.69:35640) 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 -- 172.21.15.69:0/1205890132 learned_addr learned my addr 172.21.15.69:0/1205890132 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 0x7fe0a019ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 0x7fe0a01004e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 -- 172.21.15.69:0/1205890132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 msgr2=0x7fe0a019ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 0x7fe0a019ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 -- 172.21.15.69:0/1205890132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 msgr2=0x7fe0a01004e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.801+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 0x7fe0a01004e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe09dd9b640 1 -- 172.21.15.69:0/1205890132 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe0a019f6e0 con 0x7fe0a01013e0 2024-05-09T09:33:26.805 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe09400e9f0 tx=0x7fe09400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe08f7fe640 1 -- 172.21.15.69:0/1205890132 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09400cd80 con 0x7fe0a01013e0 2024-05-09T09:33:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe0a019f9d0 con 0x7fe0a01013e0 2024-05-09T09:33:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe0a019ff70 con 0x7fe0a01013e0 2024-05-09T09:33:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe08f7fe640 1 -- 172.21.15.69:0/1205890132 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe094010910 con 0x7fe0a01013e0 2024-05-09T09:33:26.806 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.802+0000 7fe08f7fe640 1 -- 172.21.15.69:0/1205890132 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe094018980 con 0x7fe0a01013e0 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.804+0000 7fe08f7fe640 1 -- 172.21.15.69:0/1205890132 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe094010ac0 con 0x7fe0a01013e0 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.804+0000 7fe0a77a6740 1 Processor -- start 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 -- start start 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 0x56519647c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196484240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.813 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5651963e0340 con 0x565196481db0 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x565196388290 con 0x56519647c250 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5651963dc300 con 0x56519647cc40 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09ed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 0x56519647c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196484240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:55306/0 (socket says 172.21.15.69:55306) 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 -- 172.21.15.69:0/1637723801 learned_addr learned my addr 172.21.15.69:0/1637723801 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe08f7fe640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 0x7fe0880a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 -- 172.21.15.69:0/1637723801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 msgr2=0x56519647c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 0x56519647c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 -- 172.21.15.69:0/1637723801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 msgr2=0x565196484240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196484240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.814 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09e59c640 1 -- 172.21.15.69:0/1637723801 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5651963ac8e0 con 0x56519647cc40 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe08f7fe640 1 -- 172.21.15.69:0/1205890132 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe0940cf790 con 0x7fe0a01013e0 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 0x56519647c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.805+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 0x7fe0880a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.806+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe09000bcb0 tx=0x7fe0900107a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=fd20867bd03c29a0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.806+0000 7fe08cff9640 1 -- 172.21.15.69:0/1637723801 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe090005020 con 0x56519647cc40 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.806+0000 7fe08cff9640 1 -- 172.21.15.69:0/1637723801 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0900047a0 con 0x56519647cc40 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.806+0000 7fe08cff9640 1 -- 172.21.15.69:0/1637723801 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0900117c0 con 0x56519647cc40 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1637723801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 msgr2=0x565196481740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 0x7fe0880a50f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x56519647d4a0 tx=0x7fe09803e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fe09000bcb0 tx=0x7fe0900107a0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1637723801 shutdown_connections 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196484240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.815 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196481740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1637723801 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647c250 0x56519647c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1637723801 >> 172.21.15.69:0/1637723801 conn(0x5651964778c0 msgr2=0x565196477cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1637723801 shutdown_connections 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1637723801 wait complete. 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- start start 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 Processor -- start 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196486a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x565196484d60 con 0x56519647c250 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56519639b790 con 0x565196481db0 2024-05-09T09:33:26.816 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x565196484ee0 con 0x56519647cc40 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe0a77a6740 10 client.0 _ll_register_callbacks cb 0x5651963d6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35660/0 (socket says 172.21.15.69:35660) 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09e59c640 1 -- 172.21.15.69:0/1069898459 learned_addr learned my addr 172.21.15.69:0/1069898459 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196486a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.807+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 -- 172.21.15.69:0/1069898459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 msgr2=0x565196486a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196486a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 -- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 msgr2=0x56519648da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 -- 172.21.15.69:0/1069898459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5651960f4210 con 0x56519647c250 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196486a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe09e59c640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b5f0 tx=0x7fe09000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=49ae4c4dfd6f4ab9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.817 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09002b060 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe0900047a0 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe090004b60 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe090032050 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe0a77a6740 10 client.17220 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5651961c0620 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.808+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe0900047a0 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fe090011b40 con 0x56519647c250 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 10 client.17220 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 10 client.17220 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 20 client.17220 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565196491c50 0x565196494040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.818 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 4 client.17220 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5651963eda70 con 0x565196491c50 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565196491c50 0x565196494040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe085ffb640 10 client.17220 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.809+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565196491c50 0x565196494040 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc62c4947f28a572 server_cookie=a74847d257a7f17b in_seq=0 out_seq=0 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe085ffb640 1 -- 172.21.15.69:0/1069898459 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5651963eda70 con 0x565196491c50 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe085ffb640 10 client.17220 handle_command_reply: tid=0 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 client.17220 shutdown 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 msgr2=0x5651964864c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7fe09000b5f0 tx=0x7fe09000bfd0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 shutdown_connections 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x565196491c50 0x565196494040 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647cc40 0x565196486a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe085ffb640 0 client.17220 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:26.819 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/1069898459 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647c250 0x5651964864c0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 >> 172.21.15.69:0/1069898459 conn(0x5651964778c0 msgr2=0x565196476a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 shutdown_connections 2024-05-09T09:33:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.812+0000 7fe0a77a6740 1 -- 172.21.15.69:0/1069898459 wait complete. 2024-05-09T09:33:26.820 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.813+0000 7fe0a77a6740 20 client.17220 trim_cache size 0 max 16384 2024-05-09T09:33:26.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 47307 32.36 MB/sec warmup 84 sec latency 66.934 ms 2024-05-09T09:33:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 Processor -- start 2024-05-09T09:33:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 -- start start 2024-05-09T09:33:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x565196476170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647cc40 0x5651964766b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01acd10 con 0x565196481db0 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01aa550 con 0x56519647cc40 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01a83a0 con 0x56519647c250 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35662/0 (socket says 172.21.15.69:35662) 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09ed9d640 1 -- 172.21.15.69:0/2256227802 learned_addr learned my addr 172.21.15.69:0/2256227802 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09e59c640 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x565196476170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647cc40 0x5651964766b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09ed9d640 1 -- 172.21.15.69:0/2256227802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 msgr2=0x565196476170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.831+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x565196476170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe09ed9d640 1 -- 172.21.15.69:0/2256227802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647cc40 msgr2=0x5651964766b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647cc40 0x5651964766b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe09ed9d640 1 -- 172.21.15.69:0/2256227802 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x565196494670 con 0x565196481db0 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe098009cd0 tx=0x7fe098042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cb5cb7e13b0563c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe087fff640 1 -- 172.21.15.69:0/2256227802 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe0980043c0 con 0x565196481db0 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe087fff640 1 -- 172.21.15.69:0/2256227802 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe098036050 con 0x565196481db0 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.832+0000 7fe087fff640 1 -- 172.21.15.69:0/2256227802 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe098044670 con 0x565196481db0 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- 172.21.15.69:0/2256227802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 msgr2=0x565196476bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7fe098009cd0 tx=0x7fe098042000 comp rx=0 tx=0).stop 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- 172.21.15.69:0/2256227802 shutdown_connections 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x565196481db0 0x565196476bf0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56519647cc40 0x5651964766b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/2256227802 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x565196476170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- 172.21.15.69:0/2256227802 >> 172.21.15.69:0/2256227802 conn(0x5651964778c0 msgr2=0x5651964814c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:26.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- 172.21.15.69:0/2256227802 shutdown_connections 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- 172.21.15.69:0/2256227802 wait complete. 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- start start 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 Processor -- start 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x56519648a290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01a8a10 con 0x56519647cc40 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01a9e80 con 0x565196481db0 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.833+0000 7fe0a77a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe0a01a90e0 con 0x56519647c250 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe0a77a6740 10 client.0 _ll_register_callbacks cb 0x7fe09409baf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35666/0 (socket says 172.21.15.69:35666) 2024-05-09T09:33:26.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 -- 172.21.15.69:0/4169386333 learned_addr learned my addr 172.21.15.69:0/4169386333 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09e59c640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x56519648a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09ed9d640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 -- 172.21.15.69:0/4169386333 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 msgr2=0x56519648a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x56519648a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 -- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 msgr2=0x56519648b760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648b760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 -- 172.21.15.69:0/4169386333 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56519648bd50 con 0x56519647cc40 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe09dd9b640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe0940ccf10 tx=0x7fe09409bc20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5919710fc672bf1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe085ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe09409bd20 con 0x56519647cc40 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fe098036050 con 0x56519647cc40 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe09409b170 con 0x56519647cc40 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fe09409b370 con 0x56519647cc40 2024-05-09T09:33:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe0a77a6740 10 client.17226 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.834+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56519648d180 con 0x56519647cc40 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe098036050 con 0x56519647cc40 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fe094010540 con 0x56519647cc40 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 10 client.17226 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 10 client.17226 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 20 client.17226 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651964f8800 0x5651964fac20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 4 client.17226 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5651963e7dc0 con 0x5651964f8800 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe09e59c640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651964f8800 0x5651964fac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe085ffb640 10 client.17226 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:26.835+0000 7fe09e59c640 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651964f8800 0x5651964fac20 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a9dc04d4b025064 server_cookie=b4e8ed7d7cdea589 in_seq=0 out_seq=0 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.267+0000 7fe085ffb640 1 -- 172.21.15.69:0/4169386333 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5651963e7dc0 con 0x5651964f8800 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.267+0000 7fe085ffb640 10 client.17226 handle_command_reply: tid=0 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 client.17226 shutdown 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 msgr2=0x56519648a7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7fe0940ccf10 tx=0x7fe09409bc20 comp rx=0 tx=0).stop 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 shutdown_connections 2024-05-09T09:33:27.269 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x565196481db0 0x56519648b760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56519647cc40 0x56519648a7d0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5651964f8800 0x5651964fac20 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe085ffb640 0 client.17226 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 --2- 172.21.15.69:0/4169386333 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56519647c250 0x56519648a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 >> 172.21.15.69:0/4169386333 conn(0x5651964778c0 msgr2=0x56519647a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 shutdown_connections 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 1 -- 172.21.15.69:0/4169386333 wait complete. 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.268+0000 7fe0a77a6740 20 client.17226 trim_cache size 0 max 16384 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:33:27.270 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:33:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:33:27.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 msgr2=0x7fe0880a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 0x7fe0880a50f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x56519647d4a0 tx=0x7fe09803e040 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 msgr2=0x7fe0a019c7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7fe09400e9f0 tx=0x7fe09400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 shutdown_connections 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.269+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe0880a2c30 0x7fe0880a50f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.270+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe0a0101d80 0x7fe0a019ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.270+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe0a01013e0 0x7fe0a019c7b0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.270+0000 7fe0a49b7640 1 --2- 172.21.15.69:0/1205890132 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe0a0100a20 0x7fe0a01004e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.270+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 >> 172.21.15.69:0/1205890132 conn(0x7fe0a00fb490 msgr2=0x7fe0a0104cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.272+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 shutdown_connections 2024-05-09T09:33:27.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:27.272+0000 7fe0a49b7640 1 -- 172.21.15.69:0/1205890132 wait complete. 2024-05-09T09:33:27.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:27 smithi103 ceph-mon[34421]: pgmap v806: 129 pgs: 129 active+clean; 455 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 1.13k op/s 2024-05-09T09:33:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:27 smithi069 ceph-mon[30123]: pgmap v806: 129 pgs: 129 active+clean; 455 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 1.13k op/s 2024-05-09T09:33:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:27 smithi163 ceph-mon[34354]: pgmap v806: 129 pgs: 129 active+clean; 455 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 18 MiB/s wr, 1.13k op/s 2024-05-09T09:33:27.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 47779 32.41 MB/sec warmup 85 sec latency 288.161 ms 2024-05-09T09:33:28.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 48330 32.38 MB/sec warmup 86 sec latency 45.586 ms 2024-05-09T09:33:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:29 smithi103 ceph-mon[34421]: pgmap v807: 129 pgs: 129 active+clean; 461 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 988 op/s 2024-05-09T09:33:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:29 smithi069 ceph-mon[30123]: pgmap v807: 129 pgs: 129 active+clean; 461 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 988 op/s 2024-05-09T09:33:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:29 smithi163 ceph-mon[34354]: pgmap v807: 129 pgs: 129 active+clean; 461 MiB data, 4.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 16 MiB/s wr, 988 op/s 2024-05-09T09:33:29.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 48786 32.23 MB/sec warmup 87 sec latency 63.082 ms 2024-05-09T09:33:30.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 49514 32.16 MB/sec warmup 88 sec latency 79.900 ms 2024-05-09T09:33:31.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:31 smithi103 ceph-mon[34421]: pgmap v808: 129 pgs: 129 active+clean; 462 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-05-09T09:33:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:31 smithi069 ceph-mon[30123]: pgmap v808: 129 pgs: 129 active+clean; 462 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-05-09T09:33:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:31 smithi163 ceph-mon[34354]: pgmap v808: 129 pgs: 129 active+clean; 462 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.8 MiB/s wr, 815 op/s 2024-05-09T09:33:31.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 50368 32.42 MB/sec warmup 89 sec latency 92.780 ms 2024-05-09T09:33:32.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 51122 32.58 MB/sec warmup 90 sec latency 124.899 ms 2024-05-09T09:33:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:33 smithi103 ceph-mon[34421]: pgmap v809: 129 pgs: 129 active+clean; 467 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.1 MiB/s wr, 738 op/s 2024-05-09T09:33:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:33 smithi069 ceph-mon[30123]: pgmap v809: 129 pgs: 129 active+clean; 467 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.1 MiB/s wr, 738 op/s 2024-05-09T09:33:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:33 smithi163 ceph-mon[34354]: pgmap v809: 129 pgs: 129 active+clean; 467 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 5.1 MiB/s wr, 738 op/s 2024-05-09T09:33:33.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 51720 32.55 MB/sec warmup 91 sec latency 45.388 ms 2024-05-09T09:33:34.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 52241 32.51 MB/sec warmup 92 sec latency 86.013 ms 2024-05-09T09:33:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:35 smithi103 ceph-mon[34421]: pgmap v810: 129 pgs: 129 active+clean; 470 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 707 op/s 2024-05-09T09:33:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:35 smithi069 ceph-mon[30123]: pgmap v810: 129 pgs: 129 active+clean; 470 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 707 op/s 2024-05-09T09:33:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:35 smithi163 ceph-mon[34354]: pgmap v810: 129 pgs: 129 active+clean; 470 MiB data, 4.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 707 op/s 2024-05-09T09:33:35.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 53117 32.59 MB/sec warmup 93 sec latency 58.464 ms 2024-05-09T09:33:36.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 53715 32.67 MB/sec warmup 94 sec latency 153.952 ms 2024-05-09T09:33:37.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:37 smithi103 ceph-mon[34421]: pgmap v811: 129 pgs: 129 active+clean; 477 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.1 MiB/s wr, 629 op/s 2024-05-09T09:33:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:37 smithi069 ceph-mon[30123]: pgmap v811: 129 pgs: 129 active+clean; 477 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.1 MiB/s wr, 629 op/s 2024-05-09T09:33:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:37 smithi163 ceph-mon[34354]: pgmap v811: 129 pgs: 129 active+clean; 477 MiB data, 5.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.2 MiB/s rd, 4.1 MiB/s wr, 629 op/s 2024-05-09T09:33:37.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 54318 32.73 MB/sec warmup 95 sec latency 49.061 ms 2024-05-09T09:33:38.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 54836 32.54 MB/sec warmup 96 sec latency 63.349 ms 2024-05-09T09:33:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:39 smithi103 ceph-mon[34421]: pgmap v812: 129 pgs: 129 active+clean; 485 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 4.1 MiB/s wr, 585 op/s 2024-05-09T09:33:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:39 smithi069 ceph-mon[30123]: pgmap v812: 129 pgs: 129 active+clean; 485 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 4.1 MiB/s wr, 585 op/s 2024-05-09T09:33:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:39 smithi163 ceph-mon[34354]: pgmap v812: 129 pgs: 129 active+clean; 485 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 4.1 MiB/s wr, 585 op/s 2024-05-09T09:33:39.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 55552 32.65 MB/sec warmup 97 sec latency 63.233 ms 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 -- 172.21.15.69:0/3650626297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 msgr2=0x7f3434101e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 --2- 172.21.15.69:0/3650626297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434101e40 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f3428009900 tx=0x7f342802a410 comp rx=0 tx=0).stop 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 -- 172.21.15.69:0/3650626297 shutdown_connections 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 --2- 172.21.15.69:0/3650626297 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3434102380 0x7f3434108c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 --2- 172.21.15.69:0/3650626297 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434101e40 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 --2- 172.21.15.69:0/3650626297 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f34341007d0 0x7f3434100bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 -- 172.21.15.69:0/3650626297 >> 172.21.15.69:0/3650626297 conn(0x7f34340fbd40 msgr2=0x7f34340fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 -- 172.21.15.69:0/3650626297 shutdown_connections 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 -- 172.21.15.69:0/3650626297 wait complete. 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.375+0000 7f3439d42640 1 Processor -- start 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 -- start start 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34341007d0 0x7f3434104760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3434102380 0x7f34341032e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f343410c450 con 0x7f34341019c0 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f343410c2d0 con 0x7f3434102380 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3439d42640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f343410c5d0 con 0x7f34341007d0 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52928/0 (socket says 172.21.15.69:52928) 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 -- 172.21.15.69:0/1722910756 learned_addr learned my addr 172.21.15.69:0/1722910756 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f34337fe640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34341007d0 0x7f3434104760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3433fff640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3434102380 0x7f34341032e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 -- 172.21.15.69:0/1722910756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34341007d0 msgr2=0x7f3434104760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34341007d0 0x7f3434104760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 -- 172.21.15.69:0/1722910756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3434102380 msgr2=0x7f34341032e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3434102380 0x7f34341032e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 -- 172.21.15.69:0/1722910756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3434103b70 con 0x7f34341019c0 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.376+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3428009230 tx=0x7f34280042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.377+0000 7f3430ff9640 1 -- 172.21.15.69:0/1722910756 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342803f030 con 0x7f34341019c0 2024-05-09T09:33:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.377+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f34341a1b90 con 0x7f34341019c0 2024-05-09T09:33:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.377+0000 7f3430ff9640 1 -- 172.21.15.69:0/1722910756 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3428003ce0 con 0x7f34341019c0 2024-05-09T09:33:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.377+0000 7f3430ff9640 1 -- 172.21.15.69:0/1722910756 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428035070 con 0x7f34341019c0 2024-05-09T09:33:40.381 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.377+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f34341a2130 con 0x7f34341019c0 2024-05-09T09:33:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.379+0000 7f3430ff9640 1 -- 172.21.15.69:0/1722910756 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3428042030 con 0x7f34341019c0 2024-05-09T09:33:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.379+0000 7f343ca50740 1 Processor -- start 2024-05-09T09:33:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.379+0000 7f343ca50740 1 -- start start 2024-05-09T09:33:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3430ff9640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 0x7f341c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 0x56380c2fb740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2fbdb0 0x56380c2fe240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56380c25a340 con 0x56380c2f6250 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56380c202290 con 0x56380c2fbdb0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f343ca50740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56380c256300 con 0x56380c2f6c40 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f34337fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3433fff640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 0x7f341c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f34337fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52940/0 (socket says 172.21.15.69:52940) 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f34337fe640 1 -- 172.21.15.69:0/372467535 learned_addr learned my addr 172.21.15.69:0/372467535 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 0x56380c2fb740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3430ff9640 1 -- 172.21.15.69:0/1722910756 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f34280ef790 con 0x7f34341019c0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3433fff640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2fbdb0 0x56380c2fe240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 -- 172.21.15.69:0/372467535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2fbdb0 msgr2=0x56380c2fe240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2fbdb0 0x56380c2fe240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 -- 172.21.15.69:0/372467535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 msgr2=0x56380c2f6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 -- 172.21.15.69:0/372467535 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380c2268e0 con 0x56380c2f6c40 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f34337fe640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3433fff640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 0x7f341c0a5170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f342c005fd0 tx=0x7f342c009e00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.380+0000 7f3432ffd640 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 0x56380c2fb740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34280bba20 tx=0x7f342802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3ceed66dd727df server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f3419ffb640 1 -- 172.21.15.69:0/372467535 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3428049070 con 0x56380c2f6c40 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f3419ffb640 1 -- 172.21.15.69:0/372467535 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34280bbd20 con 0x56380c2f6c40 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f3419ffb640 1 -- 172.21.15.69:0/372467535 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34280c0f50 con 0x56380c2f6c40 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 -- 172.21.15.69:0/372467535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 msgr2=0x56380c2fb740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 0x56380c2fb740 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f34280bba20 tx=0x7f342802a410 comp rx=0 tx=0).stop 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 -- 172.21.15.69:0/372467535 shutdown_connections 2024-05-09T09:33:40.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2fbdb0 0x56380c2fe240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6c40 0x56380c2fb740 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 --2- 172.21.15.69:0/372467535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 -- 172.21.15.69:0/372467535 >> 172.21.15.69:0/372467535 conn(0x56380c2f18c0 msgr2=0x56380c2f1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 -- 172.21.15.69:0/372467535 shutdown_connections 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.381+0000 7f343ca50740 1 -- 172.21.15.69:0/372467535 wait complete. 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 -- start start 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 Processor -- start 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 0x56380c3000c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56380c2feb20 con 0x56380c2fbdb0 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56380c215790 con 0x56380c2f6c40 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56380c2feca0 con 0x56380c2f6250 2024-05-09T09:33:40.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f343ca50740 10 client.0 _ll_register_callbacks cb 0x56380c250a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f3433fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f34337fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 0x56380c3000c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f3433fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52946/0 (socket says 172.21.15.69:52946) 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f34337fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 0x56380c3000c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:37602/0 (socket says 172.21.15.69:37602) 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f3433fff640 1 -- 172.21.15.69:0/1077236983 learned_addr learned my addr 172.21.15.69:0/1077236983 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.382+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40302/0 (socket says 172.21.15.69:40302) 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 -- 172.21.15.69:0/1077236983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 msgr2=0x56380c3000c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 0x56380c3000c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 -- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 msgr2=0x56380c300600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 -- 172.21.15.69:0/1077236983 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380bf6e210 con 0x56380c2fbdb0 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:40.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3433fff640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f342c013790 tx=0x7f342c01d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68052943359ce94f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f342c004b20 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f342c01c040 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34280bbd20 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f342c031050 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f343ca50740 10 client.17238 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56380c03a620 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.383+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34280bbd20 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f342c028070 con 0x56380c2fbdb0 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 10 client.17238 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 10 client.17238 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 20 client.17238 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c30bd20 0x56380c30e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 4 client.17238 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:33:40.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56380c267a70 con 0x56380c30bd20 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f34337fe640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c30bd20 0x56380c30e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f34337fe640 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c30bd20 0x56380c30e1a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eea5912435252755 server_cookie=794920337a9f6e0a in_seq=0 out_seq=0 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.384+0000 7f3412ffd640 10 client.17238 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.385+0000 7f3412ffd640 1 -- 172.21.15.69:0/1077236983 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56380c267a70 con 0x56380c30bd20 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.385+0000 7f3412ffd640 10 client.17238 handle_command_reply: tid=0 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.385+0000 7f343ca50740 1 client.17238 shutdown 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 msgr2=0x56380c307640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f342c013790 tx=0x7f342c01d000 comp rx=0 tx=0).stop 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 shutdown_connections 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2fbdb0 0x56380c307640 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c300600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c30bd20 0x56380c30e1a0 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 --2- 172.21.15.69:0/1077236983 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2f6250 0x56380c3000c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f3412ffd640 0 client.17238 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:40.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 >> 172.21.15.69:0/1077236983 conn(0x56380c2f18c0 msgr2=0x56380c2f0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 shutdown_connections 2024-05-09T09:33:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 1 -- 172.21.15.69:0/1077236983 wait complete. 2024-05-09T09:33:40.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.386+0000 7f343ca50740 20 client.17238 trim_cache size 0 max 16384 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 Processor -- start 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 -- start start 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c2f8300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c307600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34341aee80 con 0x56380c2f6250 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34341ac6c0 con 0x56380c2f6c40 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.401+0000 7f343ca50740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34341aa510 con 0x56380c2fbdb0 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52962/0 (socket says 172.21.15.69:52962) 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 -- 172.21.15.69:0/1856532280 learned_addr learned my addr 172.21.15.69:0/1856532280 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f3433fff640 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c307600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f3432ffd640 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c2f8300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 -- 172.21.15.69:0/1856532280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 msgr2=0x56380c307600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c307600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 -- 172.21.15.69:0/1856532280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 msgr2=0x56380c2f8300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c2f8300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 -- 172.21.15.69:0/1856532280 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56380c2eff00 con 0x56380c2f6250 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34337fe640 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x56380c30e9e0 tx=0x7f3424005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=782410cd2f07c77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34197fa640 1 -- 172.21.15.69:0/1856532280 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424003da0 con 0x56380c2f6250 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34197fa640 1 -- 172.21.15.69:0/1856532280 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34240205d0 con 0x56380c2f6250 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.402+0000 7f34197fa640 1 -- 172.21.15.69:0/1856532280 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3424010ac0 con 0x56380c2f6250 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- 172.21.15.69:0/1856532280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 msgr2=0x56380c2f7dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x56380c30e9e0 tx=0x7f3424005f90 comp rx=0 tx=0).stop 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- 172.21.15.69:0/1856532280 shutdown_connections 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c307600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6c40 0x56380c2f8300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 --2- 172.21.15.69:0/1856532280 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6250 0x56380c2f7dc0 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- 172.21.15.69:0/1856532280 >> 172.21.15.69:0/1856532280 conn(0x56380c2f18c0 msgr2=0x56380c2f1ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- 172.21.15.69:0/1856532280 shutdown_connections 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- 172.21.15.69:0/1856532280 wait complete. 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.403+0000 7f343ca50740 1 -- start start 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 Processor -- start 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6250 0x56380c3047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c378ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f34341aab80 con 0x56380c2f6c40 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f34341abff0 con 0x56380c2f6250 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f34341ab250 con 0x56380c2fbdb0 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f343ca50740 10 client.0 _ll_register_callbacks cb 0x7f342c005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:40.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52966/0 (socket says 172.21.15.69:52966) 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 -- 172.21.15.69:0/131528455 learned_addr learned my addr 172.21.15.69:0/131528455 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f34337fe640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6250 0x56380c3047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3433fff640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c378ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 -- 172.21.15.69:0/131528455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 msgr2=0x56380c378ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c378ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 -- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6250 msgr2=0x56380c3047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6250 0x56380c3047e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 -- 172.21.15.69:0/131528455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56380c307860 con 0x56380c2f6c40 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3432ffd640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f34280b86f0 tx=0x7f34280c1220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65a7217de89b91e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3412ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:33:40.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.404+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f34280d8020 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f34280b8390 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f34240205d0 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f3428049040 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 10 client.17244 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x56380c379fd0 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f34240205d0 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f342803bc10 con 0x56380c2f6c40 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 10 client.17244 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 10 client.17244 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 20 client.17244 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c37a640 0x56380c37ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 4 client.17244 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56380c261dc0 con 0x56380c37a640 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.405+0000 7f34337fe640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c37a640 0x56380c37ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:40.411 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.406+0000 7f3412ffd640 10 client.17244 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:40.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.406+0000 7f34337fe640 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c37a640 0x56380c37ca60 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7d1ac94fdec3e9b2 server_cookie=92df783e4f231fc4 in_seq=0 out_seq=0 2024-05-09T09:33:40.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 56093 32.61 MB/sec warmup 98 sec latency 332.423 ms 2024-05-09T09:33:40.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f3412ffd640 1 -- 172.21.15.69:0/131528455 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56380c261dc0 con 0x56380c37a640 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f3412ffd640 10 client.17244 handle_command_reply: tid=0 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 client.17244 shutdown 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 msgr2=0x56380c304d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f34280b86f0 tx=0x7f34280c1220 comp rx=0 tx=0).stop 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 shutdown_connections 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56380c37a640 0x56380c37ca60 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56380c2fbdb0 0x56380c378ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56380c2f6c40 0x56380c304d20 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 --2- 172.21.15.69:0/131528455 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56380c2f6250 0x56380c3047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 >> 172.21.15.69:0/131528455 conn(0x56380c2f18c0 msgr2=0x56380c2f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f3412ffd640 0 client.17244 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 shutdown_connections 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.914+0000 7f343ca50740 1 -- 172.21.15.69:0/131528455 wait complete. 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f343ca50740 20 client.17244 trim_cache size 0 max 16384 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:33:40.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 msgr2=0x7f341c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 0x7f341c0a5170 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f342c005fd0 tx=0x7f342c009e00 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 msgr2=0x7f3434102da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f3428009230 tx=0x7f34280042f0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 shutdown_connections 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f341c0a2cb0 0x7f341c0a5170 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3434102380 0x7f34341032e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f34341019c0 0x7f3434102da0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 --2- 172.21.15.69:0/1722910756 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f34341007d0 0x7f3434104760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:40.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 >> 172.21.15.69:0/1722910756 conn(0x7f34340fbd40 msgr2=0x7f34340fd710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 shutdown_connections 2024-05-09T09:33:40.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:40.915+0000 7f3439d42640 1 -- 172.21.15.69:0/1722910756 wait complete. 2024-05-09T09:33:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:41 smithi103 ceph-mon[34421]: pgmap v813: 129 pgs: 129 active+clean; 487 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-05-09T09:33:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:41 smithi069 ceph-mon[30123]: pgmap v813: 129 pgs: 129 active+clean; 487 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-05-09T09:33:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:41 smithi163 ceph-mon[34354]: pgmap v813: 129 pgs: 129 active+clean; 487 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 516 op/s 2024-05-09T09:33:41.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 56791 32.68 MB/sec warmup 99 sec latency 385.177 ms 2024-05-09T09:33:42.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 57351 32.69 MB/sec warmup 100 sec latency 62.077 ms 2024-05-09T09:33:43.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:43 smithi103 ceph-mon[34421]: pgmap v814: 129 pgs: 129 active+clean; 492 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-05-09T09:33:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:43 smithi069 ceph-mon[30123]: pgmap v814: 129 pgs: 129 active+clean; 492 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-05-09T09:33:43.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:43 smithi163 ceph-mon[34354]: pgmap v814: 129 pgs: 129 active+clean; 492 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 497 op/s 2024-05-09T09:33:43.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 57913 32.66 MB/sec warmup 101 sec latency 68.641 ms 2024-05-09T09:33:44.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 58685 32.60 MB/sec warmup 102 sec latency 121.466 ms 2024-05-09T09:33:45.357 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:45 smithi163 ceph-mon[34354]: pgmap v815: 129 pgs: 129 active+clean; 497 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 558 op/s 2024-05-09T09:33:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:45 smithi103 ceph-mon[34421]: pgmap v815: 129 pgs: 129 active+clean; 497 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 558 op/s 2024-05-09T09:33:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:45 smithi069 ceph-mon[30123]: pgmap v815: 129 pgs: 129 active+clean; 497 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 558 op/s 2024-05-09T09:33:45.822 INFO:tasks.workunit.client.0.smithi069.stdout: 10 59248 32.63 MB/sec warmup 103 sec latency 147.090 ms 2024-05-09T09:33:46.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 60071 32.85 MB/sec warmup 104 sec latency 69.912 ms 2024-05-09T09:33:47.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:47 smithi103 ceph-mon[34421]: pgmap v816: 129 pgs: 129 active+clean; 499 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.0 MiB/s wr, 606 op/s 2024-05-09T09:33:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:47 smithi069 ceph-mon[30123]: pgmap v816: 129 pgs: 129 active+clean; 499 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.0 MiB/s wr, 606 op/s 2024-05-09T09:33:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:47 smithi163 ceph-mon[34354]: pgmap v816: 129 pgs: 129 active+clean; 499 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.0 MiB/s wr, 606 op/s 2024-05-09T09:33:47.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 60692 32.81 MB/sec warmup 105 sec latency 41.608 ms 2024-05-09T09:33:48.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 61204 32.79 MB/sec warmup 106 sec latency 90.506 ms 2024-05-09T09:33:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:49 smithi103 ceph-mon[34421]: pgmap v817: 129 pgs: 129 active+clean; 506 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 4.4 MiB/s wr, 616 op/s 2024-05-09T09:33:49.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:49 smithi069 ceph-mon[30123]: pgmap v817: 129 pgs: 129 active+clean; 506 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 4.4 MiB/s wr, 616 op/s 2024-05-09T09:33:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:49 smithi163 ceph-mon[34354]: pgmap v817: 129 pgs: 129 active+clean; 506 MiB data, 5.1 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 4.4 MiB/s wr, 616 op/s 2024-05-09T09:33:49.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 61877 32.79 MB/sec warmup 107 sec latency 51.887 ms 2024-05-09T09:33:50.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 62492 32.71 MB/sec warmup 108 sec latency 133.543 ms 2024-05-09T09:33:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:51 smithi103 ceph-mon[34421]: pgmap v818: 129 pgs: 129 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 570 op/s 2024-05-09T09:33:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:51 smithi069 ceph-mon[30123]: pgmap v818: 129 pgs: 129 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 570 op/s 2024-05-09T09:33:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:51 smithi163 ceph-mon[34354]: pgmap v818: 129 pgs: 129 active+clean; 509 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.0 MiB/s wr, 570 op/s 2024-05-09T09:33:51.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 63076 32.74 MB/sec warmup 109 sec latency 78.201 ms 2024-05-09T09:33:52.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 63930 32.92 MB/sec warmup 110 sec latency 73.825 ms 2024-05-09T09:33:53.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:53 smithi103 ceph-mon[34421]: pgmap v819: 129 pgs: 129 active+clean; 515 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 4.8 MiB/s wr, 643 op/s 2024-05-09T09:33:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:53 smithi069 ceph-mon[30123]: pgmap v819: 129 pgs: 129 active+clean; 515 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 4.8 MiB/s wr, 643 op/s 2024-05-09T09:33:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:53 smithi163 ceph-mon[34354]: pgmap v819: 129 pgs: 129 active+clean; 515 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 4.8 MiB/s wr, 643 op/s 2024-05-09T09:33:53.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 64570 32.96 MB/sec warmup 111 sec latency 37.088 ms 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 -- 172.21.15.69:0/885274112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1581023b0 msgr2=0x7fd1581047a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/885274112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1581023b0 0x7fd1581047a0 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7fd148009900 tx=0x7fd14802a410 comp rx=0 tx=0).stop 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 -- 172.21.15.69:0/885274112 shutdown_connections 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/885274112 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd158107610 0x7fd158109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/885274112 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 0x7fd1581070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/885274112 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd1581023b0 0x7fd1581047a0 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 -- 172.21.15.69:0/885274112 >> 172.21.15.69:0/885274112 conn(0x7fd1580fbd40 msgr2=0x7fd1580fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 -- 172.21.15.69:0/885274112 shutdown_connections 2024-05-09T09:33:54.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.027+0000 7fd15ddd5640 1 -- 172.21.15.69:0/885274112 wait complete. 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 Processor -- start 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 -- start start 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1581023b0 0x7fd158101d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 0x7fd158100360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810d570 con 0x7fd158107610 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810d3f0 con 0x7fd158104ce0 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd15ddd5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd15810d6f0 con 0x7fd1581023b0 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd157fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.028+0000 7fd157fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56306/0 (socket says 172.21.15.69:56306) 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 -- 172.21.15.69:0/728951895 learned_addr learned my addr 172.21.15.69:0/728951895 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd156ffd640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 0x7fd158100360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd1577fe640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1581023b0 0x7fd158101d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 -- 172.21.15.69:0/728951895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1581023b0 msgr2=0x7fd158101d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1581023b0 0x7fd158101d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 -- 172.21.15.69:0/728951895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 msgr2=0x7fd158100360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 0x7fd158100360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 -- 172.21.15.69:0/728951895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd158101130 con 0x7fd158107610 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd157fff640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd15000eb10 tx=0x7fd15000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd154ff9640 1 -- 172.21.15.69:0/728951895 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150004b20 con 0x7fd158107610 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd154ff9640 1 -- 172.21.15.69:0/728951895 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd150004cc0 con 0x7fd158107610 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd154ff9640 1 -- 172.21.15.69:0/728951895 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150010760 con 0x7fd158107610 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd1581a1c30 con 0x7fd158107610 2024-05-09T09:33:54.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.029+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd1581a21d0 con 0x7fd158107610 2024-05-09T09:33:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.032+0000 7fd160bbf740 1 Processor -- start 2024-05-09T09:33:54.032 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.032+0000 7fd154ff9640 1 -- 172.21.15.69:0/728951895 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd150005020 con 0x7fd158107610 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 -- start start 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d61740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d64240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5638e8cc0340 con 0x5638e8d5cc40 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5638e8c68290 con 0x5638e8d61db0 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd160bbf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5638e8cbc300 con 0x5638e8d5c250 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd1577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d61740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39924/0 (socket says 172.21.15.69:39924) 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd156ffd640 1 -- 172.21.15.69:0/2500341817 learned_addr learned my addr 172.21.15.69:0/2500341817 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:54.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.033+0000 7fd156ffd640 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d64240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd1577fe640 1 -- 172.21.15.69:0/2500341817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 msgr2=0x5638e8d5c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd1577fe640 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd1577fe640 1 -- 172.21.15.69:0/2500341817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 msgr2=0x5638e8d64240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd1577fe640 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d64240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd1577fe640 1 -- 172.21.15.69:0/2500341817 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638e8c8c8e0 con 0x5638e8d5cc40 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd156ffd640 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd154ff9640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 0x7fd1400a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.034+0000 7fd154ff9640 1 -- 172.21.15.69:0/728951895 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd1500cfa90 con 0x7fd158107610 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd156ffd640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 0x7fd1400a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd1577fe640 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d61740 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd148031440 tx=0x7fd148005e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6dbab4d242e7c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd156ffd640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 0x7fd1400a5170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x5638e8d5d4a0 tx=0x7fd14c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd13dffb640 1 -- 172.21.15.69:0/2500341817 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148040070 con 0x5638e8d5cc40 2024-05-09T09:33:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd13dffb640 1 -- 172.21.15.69:0/2500341817 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd148004030 con 0x5638e8d5cc40 2024-05-09T09:33:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.035+0000 7fd13dffb640 1 -- 172.21.15.69:0/2500341817 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd148035040 con 0x5638e8d5cc40 2024-05-09T09:33:54.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.036+0000 7fd160bbf740 1 -- 172.21.15.69:0/2500341817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 msgr2=0x5638e8d61740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.036+0000 7fd160bbf740 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d61740 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7fd148031440 tx=0x7fd148005e00 comp rx=0 tx=0).stop 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 -- 172.21.15.69:0/2500341817 shutdown_connections 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d64240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d61740 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 --2- 172.21.15.69:0/2500341817 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 -- 172.21.15.69:0/2500341817 >> 172.21.15.69:0/2500341817 conn(0x5638e8d578c0 msgr2=0x5638e8d57cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 -- 172.21.15.69:0/2500341817 shutdown_connections 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 -- 172.21.15.69:0/2500341817 wait complete. 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.037+0000 7fd160bbf740 1 -- start start 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.038+0000 7fd160bbf740 1 Processor -- start 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.038+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 0x5638e8d6ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.038+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d61db0 0x5638e8d76250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd160bbf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5638e8d64d00 con 0x5638e8d61db0 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd160bbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5638e8c7b790 con 0x5638e8d5c250 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd160bbf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5638e8d64e80 con 0x5638e8d5cc40 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd1577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 0x5638e8d6ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39928/0 (socket says 172.21.15.69:39928) 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd1577fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 0x5638e8d6ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60438/0 (socket says 172.21.15.69:60438) 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 -- 172.21.15.69:0/4075752120 learned_addr learned my addr 172.21.15.69:0/4075752120 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd160bbf740 10 client.0 _ll_register_callbacks cb 0x5638e8cb6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd157fff640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d61db0 0x5638e8d76250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 -- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 msgr2=0x5638e8d6ecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 0x5638e8d6ecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 -- 172.21.15.69:0/4075752120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d61db0 msgr2=0x5638e8d76250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d61db0 0x5638e8d76250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 -- 172.21.15.69:0/4075752120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638e89d4210 con 0x5638e8d5cc40 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd156ffd640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009530 tx=0x7fd14c0046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e18387d0fe88eedf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.039+0000 7fd136ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c0229b0 con 0x5638e8d5cc40 2024-05-09T09:33:54.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd148004030 con 0x5638e8d5cc40 2024-05-09T09:33:54.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd14c022b50 con 0x5638e8d5cc40 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd14c022d50 con 0x5638e8d5cc40 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd160bbf740 10 client.26461 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5638e8aa0620 con 0x5638e8d5cc40 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.040+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd148004030 con 0x5638e8d5cc40 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.041+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fd14c005230 con 0x5638e8d5cc40 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 10 client.26461 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:54.042 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 10 client.26461 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 20 client.26461 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8d7cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 4 client.26461 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5638e8ccda70 con 0x5638e8d7a810 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.042+0000 7fd1577fe640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8d7cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.043+0000 7fd136ffd640 10 client.26461 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:54.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.043+0000 7fd1577fe640 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8d7cc90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f396ac7003cef45f server_cookie=8a61a3f2f72116bb in_seq=0 out_seq=0 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.049+0000 7fd136ffd640 1 -- 172.21.15.69:0/4075752120 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5638e8ccda70 con 0x5638e8d7a810 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.049+0000 7fd136ffd640 10 client.26461 handle_command_reply: tid=0 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 client.26461 shutdown 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 msgr2=0x5638e8d6f210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fd14c009530 tx=0x7fd14c0046c0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 shutdown_connections 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d61db0 0x5638e8d76250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8d7cc90 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd136ffd640 0 client.26461 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5cc40 0x5638e8d6f210 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 --2- 172.21.15.69:0/4075752120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5c250 0x5638e8d6ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 >> 172.21.15.69:0/4075752120 conn(0x5638e8d578c0 msgr2=0x5638e8d56a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 shutdown_connections 2024-05-09T09:33:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 1 -- 172.21.15.69:0/4075752120 wait complete. 2024-05-09T09:33:54.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.050+0000 7fd160bbf740 20 client.26461 trim_cache size 0 max 16384 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 Processor -- start 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 -- start start 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5a870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d5adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581aab50 con 0x5638e8d5cc40 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a8390 con 0x5638e8d61db0 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd160bbf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a61e0 con 0x5638e8d5c250 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd1577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5a870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd156ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56342/0 (socket says 172.21.15.69:56342) 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd156ffd640 1 -- 172.21.15.69:0/1090988664 learned_addr learned my addr 172.21.15.69:0/1090988664 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.070+0000 7fd157fff640 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d5adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 -- 172.21.15.69:0/1090988664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 msgr2=0x5638e8d5a870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5a870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 -- 172.21.15.69:0/1090988664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 msgr2=0x5638e8d5adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d5adb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 -- 172.21.15.69:0/1090988664 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5638e8d5d4d0 con 0x5638e8d5cc40 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd1577fe640 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd156ffd640 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd14c011f40 tx=0x7fd14c004060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=306275e7decc81fe server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd13d7fa640 1 -- 172.21.15.69:0/1090988664 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c005230 con 0x5638e8d5cc40 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd13d7fa640 1 -- 172.21.15.69:0/1090988664 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd14c03b020 con 0x5638e8d5cc40 2024-05-09T09:33:54.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.071+0000 7fd13d7fa640 1 -- 172.21.15.69:0/1090988664 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd14c011290 con 0x5638e8d5cc40 2024-05-09T09:33:54.075 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 -- 172.21.15.69:0/1090988664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 msgr2=0x5638e8d76210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7fd14c011f40 tx=0x7fd14c004060 comp rx=0 tx=0).stop 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 -- 172.21.15.69:0/1090988664 shutdown_connections 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d61db0 0x5638e8d5adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5cc40 0x5638e8d76210 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 --2- 172.21.15.69:0/1090988664 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d5c250 0x5638e8d5a870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 -- 172.21.15.69:0/1090988664 >> 172.21.15.69:0/1090988664 conn(0x5638e8d578c0 msgr2=0x5638e8d57ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 -- 172.21.15.69:0/1090988664 shutdown_connections 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.072+0000 7fd160bbf740 1 -- 172.21.15.69:0/1090988664 wait complete. 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 -- start start 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 Processor -- start 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 0x5638e8d73540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5cc40 0x5638e8d73a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a6850 con 0x5638e8d5c250 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a7cc0 con 0x5638e8d5cc40 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd1581a6f20 con 0x5638e8d61db0 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd160bbf740 10 client.0 _ll_register_callbacks cb 0x7fd14c0099d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd157fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd1577fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 0x5638e8d73540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd157fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39940/0 (socket says 172.21.15.69:39940) 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.073+0000 7fd157fff640 1 -- 172.21.15.69:0/991360503 learned_addr learned my addr 172.21.15.69:0/991360503 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd156ffd640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5cc40 0x5638e8d73a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 -- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5cc40 msgr2=0x5638e8d73a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5cc40 0x5638e8d73a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 -- 172.21.15.69:0/991360503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 msgr2=0x5638e8d73540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 0x5638e8d73540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 -- 172.21.15.69:0/991360503 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5638e8d75080 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd1577fe640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 0x5638e8d73540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd157fff640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd150098370 tx=0x7fd15009b870 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c710475edf3d597e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd136ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd150005510 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd1500bab50 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd14c03b020 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.074+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fd150082050 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 10 client.26467 fetch_fsmap learned FSMap version 43 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x5638e8d76490 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd1500a02c0 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fd15009add0 con 0x5638e8d61db0 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 10 client.26467 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:33:54.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 10 client.26467 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 20 client.26467 populate_metadata read hostname 'smithi069' 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8de35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 4 client.26467 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd14c03b020 con 0x5638e8d7a810 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.075+0000 7fd1577fe640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8de35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.076+0000 7fd136ffd640 10 client.26467 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:54.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.076+0000 7fd1577fe640 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8de35c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc193b17488ae96d server_cookie=f4b2e75ac81e3e71 in_seq=0 out_seq=0 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd136ffd640 1 -- 172.21.15.69:0/991360503 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd14c03b020 con 0x5638e8d7a810 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd136ffd640 10 client.26467 handle_command_reply: tid=0 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 client.26467 shutdown 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 msgr2=0x5638e8d74a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fd150098370 tx=0x7fd15009b870 comp rx=0 tx=0).stop 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 shutdown_connections 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5638e8d7a810 0x5638e8de35c0 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5638e8d61db0 0x5638e8d74a90 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.722 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5638e8d5cc40 0x5638e8d73a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd136ffd640 0 client.26467 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 --2- 172.21.15.69:0/991360503 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5638e8d5c250 0x5638e8d73540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 >> 172.21.15.69:0/991360503 conn(0x5638e8d578c0 msgr2=0x5638e8d5a870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 shutdown_connections 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 1 -- 172.21.15.69:0/991360503 wait complete. 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.722+0000 7fd160bbf740 20 client.26467 trim_cache size 0 max 16384 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:33:54.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:33:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 msgr2=0x7fd1400a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 0x7fd1400a5170 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x5638e8d5d4a0 tx=0x7fd14c005f90 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 msgr2=0x7fd1581008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7fd15000eb10 tx=0x7fd15000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 shutdown_connections 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd1400a2cb0 0x7fd1400a5170 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd158107610 0x7fd1581008a0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd158104ce0 0x7fd158100360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 --2- 172.21.15.69:0/728951895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd1581023b0 0x7fd158101d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 >> 172.21.15.69:0/728951895 conn(0x7fd1580fbd40 msgr2=0x7fd158102df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 shutdown_connections 2024-05-09T09:33:54.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:33:54.723+0000 7fd15ddd5640 1 -- 172.21.15.69:0/728951895 wait complete. 2024-05-09T09:33:54.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 64952 32.78 MB/sec warmup 112 sec latency 351.760 ms 2024-05-09T09:33:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:55 smithi103 ceph-mon[34421]: pgmap v820: 129 pgs: 129 active+clean; 520 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 648 op/s 2024-05-09T09:33:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:55 smithi069 ceph-mon[30123]: pgmap v820: 129 pgs: 129 active+clean; 520 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 648 op/s 2024-05-09T09:33:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:55 smithi163 ceph-mon[34354]: pgmap v820: 129 pgs: 129 active+clean; 520 MiB data, 5.2 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 5.5 MiB/s wr, 648 op/s 2024-05-09T09:33:55.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 65110 32.58 MB/sec warmup 113 sec latency 713.026 ms 2024-05-09T09:33:56.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 65139 32.32 MB/sec warmup 114 sec latency 1703.031 ms 2024-05-09T09:33:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:57 smithi069 ceph-mon[30123]: pgmap v821: 129 pgs: 129 active+clean; 524 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-05-09T09:33:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:57 smithi163 ceph-mon[34354]: pgmap v821: 129 pgs: 129 active+clean; 524 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-05-09T09:33:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:57 smithi103 ceph-mon[34421]: pgmap v821: 129 pgs: 129 active+clean; 524 MiB data, 5.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 5.4 MiB/s wr, 580 op/s 2024-05-09T09:33:57.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 65173 32.05 MB/sec warmup 115 sec latency 1829.256 ms 2024-05-09T09:33:58.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 65407 31.88 MB/sec warmup 116 sec latency 186.958 ms 2024-05-09T09:33:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:33:59 smithi069 ceph-mon[30123]: pgmap v822: 129 pgs: 129 active+clean; 604 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 13 MiB/s wr, 735 op/s 2024-05-09T09:33:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:33:59 smithi163 ceph-mon[34354]: pgmap v822: 129 pgs: 129 active+clean; 604 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 13 MiB/s wr, 735 op/s 2024-05-09T09:33:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:33:59 smithi103 ceph-mon[34421]: pgmap v822: 129 pgs: 129 active+clean; 604 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 13 MiB/s wr, 735 op/s 2024-05-09T09:33:59.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 65741 31.76 MB/sec warmup 117 sec latency 102.020 ms 2024-05-09T09:34:00.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 66085 31.65 MB/sec warmup 118 sec latency 125.045 ms 2024-05-09T09:34:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:01 smithi069 ceph-mon[30123]: pgmap v823: 129 pgs: 129 active+clean; 624 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 809 op/s 2024-05-09T09:34:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:01 smithi163 ceph-mon[34354]: pgmap v823: 129 pgs: 129 active+clean; 624 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 809 op/s 2024-05-09T09:34:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:01 smithi103 ceph-mon[34421]: pgmap v823: 129 pgs: 129 active+clean; 624 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 809 op/s 2024-05-09T09:34:01.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 66799 31.68 MB/sec warmup 119 sec latency 65.322 ms 2024-05-09T09:34:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:03 smithi069 ceph-mon[30123]: pgmap v824: 129 pgs: 129 active+clean; 605 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 932 op/s 2024-05-09T09:34:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:03 smithi163 ceph-mon[34354]: pgmap v824: 129 pgs: 129 active+clean; 605 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 932 op/s 2024-05-09T09:34:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:03 smithi103 ceph-mon[34421]: pgmap v824: 129 pgs: 129 active+clean; 605 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 19 MiB/s wr, 932 op/s 2024-05-09T09:34:03.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 68267 51.76 MB/sec execute 1 sec latency 172.097 ms 2024-05-09T09:34:04.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 68804 39.53 MB/sec execute 2 sec latency 64.429 ms 2024-05-09T09:34:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:05 smithi069 ceph-mon[30123]: pgmap v825: 129 pgs: 129 active+clean; 529 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-05-09T09:34:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:05 smithi163 ceph-mon[34354]: pgmap v825: 129 pgs: 129 active+clean; 529 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-05-09T09:34:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:05 smithi103 ceph-mon[34421]: pgmap v825: 129 pgs: 129 active+clean; 529 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 19 MiB/s wr, 1.03k op/s 2024-05-09T09:34:05.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 69312 32.43 MB/sec execute 3 sec latency 63.155 ms 2024-05-09T09:34:06.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 70017 33.83 MB/sec execute 4 sec latency 56.923 ms 2024-05-09T09:34:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:07 smithi069 ceph-mon[30123]: pgmap v826: 129 pgs: 129 active+clean; 494 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:07.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:07 smithi163 ceph-mon[34354]: pgmap v826: 129 pgs: 129 active+clean; 494 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:07 smithi103 ceph-mon[34421]: pgmap v826: 129 pgs: 129 active+clean; 494 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:07.823 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 -- 172.21.15.69:0/1890129118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 msgr2=0x7f26dc108bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 --2- 172.21.15.69:0/1890129118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc108bf0 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7f26d4009ca0 tx=0x7f26d402a8c0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 -- 172.21.15.69:0/1890129118 shutdown_connections 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 --2- 172.21.15.69:0/1890129118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc108bf0 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 --2- 172.21.15.69:0/1890129118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 0x7f26dc101e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 --2- 172.21.15.69:0/1890129118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.823+0000 7f26e1934640 1 -- 172.21.15.69:0/1890129118 >> 172.21.15.69:0/1890129118 conn(0x7f26dc0fbd20 msgr2=0x7f26dc0fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- 172.21.15.69:0/1890129118 shutdown_connections 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- 172.21.15.69:0/1890129118 wait complete. 2024-05-09T09:34:07.824 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 Processor -- start 2024-05-09T09:34:07.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 70775 34.14 MB/sec execute 5 sec latency 197.849 ms 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- start start 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 0x7f26dc197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc10c430 con 0x7f26dc102360 2024-05-09T09:34:07.829 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc10c2b0 con 0x7f26dc1007b0 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.824+0000 7f26e1934640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc10c5b0 con 0x7f26dc1019a0 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26daffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26daffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54446/0 (socket says 172.21.15.69:54446) 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53992/0 (socket says 172.21.15.69:53992) 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 0x7f26dc197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26daffd640 1 -- 172.21.15.69:0/1327422928 learned_addr learned my addr 172.21.15.69:0/1327422928 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 -- 172.21.15.69:0/1327422928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 msgr2=0x7f26dc197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 0x7f26dc197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 -- 172.21.15.69:0/1327422928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 msgr2=0x7f26dc197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.830 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 -- 172.21.15.69:0/1327422928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f26dc19f690 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26daffd640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26db7fe640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f26d4031ae0 tx=0x7f26d4004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26e0932640 1 -- 172.21.15.69:0/1327422928 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d4003b70 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26e0932640 1 -- 172.21.15.69:0/1327422928 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d4003d10 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26e0932640 1 -- 172.21.15.69:0/1327422928 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d402fc40 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f26dc19f920 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.825+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f26dc19fe40 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e0932640 1 -- 172.21.15.69:0/1327422928 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f26d4004050 con 0x7f26dc102360 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 Processor -- start 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 -- start start 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843f2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843f7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db843fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55db84356340 con 0x55db843f2c40 2024-05-09T09:34:07.831 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55db842fe290 con 0x55db843f2250 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e4643740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db84352300 con 0x55db843f7db0 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54452/0 (socket says 172.21.15.69:54452) 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26da7fc640 1 -- 172.21.15.69:0/3988798109 learned_addr learned my addr 172.21.15.69:0/3988798109 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26da7fc640 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db843fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 -- 172.21.15.69:0/3988798109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 msgr2=0x55db843fa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db843fa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 -- 172.21.15.69:0/3988798109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 msgr2=0x55db843f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26daffd640 1 -- 172.21.15.69:0/3988798109 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db843228e0 con 0x55db843f2c40 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26da7fc640 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db843fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:34:07.832 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.828+0000 7f26e0932640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 0x7f26c40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26da7fc640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 0x7f26c40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26daffd640 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843f7740 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0114c0 tx=0x7f26cc011980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0918d2a85525017 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26e0932640 1 -- 172.21.15.69:0/1327422928 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f26d40f0320 con 0x7f26dc102360 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26c1ffb640 1 -- 172.21.15.69:0/3988798109 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc005020 con 0x55db843f2c40 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26c1ffb640 1 -- 172.21.15.69:0/3988798109 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc0047a0 con 0x55db843f2c40 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.829+0000 7f26c1ffb640 1 -- 172.21.15.69:0/3988798109 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26cc00f8d0 con 0x55db843f2c40 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- 172.21.15.69:0/3988798109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 msgr2=0x55db843f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843f7740 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f26cc0114c0 tx=0x7f26cc011980 comp rx=0 tx=0).stop 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26da7fc640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 0x7f26c40a50f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009fd0 tx=0x7f26d00097d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- 172.21.15.69:0/3988798109 shutdown_connections 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db843fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843f7740 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 --2- 172.21.15.69:0/3988798109 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.833 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- 172.21.15.69:0/3988798109 >> 172.21.15.69:0/3988798109 conn(0x55db843ed8c0 msgr2=0x55db843edcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- 172.21.15.69:0/3988798109 shutdown_connections 2024-05-09T09:34:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- 172.21.15.69:0/3988798109 wait complete. 2024-05-09T09:34:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 -- start start 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.830+0000 7f26e4643740 1 Processor -- start 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db844039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55db843fad60 con 0x55db843f2c40 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55db84311790 con 0x55db843f2250 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55db843faee0 con 0x55db843f7db0 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26e4643740 10 client.0 _ll_register_callbacks cb 0x55db8434ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54004/0 (socket says 172.21.15.69:54004) 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 -- 172.21.15.69:0/2646127446 learned_addr learned my addr 172.21.15.69:0/2646127446 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26daffd640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26db7fe640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db844039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 -- 172.21.15.69:0/2646127446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 msgr2=0x55db844039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db844039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 -- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 msgr2=0x55db843fc440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843fc440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 -- 172.21.15.69:0/2646127446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db8406a210 con 0x55db843f2c40 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26db7fe640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db844039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f26d0010a30 tx=0x7f26d0004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1fe90dd7fb73ead2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.831+0000 7f26baffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d00216f0 con 0x55db843f2c40 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f26d0021890 con 0x55db843f2c40 2024-05-09T09:34:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26cc0047a0 con 0x55db843f2c40 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f26d0036050 con 0x55db843f2c40 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 10 client.17274 fetch_fsmap learned FSMap version 43 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55db84136620 con 0x55db843f2c40 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26cc0047a0 con 0x55db843f2c40 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f26d001fc70 con 0x55db843f2c40 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 10 client.17274 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 10 client.17274 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 20 client.17274 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db84407e30 0x55db8440a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 4 client.17274 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.832+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55db84363a70 con 0x55db84407e30 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.833+0000 7f26daffd640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db84407e30 0x55db8440a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.833+0000 7f26daffd640 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db84407e30 0x55db8440a2d0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=164fc2a9c3ab9bae server_cookie=d41610b398f0186c in_seq=0 out_seq=0 2024-05-09T09:34:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.833+0000 7f26baffd640 10 client.17274 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.841+0000 7f26baffd640 1 -- 172.21.15.69:0/2646127446 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55db84363a70 con 0x55db84407e30 2024-05-09T09:34:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.841+0000 7f26baffd640 10 client.17274 handle_command_reply: tid=0 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.841+0000 7f26e4643740 1 client.17274 shutdown 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 msgr2=0x55db843fc980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f26d0010a30 tx=0x7f26d0004730 comp rx=0 tx=0).stop 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 shutdown_connections 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db84407e30 0x55db8440a2d0 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db844039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db843fc980 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26baffd640 0 client.17274 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 --2- 172.21.15.69:0/2646127446 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db843fc440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 >> 172.21.15.69:0/2646127446 conn(0x55db843ed8c0 msgr2=0x55db843eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 shutdown_connections 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 1 -- 172.21.15.69:0/2646127446 wait complete. 2024-05-09T09:34:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.842+0000 7f26e4643740 20 client.17274 trim_cache size 0 max 16384 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 Processor -- start 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 -- start start 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f2250 0x55db8440fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 0x55db84410120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1acc50 con 0x55db843f7db0 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1aa490 con 0x55db843f2c40 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26e4643740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1a82e0 con 0x55db843f2250 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54018/0 (socket says 172.21.15.69:54018) 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 -- 172.21.15.69:0/4016478306 learned_addr learned my addr 172.21.15.69:0/4016478306 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26da7fc640 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 0x55db84410120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 -- 172.21.15.69:0/4016478306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f2250 msgr2=0x55db8440fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f2250 0x55db8440fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 -- 172.21.15.69:0/4016478306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 msgr2=0x55db84410120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 0x55db84410120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 -- 172.21.15.69:0/4016478306 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55db843f3910 con 0x55db843f7db0 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26da7fc640 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 0x55db84410120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26db7fe640 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f26d4005e30 tx=0x7f26d40a7040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f94d7efeb86ef7b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.858+0000 7f26c17fa640 1 -- 172.21.15.69:0/4016478306 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d40f0160 con 0x55db843f7db0 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26c17fa640 1 -- 172.21.15.69:0/4016478306 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d4036040 con 0x55db843f7db0 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26c17fa640 1 -- 172.21.15.69:0/4016478306 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d402fde0 con 0x55db843f7db0 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 -- 172.21.15.69:0/4016478306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 msgr2=0x55db84403980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f26d4005e30 tx=0x7f26d40a7040 comp rx=0 tx=0).stop 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 -- 172.21.15.69:0/4016478306 shutdown_connections 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f7db0 0x55db84403980 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2c40 0x55db84410120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 --2- 172.21.15.69:0/4016478306 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f2250 0x55db8440fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 -- 172.21.15.69:0/4016478306 >> 172.21.15.69:0/4016478306 conn(0x55db843ed8c0 msgr2=0x55db843edca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:07.861 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 -- 172.21.15.69:0/4016478306 shutdown_connections 2024-05-09T09:34:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.859+0000 7f26e4643740 1 -- 172.21.15.69:0/4016478306 wait complete. 2024-05-09T09:34:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 -- start start 2024-05-09T09:34:07.862 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 Processor -- start 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db84401400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db84402900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1a8950 con 0x55db843f2c40 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1a9dc0 con 0x55db843f2250 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f26dc1a9020 con 0x55db843f7db0 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26e4643740 10 client.0 _ll_register_callbacks cb 0x7f26d00099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:07.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26da7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:54022/0 (socket says 172.21.15.69:54022) 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26da7fc640 1 -- 172.21.15.69:0/2540113912 learned_addr learned my addr 172.21.15.69:0/2540113912 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26daffd640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db84401400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.860+0000 7f26db7fe640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db84402900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 -- 172.21.15.69:0/2540113912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 msgr2=0x55db84402900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db84402900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 -- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 msgr2=0x55db84401400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db84401400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 -- 172.21.15.69:0/2540113912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55db84402ef0 con 0x55db843f2c40 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26daffd640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db84401400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26da7fc640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d002b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e0c880c793e092d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:07.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26baffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f26d0010800 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f26d002bd20 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f26d4036040 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7f26d0048050 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26e4643740 10 client.17280 fetch_fsmap learned FSMap version 43 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.861+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55db8446e710 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f26d4036040 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7f26d0005390 con 0x55db843f2c40 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 10 client.17280 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 10 client.17280 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 20 client.17280 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:07.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db8446f1f0 0x55db84471610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 4 client.17280 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:34:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55db8435ddc0 con 0x55db8446f1f0 2024-05-09T09:34:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.862+0000 7f26daffd640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db8446f1f0 0x55db84471610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.863+0000 7f26daffd640 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db8446f1f0 0x55db84471610 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ea807fcd8aa2dd3 server_cookie=1e1484e031edaae7 in_seq=0 out_seq=0 2024-05-09T09:34:07.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:07.863+0000 7f26baffd640 10 client.17280 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26baffd640 1 -- 172.21.15.69:0/2540113912 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55db8435ddc0 con 0x55db8446f1f0 2024-05-09T09:34:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26baffd640 10 client.17280 handle_command_reply: tid=0 2024-05-09T09:34:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26e4643740 1 client.17280 shutdown 2024-05-09T09:34:08.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 msgr2=0x55db84401940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f26d0000c00 tx=0x7f26d002b970 comp rx=0 tx=0).stop 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 shutdown_connections 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55db8446f1f0 0x55db84471610 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.351+0000 7f26baffd640 0 client.17280 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55db843f7db0 0x55db84402900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55db843f2c40 0x55db84401940 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 --2- 172.21.15.69:0/2540113912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55db843f2250 0x55db84401400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 >> 172.21.15.69:0/2540113912 conn(0x55db843ed8c0 msgr2=0x55db843fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 shutdown_connections 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 1 -- 172.21.15.69:0/2540113912 wait complete. 2024-05-09T09:34:08.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.352+0000 7f26e4643740 20 client.17280 trim_cache size 0 max 16384 2024-05-09T09:34:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:34:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:34:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:34:08.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:34:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.353+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 msgr2=0x7f26c40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.353+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 0x7f26c40a50f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f26d0009fd0 tx=0x7f26d00097d0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.353+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 msgr2=0x7f26dc19f020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:08.354 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.353+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7f26d4031ae0 tx=0x7f26d4004ab0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 shutdown_connections 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f26c40a2c30 0x7f26c40a50f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f26dc102360 0x7f26dc19f020 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f26dc1019a0 0x7f26dc197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 --2- 172.21.15.69:0/1327422928 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f26dc1007b0 0x7f26dc197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 >> 172.21.15.69:0/1327422928 conn(0x7f26dc0fbd20 msgr2=0x7f26dc0fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 shutdown_connections 2024-05-09T09:34:08.355 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:08.354+0000 7f26e1934640 1 -- 172.21.15.69:0/1327422928 wait complete. 2024-05-09T09:34:08.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 71301 34.14 MB/sec execute 6 sec latency 327.545 ms 2024-05-09T09:34:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:09 smithi069 ceph-mon[30123]: pgmap v827: 129 pgs: 129 active+clean; 499 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:09.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:09 smithi163 ceph-mon[34354]: pgmap v827: 129 pgs: 129 active+clean; 499 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:09 smithi103 ceph-mon[34421]: pgmap v827: 129 pgs: 129 active+clean; 499 MiB data, 5.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 18 MiB/s wr, 1.10k op/s 2024-05-09T09:34:09.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 71887 35.52 MB/sec execute 7 sec latency 58.850 ms 2024-05-09T09:34:10.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 72395 34.07 MB/sec execute 8 sec latency 50.661 ms 2024-05-09T09:34:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:11 smithi069 ceph-mon[30123]: pgmap v828: 129 pgs: 129 active+clean; 502 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 911 op/s 2024-05-09T09:34:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:11 smithi163 ceph-mon[34354]: pgmap v828: 129 pgs: 129 active+clean; 502 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 911 op/s 2024-05-09T09:34:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:11 smithi103 ceph-mon[34421]: pgmap v828: 129 pgs: 129 active+clean; 502 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 11 MiB/s wr, 911 op/s 2024-05-09T09:34:11.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 72819 32.17 MB/sec execute 9 sec latency 119.022 ms 2024-05-09T09:34:12.823 INFO:tasks.workunit.client.0.smithi069.stdout: 10 73550 32.47 MB/sec execute 10 sec latency 88.593 ms 2024-05-09T09:34:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:13 smithi069 ceph-mon[30123]: pgmap v829: 129 pgs: 129 active+clean; 500 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.9 MiB/s wr, 807 op/s 2024-05-09T09:34:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:13 smithi163 ceph-mon[34354]: pgmap v829: 129 pgs: 129 active+clean; 500 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.9 MiB/s wr, 807 op/s 2024-05-09T09:34:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:13 smithi103 ceph-mon[34421]: pgmap v829: 129 pgs: 129 active+clean; 500 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.9 MiB/s wr, 807 op/s 2024-05-09T09:34:13.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 74500 34.01 MB/sec execute 11 sec latency 31.610 ms 2024-05-09T09:34:14.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 75267 35.52 MB/sec execute 12 sec latency 54.597 ms 2024-05-09T09:34:15.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:15 smithi103 ceph-mon[34421]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.8 MiB/s wr, 747 op/s 2024-05-09T09:34:15.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:15 smithi069 ceph-mon[30123]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.8 MiB/s wr, 747 op/s 2024-05-09T09:34:15.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:15 smithi163 ceph-mon[34354]: pgmap v830: 129 pgs: 129 active+clean; 505 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.8 MiB/s wr, 747 op/s 2024-05-09T09:34:15.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 75838 35.20 MB/sec execute 13 sec latency 45.347 ms 2024-05-09T09:34:16.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 76261 34.08 MB/sec execute 14 sec latency 129.594 ms 2024-05-09T09:34:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:17 smithi069 ceph-mon[30123]: pgmap v831: 129 pgs: 129 active+clean; 511 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.4 MiB/s wr, 625 op/s 2024-05-09T09:34:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:17 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:34:17.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:17 smithi163 ceph-mon[34354]: pgmap v831: 129 pgs: 129 active+clean; 511 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.4 MiB/s wr, 625 op/s 2024-05-09T09:34:17.558 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:17 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:34:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:17 smithi103 ceph-mon[34421]: pgmap v831: 129 pgs: 129 active+clean; 511 MiB data, 5.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.4 MiB/s wr, 625 op/s 2024-05-09T09:34:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:17 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:34:17.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 76758 33.41 MB/sec execute 15 sec latency 132.294 ms 2024-05-09T09:34:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:34:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:34:18.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:34:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:34:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:34:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:34:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:34:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:34:18.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:34:18.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 77531 33.90 MB/sec execute 16 sec latency 51.251 ms 2024-05-09T09:34:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:19 smithi069 ceph-mon[30123]: pgmap v832: 129 pgs: 129 active+clean; 521 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 598 op/s 2024-05-09T09:34:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:19 smithi163 ceph-mon[34354]: pgmap v832: 129 pgs: 129 active+clean; 521 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 598 op/s 2024-05-09T09:34:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:19 smithi103 ceph-mon[34421]: pgmap v832: 129 pgs: 129 active+clean; 521 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 11 MiB/s rd, 4.0 MiB/s wr, 598 op/s 2024-05-09T09:34:19.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 78248 34.05 MB/sec execute 17 sec latency 102.540 ms 2024-05-09T09:34:20.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 78944 34.58 MB/sec execute 18 sec latency 121.471 ms 2024-05-09T09:34:21.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:21 smithi069 ceph-mon[30123]: pgmap v833: 129 pgs: 129 active+clean; 528 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.9 MiB/s wr, 589 op/s 2024-05-09T09:34:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.526+0000 7fbd05027640 1 -- 172.21.15.69:0/110534021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd0010db50 msgr2=0x7fbd0010dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.526+0000 7fbd05027640 1 --2- 172.21.15.69:0/110534021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd0010db50 0x7fbd0010dfd0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009860 tx=0x7fbcf402a380 comp rx=0 tx=0).stop 2024-05-09T09:34:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.529+0000 7fbd05027640 1 -- 172.21.15.69:0/110534021 shutdown_connections 2024-05-09T09:34:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.529+0000 7fbd05027640 1 --2- 172.21.15.69:0/110534021 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd0010db50 0x7fbd0010dfd0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.529+0000 7fbd05027640 1 --2- 172.21.15.69:0/110534021 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010b1a0 0x7fbd0010d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.529+0000 7fbd05027640 1 --2- 172.21.15.69:0/110534021 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd00108830 0x7fbd0010ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.529+0000 7fbd05027640 1 -- 172.21.15.69:0/110534021 >> 172.21.15.69:0/110534021 conn(0x7fbd0006e8a0 msgr2=0x7fbd00070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:21.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.536+0000 7fbd05027640 1 -- 172.21.15.69:0/110534021 shutdown_connections 2024-05-09T09:34:21.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.538+0000 7fbd05027640 1 -- 172.21.15.69:0/110534021 wait complete. 2024-05-09T09:34:21.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.539+0000 7fbd05027640 1 Processor -- start 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 -- start start 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0010b1a0 0x7fbd00118240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010db50 0x7fbd00119280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbd00110510 con 0x7fbd0010b1a0 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbd00110390 con 0x7fbd00108830 2024-05-09T09:34:21.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.542+0000 7fbd05027640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbd00110690 con 0x7fbd0010db50 2024-05-09T09:34:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.544+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.544+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35356/0 (socket says 172.21.15.69:35356) 2024-05-09T09:34:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.544+0000 7fbcfed9d640 1 -- 172.21.15.69:0/3759749250 learned_addr learned my addr 172.21.15.69:0/3759749250 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:21.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.545+0000 7fbcff59e640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010db50 0x7fbd00119280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.546+0000 7fbcfed9d640 1 -- 172.21.15.69:0/3759749250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010db50 msgr2=0x7fbd00119280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.546+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010db50 0x7fbd00119280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.546 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.546+0000 7fbcfed9d640 1 -- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0010b1a0 msgr2=0x7fbd00118240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.546+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0010b1a0 0x7fbd00118240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.546+0000 7fbcfed9d640 1 -- 172.21.15.69:0/3759749250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbd001198f0 con 0x7fbd00108830 2024-05-09T09:34:21.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.549+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d950 tx=0x7fbcf000de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.552+0000 7fbce7fff640 1 -- 172.21.15.69:0/3759749250 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0017070 con 0x7fbd00108830 2024-05-09T09:34:21.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.552+0000 7fbce7fff640 1 -- 172.21.15.69:0/3759749250 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf0009c30 con 0x7fbd00108830 2024-05-09T09:34:21.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.552+0000 7fbce7fff640 1 -- 172.21.15.69:0/3759749250 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf000f6b0 con 0x7fbd00108830 2024-05-09T09:34:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:21 smithi163 ceph-mon[34354]: pgmap v833: 129 pgs: 129 active+clean; 528 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.9 MiB/s wr, 589 op/s 2024-05-09T09:34:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.553+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbd00119be0 con 0x7fbd00108830 2024-05-09T09:34:21.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.553+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbd0011a0e0 con 0x7fbd00108830 2024-05-09T09:34:21.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.557+0000 7fbce7fff640 1 -- 172.21.15.69:0/3759749250 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbcf000f850 con 0x7fbd00108830 2024-05-09T09:34:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.558+0000 7fbce7fff640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 0x7fbce80a5320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.559+0000 7fbce7fff640 1 -- 172.21.15.69:0/3759749250 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbcf00cefb0 con 0x7fbd00108830 2024-05-09T09:34:21.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.563+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 0x7fbce80a5320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.563+0000 7fbd07e0d740 1 Processor -- start 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.563+0000 7fbd07e0d740 1 -- start start 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11174c40 0x55ec11179740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec1117c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ec110d8340 con 0x55ec11174c40 2024-05-09T09:34:21.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ec11080290 con 0x55ec11179db0 2024-05-09T09:34:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ec110d4300 con 0x55ec11174250 2024-05-09T09:34:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:39052/0 (socket says 172.21.15.69:39052) 2024-05-09T09:34:21.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.564+0000 7fbcfed9d640 1 -- 172.21.15.69:0/1127285090 learned_addr learned my addr 172.21.15.69:0/1127285090 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.565+0000 7fbcfed9d640 1 -- 172.21.15.69:0/1127285090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 msgr2=0x55ec1117c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.565+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec1117c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.565+0000 7fbcfed9d640 1 -- 172.21.15.69:0/1127285090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11174c40 msgr2=0x55ec11179740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:34:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.565+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11174c40 0x55ec11179740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.565+0000 7fbcfed9d640 1 -- 172.21.15.69:0/1127285090 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec110a48e0 con 0x55ec11174250 2024-05-09T09:34:21.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.567+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0010fd0 tx=0x7fbcf0009de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=434afc1a52af7923 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.569+0000 7fbce57fa640 1 -- 172.21.15.69:0/1127285090 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf0020050 con 0x55ec11174250 2024-05-09T09:34:21.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.569+0000 7fbce57fa640 1 -- 172.21.15.69:0/1127285090 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf009f7e0 con 0x55ec11174250 2024-05-09T09:34:21.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.569+0000 7fbce57fa640 1 -- 172.21.15.69:0/1127285090 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf00ccdf0 con 0x55ec11174250 2024-05-09T09:34:21.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.571+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 0x7fbce80a5320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098d0 tx=0x7fbcf8009820 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.572+0000 7fbd07e0d740 1 -- 172.21.15.69:0/1127285090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 msgr2=0x55ec11174670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.572+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fbcf0010fd0 tx=0x7fbcf0009de0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.574+0000 7fbd07e0d740 1 -- 172.21.15.69:0/1127285090 shutdown_connections 2024-05-09T09:34:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.574+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec1117c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.574+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11174c40 0x55ec11179740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.574+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/1127285090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174250 0x55ec11174670 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.574+0000 7fbd07e0d740 1 -- 172.21.15.69:0/1127285090 >> 172.21.15.69:0/1127285090 conn(0x55ec1116f8c0 msgr2=0x55ec1116fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:21.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.576+0000 7fbd07e0d740 1 -- 172.21.15.69:0/1127285090 shutdown_connections 2024-05-09T09:34:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.580+0000 7fbd07e0d740 1 -- 172.21.15.69:0/1127285090 wait complete. 2024-05-09T09:34:21.580 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.580+0000 7fbd07e0d740 1 -- start start 2024-05-09T09:34:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.580+0000 7fbd07e0d740 1 Processor -- start 2024-05-09T09:34:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.580+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 0x55ec1117e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec1117ea00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec11185a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.581 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ec1117cd00 con 0x55ec11179db0 2024-05-09T09:34:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ec11093790 con 0x55ec11174250 2024-05-09T09:34:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ec1117ce80 con 0x55ec11174c40 2024-05-09T09:34:21.582 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.581+0000 7fbd07e0d740 10 client.0 _ll_register_callbacks cb 0x55ec110cea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.583+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 0x55ec1117e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.583+0000 7fbcff59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec11185a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.583+0000 7fbcff59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec11185a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53412/0 (socket says 172.21.15.69:53412) 2024-05-09T09:34:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.583+0000 7fbcff59e640 1 -- 172.21.15.69:0/169766771 learned_addr learned my addr 172.21.15.69:0/169766771 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:21.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.583+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec1117ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 -- 172.21.15.69:0/169766771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 msgr2=0x55ec1117ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec1117ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 -- 172.21.15.69:0/169766771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 msgr2=0x55ec11185a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.584 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec11185a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 -- 172.21.15.69:0/169766771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec10dec210 con 0x55ec11174250 2024-05-09T09:34:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.584+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 0x55ec1117e4c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55ec111754a0 tx=0x7fbcf00970d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b4a8cf899b0220bf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.585+0000 7fbce2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:34:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.589+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf00cc7e0 con 0x55ec11174250 2024-05-09T09:34:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.589+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcf009f7e0 con 0x55ec11174250 2024-05-09T09:34:21.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.589+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbcf00b4210 con 0x55ec11174250 2024-05-09T09:34:21.594 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.594+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbcf0020070 con 0x55ec11174250 2024-05-09T09:34:21.595 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.595+0000 7fbd07e0d740 10 client.26579 fetch_fsmap learned FSMap version 43 2024-05-09T09:34:21.596 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.595+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55ec10eb8620 con 0x55ec11174250 2024-05-09T09:34:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf009f7e0 con 0x55ec11174250 2024-05-09T09:34:21.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fbcf00a5050 con 0x55ec11174250 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 10 client.26579 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 10 client.26579 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 20 client.26579 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec11189ba0 0x55ec1118bf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 4 client.26579 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.599+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ec110e5a70 con 0x55ec11189ba0 2024-05-09T09:34:21.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.600+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec11189ba0 0x55ec1118bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:21 smithi103 ceph-mon[34421]: pgmap v833: 129 pgs: 129 active+clean; 528 MiB data, 5.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.9 MiB/s wr, 589 op/s 2024-05-09T09:34:21.618 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.618+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec11189ba0 0x55ec1118bf90 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=565e5e9ac1b208ff server_cookie=bc3d5c50cffcf0b1 in_seq=0 out_seq=0 2024-05-09T09:34:21.619 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.619+0000 7fbce2ff5640 10 client.26579 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.637+0000 7fbce2ff5640 1 -- 172.21.15.69:0/169766771 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ec110e5a70 con 0x55ec11189ba0 2024-05-09T09:34:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.637+0000 7fbce2ff5640 10 client.26579 handle_command_reply: tid=0 2024-05-09T09:34:21.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.638+0000 7fbd07e0d740 1 client.26579 shutdown 2024-05-09T09:34:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.639+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 msgr2=0x55ec1117e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.640 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.639+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 0x55ec1117e4c0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55ec111754a0 tx=0x7fbcf00970d0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 shutdown_connections 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec11185a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec11189ba0 0x55ec1118bf90 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec1117ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/169766771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174250 0x55ec1117e4c0 secure :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0x55ec111754a0 tx=0x7fbcf00970d0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 >> 172.21.15.69:0/169766771 conn(0x55ec1116f8c0 msgr2=0x55ec1116ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:21.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.641+0000 7fbce2ff5640 0 client.26579 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:21.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.643+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 shutdown_connections 2024-05-09T09:34:21.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.646+0000 7fbd07e0d740 1 -- 172.21.15.69:0/169766771 wait complete. 2024-05-09T09:34:21.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.647+0000 7fbd07e0d740 20 client.26579 trim_cache size 0 max 16384 2024-05-09T09:34:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.685+0000 7fbd07e0d740 1 Processor -- start 2024-05-09T09:34:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.685+0000 7fbd07e0d740 1 -- start start 2024-05-09T09:34:21.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.685+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 0x55ec1118bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.685+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec1118c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec1117e0d0 0x55ec1117e550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec012520 con 0x55ec11179db0 2024-05-09T09:34:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 0x55ec1118bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00fd60 con 0x55ec11174c40 2024-05-09T09:34:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 0x55ec1118bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35374/0 (socket says 172.21.15.69:35374) 2024-05-09T09:34:21.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbcfed9d640 1 -- 172.21.15.69:0/4029959738 learned_addr learned my addr 172.21.15.69:0/4029959738 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:21.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec1118c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.686+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00dbb0 con 0x55ec1117e0d0 2024-05-09T09:34:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.689+0000 7fbcff59e640 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec1117e0d0 0x55ec1117e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.690+0000 7fbcfe59c640 1 -- 172.21.15.69:0/4029959738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec1117e0d0 msgr2=0x55ec1117e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.690+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec1117e0d0 0x55ec1117e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.690+0000 7fbcfe59c640 1 -- 172.21.15.69:0/4029959738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 msgr2=0x55ec1118bbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.690+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 0x55ec1118bbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.690+0000 7fbcfe59c640 1 -- 172.21.15.69:0/4029959738 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ec11173e90 con 0x55ec11179db0 2024-05-09T09:34:21.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.693+0000 7fbcfe59c640 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec1118c0f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009fd0 tx=0x7fbcf8011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ce2e163d2eca85f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.697+0000 7fbce4ff9640 1 -- 172.21.15.69:0/4029959738 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8024b50 con 0x55ec11179db0 2024-05-09T09:34:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.697+0000 7fbce4ff9640 1 -- 172.21.15.69:0/4029959738 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf8029070 con 0x55ec11179db0 2024-05-09T09:34:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 msgr2=0x55ec1118c0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec1118c0f0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009fd0 tx=0x7fbcf8011000 comp rx=0 tx=0).stop 2024-05-09T09:34:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 shutdown_connections 2024-05-09T09:34:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec1117e0d0 0x55ec1117e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec11179db0 0x55ec1118c0f0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/4029959738 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11174c40 0x55ec1118bbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 >> 172.21.15.69:0/4029959738 conn(0x55ec1116f8c0 msgr2=0x55ec1116fca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 shutdown_connections 2024-05-09T09:34:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.698+0000 7fbd07e0d740 1 -- 172.21.15.69:0/4029959738 wait complete. 2024-05-09T09:34:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.699+0000 7fbd07e0d740 1 -- start start 2024-05-09T09:34:21.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 Processor -- start 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec11182540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec11182a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00e220 con 0x55ec1117e0d0 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00f690 con 0x55ec11179db0 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbcec00e8f0 con 0x55ec11174c40 2024-05-09T09:34:21.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.700+0000 7fbd07e0d740 10 client.0 _ll_register_callbacks cb 0x7fbcf009b2d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.702+0000 7fbcfed9d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec11182540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.702+0000 7fbcff59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.702+0000 7fbcfe59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec11182a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.702+0000 7fbcff59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53434/0 (socket says 172.21.15.69:53434) 2024-05-09T09:34:21.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.702+0000 7fbcff59e640 1 -- 172.21.15.69:0/2571048657 learned_addr learned my addr 172.21.15.69:0/2571048657 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:21.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 -- 172.21.15.69:0/2571048657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 msgr2=0x55ec11182540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec11182540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 -- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 msgr2=0x55ec11182a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec11182a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:21.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 -- 172.21.15.69:0/2571048657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ec11184080 con 0x55ec1117e0d0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.705+0000 7fbcff59e640 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbcf402a970 tx=0x7fbcf4030710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230d4c106fc4d3e0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.706+0000 7fbce2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.706+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4041070 con 0x55ec1117e0d0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.706+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbcf8029070 con 0x55ec1117e0d0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.706+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbcf4005450 con 0x55ec1117e0d0 2024-05-09T09:34:21.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.707+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=43) ==== 24+0+0 (secure 0 0 0) 0x7fbcf404c050 con 0x55ec1117e0d0 2024-05-09T09:34:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.714+0000 7fbd07e0d740 10 client.17292 fetch_fsmap learned FSMap version 43 2024-05-09T09:34:21.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.714+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=43,monmap=4+}) -- 0x55ec111effe0 con 0x55ec1117e0d0 2024-05-09T09:34:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbcf8029070 con 0x55ec1117e0d0 2024-05-09T09:34:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 43) ==== 8362+0+0 (secure 0 0 0) 0x7fbcf4004be0 con 0x55ec1117e0d0 2024-05-09T09:34:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 10 client.17292 fetch_fsmap finished waiting for FSMap version 43 2024-05-09T09:34:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 10 client.17292 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:21.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 20 client.17292 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec111f0a40 0x55ec111f2e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 4 client.17292 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:34:21.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.715+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ec110dfdc0 con 0x55ec111f0a40 2024-05-09T09:34:21.718 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.717+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec111f0a40 0x55ec111f2e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.723+0000 7fbce2ff5640 10 client.17292 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:21.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:21.723+0000 7fbcfed9d640 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec111f0a40 0x55ec111f2e60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3803fa1e3a4be15 server_cookie=98b85c8f7caa982d in_seq=0 out_seq=0 2024-05-09T09:34:21.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 78970 32.89 MB/sec execute 19 sec latency 1120.179 ms 2024-05-09T09:34:22.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 79014 31.45 MB/sec execute 20 sec latency 2027.946 ms 2024-05-09T09:34:23.298 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.297+0000 7fbce2ff5640 1 -- 172.21.15.69:0/2571048657 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ec110dfdc0 con 0x55ec111f0a40 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.297+0000 7fbce2ff5640 10 client.17292 handle_command_reply: tid=0 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.297+0000 7fbd07e0d740 1 client.17292 shutdown 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 msgr2=0x55ec11183a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7fbcf402a970 tx=0x7fbcf4030710 comp rx=0 tx=0).stop 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 shutdown_connections 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ec1117e0d0 0x55ec11183a90 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ec11179db0 0x55ec11182a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ec111f0a40 0x55ec111f2e60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 --2- 172.21.15.69:0/2571048657 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ec11174c40 0x55ec11182540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 >> 172.21.15.69:0/2571048657 conn(0x55ec1116f8c0 msgr2=0x55ec1118bbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbce2ff5640 0 client.17292 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:23.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 shutdown_connections 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 1 -- 172.21.15.69:0/2571048657 wait complete. 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.298+0000 7fbd07e0d740 20 client.17292 trim_cache size 0 max 16384 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:34:23.300 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 msgr2=0x7fbce80a5320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 0x7fbce80a5320 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fbcf80098d0 tx=0x7fbcf8009820 comp rx=0 tx=0).stop 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 msgr2=0x7fbd00117d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fbcf000d950 tx=0x7fbcf000de10 comp rx=0 tx=0).stop 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 shutdown_connections 2024-05-09T09:34:23.301 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbce80a2e60 0x7fbce80a5320 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbd0010db50 0x7fbd00119280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbd0010b1a0 0x7fbd00118240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 --2- 172.21.15.69:0/3759749250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbd00108830 0x7fbd00117d00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.299+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 >> 172.21.15.69:0/3759749250 conn(0x7fbd0006e8a0 msgr2=0x7fbd0010b7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.300+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 shutdown_connections 2024-05-09T09:34:23.302 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:23.300+0000 7fbd05027640 1 -- 172.21.15.69:0/3759749250 wait complete. 2024-05-09T09:34:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:23 smithi069 ceph-mon[30123]: pgmap v834: 129 pgs: 129 active+clean; 538 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 506 op/s 2024-05-09T09:34:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:23 smithi163 ceph-mon[34354]: pgmap v834: 129 pgs: 129 active+clean; 538 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 506 op/s 2024-05-09T09:34:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:23 smithi103 ceph-mon[34421]: pgmap v834: 129 pgs: 129 active+clean; 538 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 506 op/s 2024-05-09T09:34:23.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 79291 30.53 MB/sec execute 21 sec latency 423.269 ms 2024-05-09T09:34:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:24 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:24 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:24 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:24.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 79557 29.67 MB/sec execute 22 sec latency 106.287 ms 2024-05-09T09:34:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:25 smithi069 ceph-mon[30123]: pgmap v835: 129 pgs: 129 active+clean; 586 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 543 op/s 2024-05-09T09:34:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:25 smithi163 ceph-mon[34354]: pgmap v835: 129 pgs: 129 active+clean; 586 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 543 op/s 2024-05-09T09:34:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:25 smithi103 ceph-mon[34421]: pgmap v835: 129 pgs: 129 active+clean; 586 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 8.6 MiB/s wr, 543 op/s 2024-05-09T09:34:25.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 79845 29.13 MB/sec execute 23 sec latency 86.478 ms 2024-05-09T09:34:26.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 80412 29.03 MB/sec execute 24 sec latency 119.909 ms 2024-05-09T09:34:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:27 smithi069 ceph-mon[30123]: pgmap v836: 129 pgs: 129 active+clean; 618 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-05-09T09:34:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:27 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:27 smithi163 ceph-mon[34354]: pgmap v836: 129 pgs: 129 active+clean; 618 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-05-09T09:34:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:27 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:27 smithi103 ceph-mon[34421]: pgmap v836: 129 pgs: 129 active+clean; 618 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 12 MiB/s wr, 649 op/s 2024-05-09T09:34:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:27 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:27.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 81000 29.31 MB/sec execute 25 sec latency 71.385 ms 2024-05-09T09:34:28.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 81615 29.34 MB/sec execute 26 sec latency 95.418 ms 2024-05-09T09:34:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:29 smithi069 ceph-mon[30123]: pgmap v837: 129 pgs: 129 active+clean; 582 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 816 op/s 2024-05-09T09:34:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:29 smithi163 ceph-mon[34354]: pgmap v837: 129 pgs: 129 active+clean; 582 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 816 op/s 2024-05-09T09:34:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:29 smithi103 ceph-mon[34421]: pgmap v837: 129 pgs: 129 active+clean; 582 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 14 MiB/s wr, 816 op/s 2024-05-09T09:34:29.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 82348 29.50 MB/sec execute 27 sec latency 61.765 ms 2024-05-09T09:34:30.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 83147 30.24 MB/sec execute 28 sec latency 55.675 ms 2024-05-09T09:34:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:31 smithi069 ceph-mon[30123]: pgmap v838: 129 pgs: 129 active+clean; 526 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-05-09T09:34:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:31 smithi163 ceph-mon[34354]: pgmap v838: 129 pgs: 129 active+clean; 526 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-05-09T09:34:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:31 smithi103 ceph-mon[34421]: pgmap v838: 129 pgs: 129 active+clean; 526 MiB data, 5.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 14 MiB/s wr, 851 op/s 2024-05-09T09:34:31.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 83832 30.60 MB/sec execute 29 sec latency 44.313 ms 2024-05-09T09:34:32.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 84353 30.54 MB/sec execute 30 sec latency 92.790 ms 2024-05-09T09:34:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:33 smithi069 ceph-mon[30123]: pgmap v839: 129 pgs: 129 active+clean; 520 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-05-09T09:34:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:33 smithi163 ceph-mon[34354]: pgmap v839: 129 pgs: 129 active+clean; 520 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-05-09T09:34:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:33 smithi103 ceph-mon[34421]: pgmap v839: 129 pgs: 129 active+clean; 520 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 13 MiB/s wr, 804 op/s 2024-05-09T09:34:33.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 84910 30.25 MB/sec execute 31 sec latency 70.518 ms 2024-05-09T09:34:34.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 85748 30.66 MB/sec execute 32 sec latency 59.121 ms 2024-05-09T09:34:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:35 smithi069 ceph-mon[30123]: pgmap v840: 129 pgs: 129 active+clean; 531 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 14 MiB/s wr, 903 op/s 2024-05-09T09:34:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:35 smithi163 ceph-mon[34354]: pgmap v840: 129 pgs: 129 active+clean; 531 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 14 MiB/s wr, 903 op/s 2024-05-09T09:34:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:35 smithi103 ceph-mon[34421]: pgmap v840: 129 pgs: 129 active+clean; 531 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 14 MiB/s wr, 903 op/s 2024-05-09T09:34:35.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 86566 31.18 MB/sec execute 33 sec latency 42.608 ms 2024-05-09T09:34:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.424+0000 7f76de8fc640 1 -- 172.21.15.69:0/3139648578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d8108830 msgr2=0x7f76d810ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.424+0000 7f76de8fc640 1 --2- 172.21.15.69:0/3139648578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d8108830 0x7f76d810ac60 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f76c8009900 tx=0x7f76c802a410 comp rx=0 tx=0).stop 2024-05-09T09:34:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.425+0000 7f76de8fc640 1 -- 172.21.15.69:0/3139648578 shutdown_connections 2024-05-09T09:34:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.425+0000 7f76de8fc640 1 --2- 172.21.15.69:0/3139648578 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d810db50 0x7f76d810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.425+0000 7f76de8fc640 1 --2- 172.21.15.69:0/3139648578 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76d810b1a0 0x7f76d810d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.425+0000 7f76de8fc640 1 --2- 172.21.15.69:0/3139648578 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d8108830 0x7f76d810ac60 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.425+0000 7f76de8fc640 1 -- 172.21.15.69:0/3139648578 >> 172.21.15.69:0/3139648578 conn(0x7f76d806e8a0 msgr2=0x7f76d8070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.426+0000 7f76de8fc640 1 -- 172.21.15.69:0/3139648578 shutdown_connections 2024-05-09T09:34:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.426+0000 7f76de8fc640 1 -- 172.21.15.69:0/3139648578 wait complete. 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 Processor -- start 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 -- start start 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 0x7f76d81a6910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76d810b1a0 0x7f76d81a6e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76d811c320 con 0x7f76d810db50 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76d811c1a0 con 0x7f76d810b1a0 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76de8fc640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76d811c4a0 con 0x7f76d8108830 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.427+0000 7f76dce99640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38222/0 (socket says 172.21.15.69:38222) 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 -- 172.21.15.69:0/1820330120 learned_addr learned my addr 172.21.15.69:0/1820330120 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76d7fff640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 0x7f76d81a6910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 -- 172.21.15.69:0/1820330120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 msgr2=0x7f76d81a6910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 0x7f76d81a6910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 -- 172.21.15.69:0/1820330120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76d810b1a0 msgr2=0x7f76d81a6e50 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76d810b1a0 0x7f76d81a6e50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 -- 172.21.15.69:0/1820330120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f76d81ae4e0 con 0x7f76d810db50 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76d7fff640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 0x7f76d81a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.428+0000 7f76dce99640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f76d000eb10 tx=0x7f76d000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.429+0000 7f76d57fa640 1 -- 172.21.15.69:0/1820330120 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0004b20 con 0x7f76d810db50 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.429+0000 7f76d57fa640 1 -- 172.21.15.69:0/1820330120 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76d0004cc0 con 0x7f76d810db50 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.429+0000 7f76d57fa640 1 -- 172.21.15.69:0/1820330120 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76d0010760 con 0x7f76d810db50 2024-05-09T09:34:36.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.429+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f76d81ae7d0 con 0x7f76d810db50 2024-05-09T09:34:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.429+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f76d81aec70 con 0x7f76d810db50 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76d57fa640 1 -- 172.21.15.69:0/1820330120 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f76d0005020 con 0x7f76d810db50 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 Processor -- start 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 -- start start 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592f250 0x555a9592f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a95937240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a95893340 con 0x555a9592fc40 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555a9583b290 con 0x555a95934db0 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76e1606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555a9588f300 con 0x555a9592f250 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.432+0000 7f76d57fa640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 0x7f76c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.433 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38236/0 (socket says 172.21.15.69:38236) 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 -- 172.21.15.69:0/917244769 learned_addr learned my addr 172.21.15.69:0/917244769 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d77fe640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592f250 0x555a9592f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d57fa640 1 -- 172.21.15.69:0/1820330120 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f76d00cfac0 con 0x7f76d810db50 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76dce99640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 0x7f76c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d77fe640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a95937240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 -- 172.21.15.69:0/917244769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592f250 msgr2=0x555a9592f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592f250 0x555a9592f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 -- 172.21.15.69:0/917244769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 msgr2=0x555a95937240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a95937240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 -- 172.21.15.69:0/917244769 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a9585f8e0 con 0x555a9592fc40 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d77fe640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a95937240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:34:36.434 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76d7fff640 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f76c8031610 tx=0x7f76c8032cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=743f243069c70701 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76dce99640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 0x7f76c00a51f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f76d0098910 tx=0x7f76d00058d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76be7fc640 1 -- 172.21.15.69:0/917244769 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c803f070 con 0x555a9592fc40 2024-05-09T09:34:36.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76be7fc640 1 -- 172.21.15.69:0/917244769 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c80318b0 con 0x555a9592fc40 2024-05-09T09:34:36.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.433+0000 7f76be7fc640 1 -- 172.21.15.69:0/917244769 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8035070 con 0x555a9592fc40 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 -- 172.21.15.69:0/917244769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 msgr2=0x555a95934740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f76c8031610 tx=0x7f76c8032cb0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 -- 172.21.15.69:0/917244769 shutdown_connections 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a95937240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a95934740 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 --2- 172.21.15.69:0/917244769 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592f250 0x555a9592f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 -- 172.21.15.69:0/917244769 >> 172.21.15.69:0/917244769 conn(0x555a9592a8c0 msgr2=0x555a9592acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 -- 172.21.15.69:0/917244769 shutdown_connections 2024-05-09T09:34:36.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.434+0000 7f76e1606740 1 -- 172.21.15.69:0/917244769 wait complete. 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.435+0000 7f76e1606740 1 -- start start 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.435+0000 7f76e1606740 1 Processor -- start 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.435+0000 7f76e1606740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a95939980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a959409c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38248/0 (socket says 172.21.15.69:38248) 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 -- 172.21.15.69:0/2174523921 learned_addr learned my addr 172.21.15.69:0/2174523921 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555a95937d00 con 0x555a9592f250 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555a9584e790 con 0x555a95934db0 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555a95937e80 con 0x555a9592fc40 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d7fff640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a95939980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76e1606740 10 client.0 _ll_register_callbacks cb 0x555a95889a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:36.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a959409c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 -- 172.21.15.69:0/2174523921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 msgr2=0x555a95939980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a95939980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 -- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 msgr2=0x555a959409c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a959409c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.436+0000 7f76d77fe640 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a955a7210 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76d77fe640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f76cc005bc0 tx=0x7f76cc004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b8d1d38dcbeb9bd1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc014170 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f76cc004030 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76c80318b0 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f76cc025050 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.437+0000 7f76e1606740 10 client.17304 fetch_fsmap learned FSMap version 45 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555a95673620 con 0x555a9592f250 2024-05-09T09:34:36.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76c80318b0 con 0x555a9592f250 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f76cc003c90 con 0x555a9592f250 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 10 client.17304 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 10 client.17304 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 20 client.17304 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a95945320 0x555a959477e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 4 client.17304 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.438+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555a958a0a70 con 0x555a95945320 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.439+0000 7f76d7fff640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a95945320 0x555a959477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.440+0000 7f76b77fe640 10 client.17304 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:36.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.440+0000 7f76d7fff640 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a95945320 0x555a959477e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=217523f59d43ab15 server_cookie=d2f3a7d2a952a338 in_seq=0 out_seq=0 2024-05-09T09:34:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.444+0000 7f76b77fe640 1 -- 172.21.15.69:0/2174523921 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555a958a0a70 con 0x555a95945320 2024-05-09T09:34:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.444+0000 7f76b77fe640 10 client.17304 handle_command_reply: tid=0 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.444+0000 7f76e1606740 1 client.17304 shutdown 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 msgr2=0x555a95939440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f76cc005bc0 tx=0x7f76cc004730 comp rx=0 tx=0).stop 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 shutdown_connections 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a95934db0 0x555a959409c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a95945320 0x555a959477e0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a95939980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 --2- 172.21.15.69:0/2174523921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592f250 0x555a95939440 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76b77fe640 0 client.17304 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 >> 172.21.15.69:0/2174523921 conn(0x555a9592a8c0 msgr2=0x555a95929a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 shutdown_connections 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 1 -- 172.21.15.69:0/2174523921 wait complete. 2024-05-09T09:34:36.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.445+0000 7f76e1606740 20 client.17304 trim_cache size 0 max 16384 2024-05-09T09:34:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 Processor -- start 2024-05-09T09:34:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 -- start start 2024-05-09T09:34:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a95930e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a959313a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81bbb70 con 0x555a95934db0 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b93e0 con 0x555a9592f250 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76e1606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b7230 con 0x555a9592fc40 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38264/0 (socket says 172.21.15.69:38264) 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76d77fe640 1 -- 172.21.15.69:0/793575674 learned_addr learned my addr 172.21.15.69:0/793575674 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76d7fff640 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a959313a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.463+0000 7f76d77fe640 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a95930e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 -- 172.21.15.69:0/793575674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 msgr2=0x555a959313a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a959313a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 -- 172.21.15.69:0/793575674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 msgr2=0x555a95930e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a95930e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 -- 172.21.15.69:0/793575674 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555a95928f00 con 0x555a95934db0 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76d77fe640 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009870 tx=0x7f76cc005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1e449f25dd4798b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76bdffb640 1 -- 172.21.15.69:0/793575674 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc005bf0 con 0x555a95934db0 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76bdffb640 1 -- 172.21.15.69:0/793575674 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc011b90 con 0x555a95934db0 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.464+0000 7f76bdffb640 1 -- 172.21.15.69:0/793575674 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76cc021810 con 0x555a95934db0 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- 172.21.15.69:0/793575674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 msgr2=0x555a95940980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f76cc009870 tx=0x7f76cc005f90 comp rx=0 tx=0).stop 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- 172.21.15.69:0/793575674 shutdown_connections 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a95934db0 0x555a95940980 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a9592fc40 0x555a959313a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 --2- 172.21.15.69:0/793575674 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a95930e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- 172.21.15.69:0/793575674 >> 172.21.15.69:0/793575674 conn(0x555a9592a8c0 msgr2=0x555a9592aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- 172.21.15.69:0/793575674 shutdown_connections 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- 172.21.15.69:0/793575674 wait complete. 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.465+0000 7f76e1606740 1 -- start start 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 Processor -- start 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a959404d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a95934db0 0x555a9593d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b78a0 con 0x555a9592fc40 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b8d10 con 0x555a9592f250 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f76d81b7f70 con 0x555a95934db0 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76e1606740 10 client.0 _ll_register_callbacks cb 0x7f76cc009740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38272/0 (socket says 172.21.15.69:38272) 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d77fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a959404d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.470 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 -- 172.21.15.69:0/4136897317 learned_addr learned my addr 172.21.15.69:0/4136897317 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d77fe640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a95934db0 0x555a9593d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 -- 172.21.15.69:0/4136897317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a95934db0 msgr2=0x555a9593d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a95934db0 0x555a9593d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 -- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 msgr2=0x555a959404d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a959404d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.466+0000 7f76d7fff640 1 -- 172.21.15.69:0/4136897317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555a95940b00 con 0x555a9592fc40 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76d7fff640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005c40 tx=0x7f76c8005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bf86fe5971523f5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f76c8052060 con 0x555a9592fc40 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f76cc011b90 con 0x555a9592fc40 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f76c804d620 con 0x555a9592fc40 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f76c8059050 con 0x555a9592fc40 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 10 client.17310 fetch_fsmap learned FSMap version 45 2024-05-09T09:34:36.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x555a959aa930 con 0x555a9592fc40 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f76cc011b90 con 0x555a9592fc40 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f76c8043400 con 0x555a9592fc40 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 10 client.17310 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 10 client.17310 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 20 client.17310 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a959aaf10 0x555a959ad300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 4 client.17310 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.467+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555a9589adc0 con 0x555a959aaf10 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.468+0000 7f76d77fe640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a959aaf10 0x555a959ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.468+0000 7f76d77fe640 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a959aaf10 0x555a959ad300 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e99fea985d499b5c server_cookie=efc002be0eebccfb in_seq=0 out_seq=0 2024-05-09T09:34:36.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.468+0000 7f76b77fe640 10 client.17310 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:36.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 87049 31.22 MB/sec execute 34 sec latency 267.892 ms 2024-05-09T09:34:36.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76b77fe640 1 -- 172.21.15.69:0/4136897317 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555a9589adc0 con 0x555a959aaf10 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76b77fe640 10 client.17310 handle_command_reply: tid=0 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 client.17310 shutdown 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 msgr2=0x555a9593cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f76c8005c40 tx=0x7f76c8005b50 comp rx=0 tx=0).stop 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 shutdown_connections 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555a959aaf10 0x555a959ad300 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76b77fe640 0 client.17310 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:36.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555a95934db0 0x555a9593d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555a9592fc40 0x555a9593cfc0 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 --2- 172.21.15.69:0/4136897317 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555a9592f250 0x555a959404d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.997+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 >> 172.21.15.69:0/4136897317 conn(0x555a9592a8c0 msgr2=0x555a95930e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.998+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 shutdown_connections 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.998+0000 7f76e1606740 1 -- 172.21.15.69:0/4136897317 wait complete. 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.998+0000 7f76e1606740 20 client.17310 trim_cache size 0 max 16384 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:34:36.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 msgr2=0x7f76c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 0x7f76c00a51f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f76d0098910 tx=0x7f76d00058d0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 msgr2=0x7f76d81ade70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f76d000eb10 tx=0x7f76d000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 shutdown_connections 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f76c00a2d30 0x7f76c00a51f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f76d810db50 0x7f76d81ade70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f76d810b1a0 0x7f76d81a6e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 --2- 172.21.15.69:0/1820330120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f76d8108830 0x7f76d81a6910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:36.999+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 >> 172.21.15.69:0/1820330120 conn(0x7f76d806e8a0 msgr2=0x7f76d810d390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:37.000+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 shutdown_connections 2024-05-09T09:34:37.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:37.000+0000 7f76de8fc640 1 -- 172.21.15.69:0/1820330120 wait complete. 2024-05-09T09:34:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:37 smithi069 ceph-mon[30123]: pgmap v841: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 9.2 MiB/s wr, 816 op/s 2024-05-09T09:34:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:37 smithi163 ceph-mon[34354]: pgmap v841: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 9.2 MiB/s wr, 816 op/s 2024-05-09T09:34:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:37 smithi103 ceph-mon[34421]: pgmap v841: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 9.2 MiB/s wr, 816 op/s 2024-05-09T09:34:37.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 87541 31.10 MB/sec execute 35 sec latency 401.045 ms 2024-05-09T09:34:38.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 88041 30.78 MB/sec execute 36 sec latency 65.282 ms 2024-05-09T09:34:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:39 smithi069 ceph-mon[30123]: pgmap v842: 129 pgs: 129 active+clean; 539 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.8 MiB/s wr, 754 op/s 2024-05-09T09:34:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:39 smithi163 ceph-mon[34354]: pgmap v842: 129 pgs: 129 active+clean; 539 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.8 MiB/s wr, 754 op/s 2024-05-09T09:34:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:39 smithi103 ceph-mon[34421]: pgmap v842: 129 pgs: 129 active+clean; 539 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 5.8 MiB/s wr, 754 op/s 2024-05-09T09:34:39.824 INFO:tasks.workunit.client.0.smithi069.stdout: 10 88731 30.88 MB/sec execute 37 sec latency 43.092 ms 2024-05-09T09:34:40.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 89632 31.39 MB/sec execute 38 sec latency 85.810 ms 2024-05-09T09:34:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:41 smithi069 ceph-mon[30123]: pgmap v843: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 650 op/s 2024-05-09T09:34:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:41 smithi163 ceph-mon[34354]: pgmap v843: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 650 op/s 2024-05-09T09:34:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:41 smithi103 ceph-mon[34421]: pgmap v843: 129 pgs: 129 active+clean; 533 MiB data, 5.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 4.2 MiB/s wr, 650 op/s 2024-05-09T09:34:41.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 90231 31.47 MB/sec execute 39 sec latency 57.050 ms 2024-05-09T09:34:42.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 90816 31.61 MB/sec execute 40 sec latency 60.306 ms 2024-05-09T09:34:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:43 smithi069 ceph-mon[30123]: pgmap v844: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 565 op/s 2024-05-09T09:34:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:43 smithi163 ceph-mon[34354]: pgmap v844: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 565 op/s 2024-05-09T09:34:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:43 smithi103 ceph-mon[34421]: pgmap v844: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 565 op/s 2024-05-09T09:34:43.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 91372 31.64 MB/sec execute 41 sec latency 68.642 ms 2024-05-09T09:34:44.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 91984 31.38 MB/sec execute 42 sec latency 90.413 ms 2024-05-09T09:34:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:45 smithi069 ceph-mon[30123]: pgmap v845: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.1 MiB/s wr, 637 op/s 2024-05-09T09:34:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:45 smithi163 ceph-mon[34354]: pgmap v845: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.1 MiB/s wr, 637 op/s 2024-05-09T09:34:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:45 smithi103 ceph-mon[34421]: pgmap v845: 129 pgs: 129 active+clean; 530 MiB data, 5.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.1 MiB/s wr, 637 op/s 2024-05-09T09:34:45.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 92665 31.60 MB/sec execute 43 sec latency 78.964 ms 2024-05-09T09:34:46.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 93452 31.79 MB/sec execute 44 sec latency 40.693 ms 2024-05-09T09:34:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:47 smithi069 ceph-mon[30123]: pgmap v846: 129 pgs: 129 active+clean; 536 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:34:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:47 smithi163 ceph-mon[34354]: pgmap v846: 129 pgs: 129 active+clean; 536 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:34:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:47 smithi103 ceph-mon[34421]: pgmap v846: 129 pgs: 129 active+clean; 536 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:34:47.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 93917 31.75 MB/sec execute 45 sec latency 134.258 ms 2024-05-09T09:34:48.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 94557 31.80 MB/sec execute 46 sec latency 133.168 ms 2024-05-09T09:34:49.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:49 smithi069 ceph-mon[30123]: pgmap v847: 129 pgs: 129 active+clean; 540 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 602 op/s 2024-05-09T09:34:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:49 smithi163 ceph-mon[34354]: pgmap v847: 129 pgs: 129 active+clean; 540 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 602 op/s 2024-05-09T09:34:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:49 smithi103 ceph-mon[34421]: pgmap v847: 129 pgs: 129 active+clean; 540 MiB data, 6.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 3.8 MiB/s wr, 602 op/s 2024-05-09T09:34:49.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 95059 31.68 MB/sec execute 47 sec latency 74.494 ms 2024-05-09T09:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.114+0000 7f32f0542640 1 -- 172.21.15.69:0/2448299357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8101980 msgr2=0x7f32e8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.114+0000 7f32f0542640 1 --2- 172.21.15.69:0/2448299357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8101980 0x7f32e8101e00 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f32d8009900 tx=0x7f32d802a410 comp rx=0 tx=0).stop 2024-05-09T09:34:50.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 -- 172.21.15.69:0/2448299357 shutdown_connections 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 --2- 172.21.15.69:0/2448299357 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8102340 0x7f32e8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 --2- 172.21.15.69:0/2448299357 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8101980 0x7f32e8101e00 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 --2- 172.21.15.69:0/2448299357 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8100790 0x7f32e8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 -- 172.21.15.69:0/2448299357 >> 172.21.15.69:0/2448299357 conn(0x7f32e80fbce0 msgr2=0x7f32e80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 -- 172.21.15.69:0/2448299357 shutdown_connections 2024-05-09T09:34:50.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 -- 172.21.15.69:0/2448299357 wait complete. 2024-05-09T09:34:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.115+0000 7f32f0542640 1 Processor -- start 2024-05-09T09:34:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 -- start start 2024-05-09T09:34:50.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 0x7f32e8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 0x7f32e8196f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 0x7f32e819dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810c410 con 0x7f32e8102340 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 0x7f32e8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 0x7f32e819dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810c290 con 0x7f32e8101980 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 0x7f32e8196a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:47332/0 (socket says 172.21.15.69:47332) 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32ee2de640 1 -- 172.21.15.69:0/4170940988 learned_addr learned my addr 172.21.15.69:0/4170940988 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32e810c590 con 0x7f32e8100790 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32edadd640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 0x7f32e8196f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 -- 172.21.15.69:0/4170940988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 msgr2=0x7f32e8196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 0x7f32e8196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 -- 172.21.15.69:0/4170940988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 msgr2=0x7f32e8196f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 0x7f32e8196f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.116+0000 7f32eeadf640 1 -- 172.21.15.69:0/4170940988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f32e819e620 con 0x7f32e8102340 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32edadd640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 0x7f32e8196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32eeadf640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 0x7f32e819dfb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f32e0010b80 tx=0x7f32e000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32df7fe640 1 -- 172.21.15.69:0/4170940988 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e0004220 con 0x7f32e8102340 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32df7fe640 1 -- 172.21.15.69:0/4170940988 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e00043c0 con 0x7f32e8102340 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f32e8069b30 con 0x7f32e8102340 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32df7fe640 1 -- 172.21.15.69:0/4170940988 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e00035f0 con 0x7f32e8102340 2024-05-09T09:34:50.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.117+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f32e806a050 con 0x7f32e8102340 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.120+0000 7f32df7fe640 1 -- 172.21.15.69:0/4170940988 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f32e001f020 con 0x7f32e8102340 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.120+0000 7f32f3251740 1 Processor -- start 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.120+0000 7f32f3251740 1 -- start start 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020eaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf0204a340 con 0x55cf020f01c0 2024-05-09T09:34:50.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf01ff2290 con 0x55cf020ea660 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32f3251740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf02046300 con 0x55cf020eb050 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32edadd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32df7fe640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 0x7f32d40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32edadd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49226/0 (socket says 172.21.15.69:49226) 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32edadd640 1 -- 172.21.15.69:0/3450881895 learned_addr learned my addr 172.21.15.69:0/3450881895 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:47340/0 (socket says 172.21.15.69:47340) 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32edadd640 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020eaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32eeadf640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 0x7f32d40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 -- 172.21.15.69:0/3450881895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 msgr2=0x55cf020eaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020eaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 -- 172.21.15.69:0/3450881895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 msgr2=0x55cf020f2650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 -- 172.21.15.69:0/3450881895 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf020168e0 con 0x55cf020eb050 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32edadd640 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32df7fe640 1 -- 172.21.15.69:0/4170940988 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f32e00d1230 con 0x7f32e8102340 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.121+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32e4011440 tx=0x7f32e4011900 comp rx=0 tx=0).ready entity=mon.1 client_cookie=56d13effabba95a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32dcbfe640 1 -- 172.21.15.69:0/3450881895 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e4003970 con 0x55cf020eb050 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32dcbfe640 1 -- 172.21.15.69:0/3450881895 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e4004020 con 0x55cf020eb050 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32dcbfe640 1 -- 172.21.15.69:0/3450881895 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32e400f8a0 con 0x55cf020eb050 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32f3251740 1 -- 172.21.15.69:0/3450881895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 msgr2=0x55cf020efb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32f3251740 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f32e4011440 tx=0x7f32e4011900 comp rx=0 tx=0).stop 2024-05-09T09:34:50.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.122+0000 7f32eeadf640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 0x7f32d40a50f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f32e000cba0 tx=0x7f32e0004570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 -- 172.21.15.69:0/3450881895 shutdown_connections 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f2650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020efb50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 --2- 172.21.15.69:0/3450881895 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020eaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 -- 172.21.15.69:0/3450881895 >> 172.21.15.69:0/3450881895 conn(0x55cf020e18c0 msgr2=0x55cf020e1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 -- 172.21.15.69:0/3450881895 shutdown_connections 2024-05-09T09:34:50.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 -- 172.21.15.69:0/3450881895 wait complete. 2024-05-09T09:34:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 -- start start 2024-05-09T09:34:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 Processor -- start 2024-05-09T09:34:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.123+0000 7f32f3251740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 0x55cf020f4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020fbe10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf020f30e0 con 0x55cf020f01c0 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf02005790 con 0x55cf020eb050 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf020f3260 con 0x55cf020ea660 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32f3251740 10 client.0 _ll_register_callbacks cb 0x55cf02040a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 0x55cf020f4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:47346/0 (socket says 172.21.15.69:47346) 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32ee2de640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 0x55cf020f4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:56302/0 (socket says 172.21.15.69:56302) 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 -- 172.21.15.69:0/2375246911 learned_addr learned my addr 172.21.15.69:0/2375246911 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020fbe10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 -- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 msgr2=0x55cf020f4dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 0x55cf020f4dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 -- 172.21.15.69:0/2375246911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 msgr2=0x55cf020fbe10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020fbe10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 -- 172.21.15.69:0/2375246911 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf01d5e210 con 0x55cf020ea660 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32edadd640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006010 tx=0x7f32d802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ebcafa6d06d5ee82 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.124+0000 7f32d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8035040 con 0x55cf020ea660 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32e4004020 con 0x55cf020ea660 2024-05-09T09:34:50.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f32d8004020 con 0x55cf020ea660 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32d8053050 con 0x55cf020ea660 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32f3251740 10 client.26527 fetch_fsmap learned FSMap version 45 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.125+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cf01e2a620 con 0x55cf020ea660 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32e4004020 con 0x55cf020ea660 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f32d803a040 con 0x55cf020ea660 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 10 client.26527 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 10 client.26527 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 20 client.26527 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02100380 0x55cf021027f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 4 client.26527 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf02057a70 con 0x55cf02100380 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.126+0000 7f32ee2de640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02100380 0x55cf021027f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.127+0000 7f32ee2de640 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02100380 0x55cf021027f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7e59c32e74d500ca server_cookie=ba863c0a1b89870e in_seq=0 out_seq=0 2024-05-09T09:34:50.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.127+0000 7f32d1ffb640 10 client.26527 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32d1ffb640 1 -- 172.21.15.69:0/2375246911 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf02057a70 con 0x55cf02100380 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32d1ffb640 10 client.26527 handle_command_reply: tid=0 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 client.26527 shutdown 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 msgr2=0x55cf020f4890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f32d8006010 tx=0x7f32d802a410 comp rx=0 tx=0).stop 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 shutdown_connections 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020fbe10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020eb050 0x55cf020f4dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02100380 0x55cf021027f0 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 --2- 172.21.15.69:0/2375246911 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020ea660 0x55cf020f4890 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32d1ffb640 0 client.26527 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:50.133 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 >> 172.21.15.69:0/2375246911 conn(0x55cf020e18c0 msgr2=0x55cf020e0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 shutdown_connections 2024-05-09T09:34:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.130+0000 7f32f3251740 1 -- 172.21.15.69:0/2375246911 wait complete. 2024-05-09T09:34:50.134 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.131+0000 7f32f3251740 20 client.26527 trim_cache size 0 max 16384 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.151+0000 7f32f3251740 1 Processor -- start 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.151+0000 7f32f3251740 1 -- start start 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f44a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020f01c0 0x55cf020f49e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81aaab0 con 0x55cf020ea660 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81a8320 con 0x55cf020f01c0 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32f3251740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81a6170 con 0x55cf020eb050 2024-05-09T09:34:50.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49232/0 (socket says 172.21.15.69:49232) 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 -- 172.21.15.69:0/4225607384 learned_addr learned my addr 172.21.15.69:0/4225607384 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32ee2de640 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f44a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020f01c0 0x55cf020f49e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 -- 172.21.15.69:0/4225607384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 msgr2=0x55cf020f44a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f44a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 -- 172.21.15.69:0/4225607384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020f01c0 msgr2=0x55cf020f49e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020f01c0 0x55cf020f49e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 -- 172.21.15.69:0/4225607384 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf020ebd20 con 0x55cf020ea660 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32edadd640 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005c40 tx=0x7f32d8031400 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e82931c0306161c6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32d3fff640 1 -- 172.21.15.69:0/4225607384 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8035070 con 0x55cf020ea660 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32d3fff640 1 -- 172.21.15.69:0/4225607384 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d8031840 con 0x55cf020ea660 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.152+0000 7f32d3fff640 1 -- 172.21.15.69:0/4225607384 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8042040 con 0x55cf020ea660 2024-05-09T09:34:50.157 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 -- 172.21.15.69:0/4225607384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 msgr2=0x55cf020e5060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f32d8005c40 tx=0x7f32d8031400 comp rx=0 tx=0).stop 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 -- 172.21.15.69:0/4225607384 shutdown_connections 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020f01c0 0x55cf020f49e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f44a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 --2- 172.21.15.69:0/4225607384 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020ea660 0x55cf020e5060 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 -- 172.21.15.69:0/4225607384 >> 172.21.15.69:0/4225607384 conn(0x55cf020e18c0 msgr2=0x55cf020e4950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 -- 172.21.15.69:0/4225607384 shutdown_connections 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.153+0000 7f32f3251740 1 -- 172.21.15.69:0/4225607384 wait complete. 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 -- start start 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 Processor -- start 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020f8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81a67e0 con 0x55cf020f01c0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81a7c50 con 0x55cf020ea660 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f32e81a6eb0 con 0x55cf020eb050 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32f3251740 10 client.0 _ll_register_callbacks cb 0x7f32d8031ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020f8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020f8960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:56324/0 (socket says 172.21.15.69:56324) 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 -- 172.21.15.69:0/3270513797 learned_addr learned my addr 172.21.15.69:0/3270513797 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 -- 172.21.15.69:0/3270513797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 msgr2=0x55cf020f8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 -- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 msgr2=0x55cf020f8960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020f8960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.154+0000 7f32edadd640 1 -- 172.21.15.69:0/3270513797 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf020fa4a0 con 0x55cf020f01c0 2024-05-09T09:34:50.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32edadd640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f9eb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f32d8003940 tx=0x7f32d8031b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b84922a0b1e7e7d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32d1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f32d8060040 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f32d803d8a0 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f32d8031840 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) ==== 24+0+0 (secure 0 0 0) 0x7f32d8035040 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32f3251740 10 client.17328 fetch_fsmap learned FSMap version 45 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) -- 0x55cf02166970 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.155+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f32d8031840 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 45) ==== 8354+0+0 (secure 0 0 0) 0x7f32d8064020 con 0x55cf020f01c0 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 10 client.17328 fetch_fsmap finished waiting for FSMap version 45 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 10 client.17328 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 20 client.17328 populate_metadata read hostname 'smithi069' 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02166ef0 0x55cf02169310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 4 client.17328 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:34:50.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55cf02051dc0 con 0x55cf02166ef0 2024-05-09T09:34:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02166ef0 0x55cf02169310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:34:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32d1ffb640 10 client.17328 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:50.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.156+0000 7f32ee2de640 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02166ef0 0x55cf02169310 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8dba7d83665cd6d server_cookie=e2a19b6bb542aa2d in_seq=0 out_seq=0 2024-05-09T09:34:50.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 95475 31.40 MB/sec execute 48 sec latency 430.012 ms 2024-05-09T09:34:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32d1ffb640 1 -- 172.21.15.69:0/3270513797 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55cf02051dc0 con 0x55cf02166ef0 2024-05-09T09:34:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32d1ffb640 10 client.17328 handle_command_reply: tid=0 2024-05-09T09:34:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 client.17328 shutdown 2024-05-09T09:34:50.836 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 msgr2=0x55cf020f9eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f9eb0 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f32d8003940 tx=0x7f32d8031b70 comp rx=0 tx=0).stop 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 shutdown_connections 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf020f01c0 0x55cf020f9eb0 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf02166ef0 0x55cf02169310 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32d1ffb640 0 client.17328 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf020eb050 0x55cf020f8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 --2- 172.21.15.69:0/3270513797 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf020ea660 0x55cf020f8960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.837 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 >> 172.21.15.69:0/3270513797 conn(0x55cf020e18c0 msgr2=0x55cf020f44a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 shutdown_connections 2024-05-09T09:34:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.836+0000 7f32f3251740 1 -- 172.21.15.69:0/3270513797 wait complete. 2024-05-09T09:34:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f3251740 20 client.17328 trim_cache size 0 max 16384 2024-05-09T09:34:50.838 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 msgr2=0x7f32d40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 0x7f32d40a50f0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f32e000cba0 tx=0x7f32e0004570 comp rx=0 tx=0).stop 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 msgr2=0x7f32e819dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:34:50.839 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 0x7f32e819dfb0 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f32e0010b80 tx=0x7f32e000c6a0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 shutdown_connections 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f32d40a2c30 0x7f32d40a50f0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.837+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f32e8102340 0x7f32e819dfb0 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.838+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f32e8101980 0x7f32e8196f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.838+0000 7f32f0542640 1 --2- 172.21.15.69:0/4170940988 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f32e8100790 0x7f32e8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.838+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 >> 172.21.15.69:0/4170940988 conn(0x7f32e80fbce0 msgr2=0x7f32e8104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.838+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 shutdown_connections 2024-05-09T09:34:50.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:34:50.838+0000 7f32f0542640 1 -- 172.21.15.69:0/4170940988 wait complete. 2024-05-09T09:34:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:51 smithi069 ceph-mon[30123]: pgmap v848: 129 pgs: 129 active+clean; 551 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 558 op/s 2024-05-09T09:34:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:51 smithi163 ceph-mon[34354]: pgmap v848: 129 pgs: 129 active+clean; 551 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 558 op/s 2024-05-09T09:34:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:51 smithi103 ceph-mon[34421]: pgmap v848: 129 pgs: 129 active+clean; 551 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.7 MiB/s wr, 558 op/s 2024-05-09T09:34:51.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 96273 31.65 MB/sec execute 49 sec latency 57.255 ms 2024-05-09T09:34:52.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 96739 31.59 MB/sec execute 50 sec latency 569.314 ms 2024-05-09T09:34:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:53 smithi069 ceph-mon[30123]: pgmap v849: 129 pgs: 129 active+clean; 552 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-05-09T09:34:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:53 smithi163 ceph-mon[34354]: pgmap v849: 129 pgs: 129 active+clean; 552 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-05-09T09:34:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:53 smithi103 ceph-mon[34421]: pgmap v849: 129 pgs: 129 active+clean; 552 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 468 op/s 2024-05-09T09:34:53.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 97353 31.62 MB/sec execute 51 sec latency 61.146 ms 2024-05-09T09:34:54.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 98118 31.75 MB/sec execute 52 sec latency 70.660 ms 2024-05-09T09:34:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:55 smithi069 ceph-mon[30123]: pgmap v850: 129 pgs: 129 active+clean; 553 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-09T09:34:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:55 smithi163 ceph-mon[34354]: pgmap v850: 129 pgs: 129 active+clean; 553 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-09T09:34:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:55 smithi103 ceph-mon[34421]: pgmap v850: 129 pgs: 129 active+clean; 553 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.8 MiB/s wr, 518 op/s 2024-05-09T09:34:55.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 98800 31.95 MB/sec execute 53 sec latency 51.875 ms 2024-05-09T09:34:56.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 99318 31.88 MB/sec execute 54 sec latency 203.991 ms 2024-05-09T09:34:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:57 smithi069 ceph-mon[30123]: pgmap v851: 129 pgs: 129 active+clean; 555 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 571 op/s 2024-05-09T09:34:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:57 smithi163 ceph-mon[34354]: pgmap v851: 129 pgs: 129 active+clean; 555 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 571 op/s 2024-05-09T09:34:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:57 smithi103 ceph-mon[34421]: pgmap v851: 129 pgs: 129 active+clean; 555 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.7 MiB/s wr, 571 op/s 2024-05-09T09:34:57.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 99776 31.72 MB/sec execute 55 sec latency 78.625 ms 2024-05-09T09:34:58.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:58 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:58 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:58.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:58 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:34:58.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 100450 31.70 MB/sec execute 56 sec latency 61.670 ms 2024-05-09T09:34:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:34:59 smithi069 ceph-mon[30123]: pgmap v852: 129 pgs: 129 active+clean; 567 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.4 MiB/s wr, 629 op/s 2024-05-09T09:34:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:34:59 smithi163 ceph-mon[34354]: pgmap v852: 129 pgs: 129 active+clean; 567 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.4 MiB/s wr, 629 op/s 2024-05-09T09:34:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:34:59 smithi103 ceph-mon[34421]: pgmap v852: 129 pgs: 129 active+clean; 567 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 4.4 MiB/s wr, 629 op/s 2024-05-09T09:34:59.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 101267 32.02 MB/sec execute 57 sec latency 55.699 ms 2024-05-09T09:35:00.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102050 32.13 MB/sec execute 58 sec latency 52.328 ms 2024-05-09T09:35:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:01 smithi069 ceph-mon[30123]: pgmap v853: 129 pgs: 129 active+clean; 569 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.9 MiB/s wr, 622 op/s 2024-05-09T09:35:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:01 smithi163 ceph-mon[34354]: pgmap v853: 129 pgs: 129 active+clean; 569 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.9 MiB/s wr, 622 op/s 2024-05-09T09:35:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:01 smithi103 ceph-mon[34421]: pgmap v853: 129 pgs: 129 active+clean; 569 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 4.9 MiB/s wr, 622 op/s 2024-05-09T09:35:01.825 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102547 32.15 MB/sec execute 59 sec latency 159.471 ms 2024-05-09T09:35:02.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102550 31.62 MB/sec execute 60 sec latency 1159.560 ms 2024-05-09T09:35:03.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:03 smithi163 ceph-mon[34354]: pgmap v854: 129 pgs: 129 active+clean; 572 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 570 op/s 2024-05-09T09:35:03.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:03 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:35:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:03 smithi103 ceph-mon[34421]: pgmap v854: 129 pgs: 129 active+clean; 572 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 570 op/s 2024-05-09T09:35:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:03 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:35:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:03 smithi069 ceph-mon[30123]: pgmap v854: 129 pgs: 129 active+clean; 572 MiB data, 6.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 4.7 MiB/s wr, 570 op/s 2024-05-09T09:35:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:03 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:35:03.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102561 31.10 MB/sec execute 61 sec latency 2089.699 ms 2024-05-09T09:35:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.002+0000 7fbf1deba640 1 -- 172.21.15.69:0/885717175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810b1a0 msgr2=0x7fbf1810d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.002+0000 7fbf1deba640 1 --2- 172.21.15.69:0/885717175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810b1a0 0x7fbf1810d610 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009ca0 tx=0x7fbf1002a8d0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.005+0000 7fbf1deba640 1 -- 172.21.15.69:0/885717175 shutdown_connections 2024-05-09T09:35:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.005+0000 7fbf1deba640 1 --2- 172.21.15.69:0/885717175 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810db50 0x7fbf1810dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.005+0000 7fbf1deba640 1 --2- 172.21.15.69:0/885717175 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810b1a0 0x7fbf1810d610 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.005+0000 7fbf1deba640 1 --2- 172.21.15.69:0/885717175 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf18108830 0x7fbf1810ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.005+0000 7fbf1deba640 1 -- 172.21.15.69:0/885717175 >> 172.21.15.69:0/885717175 conn(0x7fbf1806e8a0 msgr2=0x7fbf18070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.007+0000 7fbf1deba640 1 -- 172.21.15.69:0/885717175 shutdown_connections 2024-05-09T09:35:04.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.010+0000 7fbf1deba640 1 -- 172.21.15.69:0/885717175 wait complete. 2024-05-09T09:35:04.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.010+0000 7fbf1deba640 1 Processor -- start 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 -- start start 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf18108830 0x7fbf181a6a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810b1a0 0x7fbf181a6f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1811c430 con 0x7fbf18108830 2024-05-09T09:35:04.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1811c2b0 con 0x7fbf1810b1a0 2024-05-09T09:35:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.011+0000 7fbf1deba640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbf1811c5b0 con 0x7fbf1810db50 2024-05-09T09:35:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.012+0000 7fbf17fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.012+0000 7fbf17fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51450/0 (socket says 172.21.15.69:51450) 2024-05-09T09:35:04.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.012+0000 7fbf17fff640 1 -- 172.21.15.69:0/1741770563 learned_addr learned my addr 172.21.15.69:0/1741770563 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810b1a0 0x7fbf181a6f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf17fff640 1 -- 172.21.15.69:0/1741770563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810b1a0 msgr2=0x7fbf181a6f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf17fff640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810b1a0 0x7fbf181a6f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf17fff640 1 -- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf18108830 msgr2=0x7fbf181a6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf17fff640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf18108830 0x7fbf181a6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.013+0000 7fbf17fff640 1 -- 172.21.15.69:0/1741770563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fbf18110090 con 0x7fbf1810db50 2024-05-09T09:35:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf17fff640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800b3f0 tx=0x7fbf0800b8b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf14ff9640 1 -- 172.21.15.69:0/1741770563 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf08015020 con 0x7fbf1810db50 2024-05-09T09:35:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf14ff9640 1 -- 172.21.15.69:0/1741770563 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf08011d10 con 0x7fbf1810db50 2024-05-09T09:35:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf14ff9640 1 -- 172.21.15.69:0/1741770563 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0800f2e0 con 0x7fbf1810db50 2024-05-09T09:35:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fbf18110380 con 0x7fbf1810db50 2024-05-09T09:35:04.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.014+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fbf18110900 con 0x7fbf1810db50 2024-05-09T09:35:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.019+0000 7fbf20bca740 1 Processor -- start 2024-05-09T09:35:04.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 -- start start 2024-05-09T09:35:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 0x558414886670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x55841488b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x55841488e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5584147ea340 con 0x558414886250 2024-05-09T09:35:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558414792290 con 0x55841488bdb0 2024-05-09T09:35:04.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.020+0000 7fbf20bca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5584147e6300 con 0x558414886c40 2024-05-09T09:35:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.022+0000 7fbf14ff9640 1 -- 172.21.15.69:0/1741770563 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fbf08018020 con 0x7fbf1810db50 2024-05-09T09:35:04.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.023+0000 7fbf14ff9640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 0x7fbf000a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.025 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.024+0000 7fbf14ff9640 1 -- 172.21.15.69:0/1741770563 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbf080cfcf0 con 0x7fbf1810db50 2024-05-09T09:35:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.026+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x55841488e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.026+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 0x558414886670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.026+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 0x558414886670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:38352/0 (socket says 172.21.15.69:38352) 2024-05-09T09:35:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.026+0000 7fbf16ffd640 1 -- 172.21.15.69:0/685317152 learned_addr learned my addr 172.21.15.69:0/685317152 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:04.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.026+0000 7fbf177fe640 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x55841488b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf16ffd640 1 -- 172.21.15.69:0/685317152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 msgr2=0x55841488b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x55841488b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf16ffd640 1 -- 172.21.15.69:0/685317152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 msgr2=0x558414886670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 0x558414886670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf16ffd640 1 -- 172.21.15.69:0/685317152 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5584147b68e0 con 0x55841488bdb0 2024-05-09T09:35:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf177fe640 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x55841488b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:04.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.028+0000 7fbf17fff640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 0x7fbf000a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.031 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.030+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x55841488e240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5584148874a0 tx=0x7fbf100095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c5dca39f9985460d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.033 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.031+0000 7fbefdffb640 1 -- 172.21.15.69:0/685317152 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10031460 con 0x55841488bdb0 2024-05-09T09:35:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.031+0000 7fbefdffb640 1 -- 172.21.15.69:0/685317152 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf10036050 con 0x55841488bdb0 2024-05-09T09:35:04.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.031+0000 7fbefdffb640 1 -- 172.21.15.69:0/685317152 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf100042a0 con 0x55841488bdb0 2024-05-09T09:35:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.035+0000 7fbf20bca740 1 -- 172.21.15.69:0/685317152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 msgr2=0x55841488e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.035+0000 7fbf20bca740 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x55841488e240 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x5584148874a0 tx=0x7fbf100095c0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf17fff640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 0x7fbf000a52e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a48b0 tx=0x7fbf080109a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf20bca740 1 -- 172.21.15.69:0/685317152 shutdown_connections 2024-05-09T09:35:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf20bca740 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x55841488e240 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf20bca740 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x55841488b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf20bca740 1 --2- 172.21.15.69:0/685317152 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414886250 0x558414886670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.039 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.038+0000 7fbf20bca740 1 -- 172.21.15.69:0/685317152 >> 172.21.15.69:0/685317152 conn(0x5584148818c0 msgr2=0x558414881cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:04.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.040+0000 7fbf20bca740 1 -- 172.21.15.69:0/685317152 shutdown_connections 2024-05-09T09:35:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.040+0000 7fbf20bca740 1 -- 172.21.15.69:0/685317152 wait complete. 2024-05-09T09:35:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.041+0000 7fbf20bca740 1 -- start start 2024-05-09T09:35:04.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.041+0000 7fbf20bca740 1 Processor -- start 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148904c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55841488bdb0 0x558414897a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55841488ed00 con 0x55841488bdb0 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5584147a5790 con 0x558414886250 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf177fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf177fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51460/0 (socket says 172.21.15.69:51460) 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf177fe640 1 -- 172.21.15.69:0/281500793 learned_addr learned my addr 172.21.15.69:0/281500793 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.042+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55841488ee80 con 0x558414886c40 2024-05-09T09:35:04.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.043+0000 7fbf20bca740 10 client.0 _ll_register_callbacks cb 0x5584147e0a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 -- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 msgr2=0x5584148904c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148904c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 -- 172.21.15.69:0/281500793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55841488bdb0 msgr2=0x558414897a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55841488bdb0 0x558414897a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 -- 172.21.15.69:0/281500793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5584144fe210 con 0x558414886c40 2024-05-09T09:35:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbf177fe640 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00e1d0 tx=0x7fbf0c00e690 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e470268cec28c691 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.045+0000 7fbef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:35:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.046+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c017070 con 0x558414886c40 2024-05-09T09:35:04.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.046+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf10036050 con 0x558414886c40 2024-05-09T09:35:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.046+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbf0c013680 con 0x558414886c40 2024-05-09T09:35:04.047 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.046+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbf0c023050 con 0x558414886c40 2024-05-09T09:35:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.048+0000 7fbf20bca740 10 client.26545 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:04.048 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.048+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5584145ca620 con 0x558414886c40 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.050+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf0c00ee60 con 0x558414886c40 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.050+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fbf0c012b20 con 0x558414886c40 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.051+0000 7fbf20bca740 10 client.26545 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.051+0000 7fbf20bca740 10 client.26545 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.051+0000 7fbf20bca740 20 client.26545 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.051+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55841489c040 0x55841489e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.051+0000 7fbf20bca740 4 client.26545 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:35:04.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.052+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fbf10036050 con 0x55841489c040 2024-05-09T09:35:04.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.053+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55841489c040 0x55841489e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.067+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55841489c040 0x55841489e4a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f61223c769913cb6 server_cookie=58721871f37ec846 in_seq=0 out_seq=0 2024-05-09T09:35:04.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.067+0000 7fbef6ffd640 10 client.26545 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbef6ffd640 1 -- 172.21.15.69:0/281500793 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fbf10036050 con 0x55841489c040 2024-05-09T09:35:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbef6ffd640 10 client.26545 handle_command_reply: tid=0 2024-05-09T09:35:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 client.26545 shutdown 2024-05-09T09:35:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 msgr2=0x558414890a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.076 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00e1d0 tx=0x7fbf0c00e690 comp rx=0 tx=0).stop 2024-05-09T09:35:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 shutdown_connections 2024-05-09T09:35:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55841488bdb0 0x558414897a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55841489c040 0x55841489e4a0 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886c40 0x558414890a00 secure :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c00e1d0 tx=0x7fbf0c00e690 comp rx=0 tx=0).stop 2024-05-09T09:35:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.076+0000 7fbf20bca740 1 --2- 172.21.15.69:0/281500793 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148904c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.077+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 >> 172.21.15.69:0/281500793 conn(0x5584148818c0 msgr2=0x558414880a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.077+0000 7fbef6ffd640 0 client.26545 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:04.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.078+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 shutdown_connections 2024-05-09T09:35:04.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.079+0000 7fbf20bca740 1 -- 172.21.15.69:0/281500793 wait complete. 2024-05-09T09:35:04.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.081+0000 7fbf20bca740 20 client.26545 trim_cache size 0 max 16384 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 Processor -- start 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 -- start start 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55841488bdb0 0x5584148a42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414880170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.109 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181c44a0 con 0x558414897a00 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181c1d70 con 0x558414886250 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf20bca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181184c0 con 0x55841488bdb0 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58466/0 (socket says 172.21.15.69:58466) 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf16ffd640 1 -- 172.21.15.69:0/1224179857 learned_addr learned my addr 172.21.15.69:0/1224179857 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414880170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.108+0000 7fbf177fe640 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55841488bdb0 0x5584148a42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.109+0000 7fbf16ffd640 1 -- 172.21.15.69:0/1224179857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55841488bdb0 msgr2=0x5584148a42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.109+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55841488bdb0 0x5584148a42f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.109+0000 7fbf16ffd640 1 -- 172.21.15.69:0/1224179857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 msgr2=0x558414880170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.109+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414880170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.109+0000 7fbf16ffd640 1 -- 172.21.15.69:0/1224179857 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558414887020 con 0x558414886250 2024-05-09T09:35:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.110+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbf1002adf0 tx=0x7fbf10000f40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=40a79160efd5b04f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.111+0000 7fbefd7fa640 1 -- 172.21.15.69:0/1224179857 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf10051b50 con 0x558414886250 2024-05-09T09:35:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.111+0000 7fbefd7fa640 1 -- 172.21.15.69:0/1224179857 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf10036070 con 0x558414886250 2024-05-09T09:35:04.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.111+0000 7fbefd7fa640 1 -- 172.21.15.69:0/1224179857 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf100042b0 con 0x558414886250 2024-05-09T09:35:04.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 -- 172.21.15.69:0/1224179857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 msgr2=0x5584148a3db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fbf1002adf0 tx=0x7fbf10000f40 comp rx=0 tx=0).stop 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 -- 172.21.15.69:0/1224179857 shutdown_connections 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414880170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55841488bdb0 0x5584148a42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 --2- 172.21.15.69:0/1224179857 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558414886250 0x5584148a3db0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 -- 172.21.15.69:0/1224179857 >> 172.21.15.69:0/1224179857 conn(0x5584148818c0 msgr2=0x558414881ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 -- 172.21.15.69:0/1224179857 shutdown_connections 2024-05-09T09:35:04.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.112+0000 7fbf20bca740 1 -- 172.21.15.69:0/1224179857 wait complete. 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.113+0000 7fbf20bca740 1 -- start start 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.113+0000 7fbf20bca740 1 Processor -- start 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.113+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 0x558414895bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.113+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x558414896110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.113+0000 7fbf20bca740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414896650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf20bca740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181c0230 con 0x558414897a00 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf20bca740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181c16a0 con 0x55841488bdb0 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 0x558414895bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf20bca740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fbf181c0900 con 0x558414886250 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf16ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 0x558414895bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51474/0 (socket says 172.21.15.69:51474) 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf16ffd640 1 -- 172.21.15.69:0/671971137 learned_addr learned my addr 172.21.15.69:0/671971137 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x558414896110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414896650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf20bca740 10 client.0 _ll_register_callbacks cb 0x7fbf0c009320 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 -- 172.21.15.69:0/671971137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 msgr2=0x558414895bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 0x558414895bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 -- 172.21.15.69:0/671971137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 msgr2=0x558414896650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414896650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 -- 172.21.15.69:0/671971137 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558414902620 con 0x55841488bdb0 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414896650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbf177fe640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x558414896110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c008000 tx=0x7fbf0c005c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=243241622dfcb4c1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:04.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.114+0000 7fbef6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:35:04.117 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c017040 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fbf10036070 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fbf0c013880 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fbf0c00ee60 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbf20bca740 10 client.26621 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.115+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558414903840 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fbf0c01b570 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fbf0c0123d0 con 0x55841488bdb0 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 10 client.26621 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 10 client.26621 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 20 client.26621 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558414904210 0x558414906630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 4 client.26621 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:35:04.118 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.116+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fbf10036070 con 0x558414904210 2024-05-09T09:35:04.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.117+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558414904210 0x558414906630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.121+0000 7fbf16ffd640 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558414904210 0x558414906630 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35eb65a50b4d16f server_cookie=5ee52736b14fe2d6 in_seq=0 out_seq=0 2024-05-09T09:35:04.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:04.121+0000 7fbef6ffd640 10 client.26621 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:04.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102596 30.61 MB/sec execute 62 sec latency 666.801 ms 2024-05-09T09:35:05.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.208+0000 7fbef6ffd640 1 -- 172.21.15.69:0/671971137 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fbf10036070 con 0x558414904210 2024-05-09T09:35:05.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.208+0000 7fbef6ffd640 10 client.26621 handle_command_reply: tid=0 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.208+0000 7fbf20bca740 1 client.26621 shutdown 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.208+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 msgr2=0x558414896110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.208+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x558414896110 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c008000 tx=0x7fbf0c005c60 comp rx=0 tx=0).stop 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 shutdown_connections 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558414897a00 0x558414896650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55841488bdb0 0x558414896110 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558414904210 0x558414906630 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbef6ffd640 0 client.26621 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 --2- 172.21.15.69:0/671971137 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558414886250 0x558414895bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 >> 172.21.15.69:0/671971137 conn(0x5584148818c0 msgr2=0x558414880170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:05.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 shutdown_connections 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 1 -- 172.21.15.69:0/671971137 wait complete. 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.209+0000 7fbf20bca740 20 client.26621 trim_cache size 0 max 16384 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:35:05.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 msgr2=0x7fbf000a52e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 0x7fbf000a52e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fbf080a48b0 tx=0x7fbf080109a0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 msgr2=0x7fbf181ae000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fbf0800b3f0 tx=0x7fbf0800b8b0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 shutdown_connections 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fbf000a2e20 0x7fbf000a52e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fbf1810db50 0x7fbf181ae000 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fbf1810b1a0 0x7fbf181a6f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 --2- 172.21.15.69:0/1741770563 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fbf18108830 0x7fbf181a6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 >> 172.21.15.69:0/1741770563 conn(0x7fbf1806e8a0 msgr2=0x7fbf18108c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 shutdown_connections 2024-05-09T09:35:05.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:05.210+0000 7fbf1deba640 1 -- 172.21.15.69:0/1741770563 wait complete. 2024-05-09T09:35:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:05 smithi163 ceph-mon[34354]: pgmap v855: 129 pgs: 129 active+clean; 581 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.6 MiB/s wr, 598 op/s 2024-05-09T09:35:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:05 smithi103 ceph-mon[34421]: pgmap v855: 129 pgs: 129 active+clean; 581 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.6 MiB/s wr, 598 op/s 2024-05-09T09:35:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:05 smithi069 ceph-mon[30123]: pgmap v855: 129 pgs: 129 active+clean; 581 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.6 MiB/s wr, 598 op/s 2024-05-09T09:35:05.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 102864 30.37 MB/sec execute 63 sec latency 338.728 ms 2024-05-09T09:35:06.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 103106 30.11 MB/sec execute 64 sec latency 127.965 ms 2024-05-09T09:35:07.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:07 smithi163 ceph-mon[34354]: pgmap v856: 129 pgs: 129 active+clean; 669 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 13 MiB/s wr, 683 op/s 2024-05-09T09:35:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:07 smithi103 ceph-mon[34421]: pgmap v856: 129 pgs: 129 active+clean; 669 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 13 MiB/s wr, 683 op/s 2024-05-09T09:35:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:07 smithi069 ceph-mon[30123]: pgmap v856: 129 pgs: 129 active+clean; 669 MiB data, 6.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 13 MiB/s wr, 683 op/s 2024-05-09T09:35:07.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 103373 29.77 MB/sec execute 65 sec latency 94.058 ms 2024-05-09T09:35:08.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 103744 29.54 MB/sec execute 66 sec latency 96.190 ms 2024-05-09T09:35:09.559 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:09 smithi163 ceph-mon[34354]: pgmap v857: 129 pgs: 129 active+clean; 690 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 702 op/s 2024-05-09T09:35:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:09 smithi103 ceph-mon[34421]: pgmap v857: 129 pgs: 129 active+clean; 690 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 702 op/s 2024-05-09T09:35:09.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:09 smithi069 ceph-mon[30123]: pgmap v857: 129 pgs: 129 active+clean; 690 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 17 MiB/s wr, 702 op/s 2024-05-09T09:35:09.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 104467 29.72 MB/sec execute 67 sec latency 79.418 ms 2024-05-09T09:35:10.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 105276 29.96 MB/sec execute 68 sec latency 59.884 ms 2024-05-09T09:35:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:11 smithi163 ceph-mon[34354]: pgmap v858: 129 pgs: 129 active+clean; 612 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 19 MiB/s wr, 876 op/s 2024-05-09T09:35:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:11 smithi103 ceph-mon[34421]: pgmap v858: 129 pgs: 129 active+clean; 612 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 19 MiB/s wr, 876 op/s 2024-05-09T09:35:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:11 smithi069 ceph-mon[30123]: pgmap v858: 129 pgs: 129 active+clean; 612 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 19 MiB/s wr, 876 op/s 2024-05-09T09:35:11.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 105820 30.00 MB/sec execute 69 sec latency 131.305 ms 2024-05-09T09:35:12.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 106464 30.07 MB/sec execute 70 sec latency 457.971 ms 2024-05-09T09:35:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:13 smithi163 ceph-mon[34354]: pgmap v859: 129 pgs: 129 active+clean; 566 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:35:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:13 smithi103 ceph-mon[34421]: pgmap v859: 129 pgs: 129 active+clean; 566 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:35:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:13 smithi069 ceph-mon[30123]: pgmap v859: 129 pgs: 129 active+clean; 566 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 18 MiB/s wr, 935 op/s 2024-05-09T09:35:13.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 107182 30.17 MB/sec execute 71 sec latency 79.759 ms 2024-05-09T09:35:14.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 107819 30.28 MB/sec execute 72 sec latency 97.423 ms 2024-05-09T09:35:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:15 smithi163 ceph-mon[34354]: pgmap v860: 129 pgs: 129 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 995 op/s 2024-05-09T09:35:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:15 smithi103 ceph-mon[34421]: pgmap v860: 129 pgs: 129 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 995 op/s 2024-05-09T09:35:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:15 smithi069 ceph-mon[30123]: pgmap v860: 129 pgs: 129 active+clean; 564 MiB data, 6.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 18 MiB/s wr, 995 op/s 2024-05-09T09:35:15.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 108460 30.26 MB/sec execute 73 sec latency 59.440 ms 2024-05-09T09:35:16.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 109166 30.35 MB/sec execute 74 sec latency 94.959 ms 2024-05-09T09:35:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:17 smithi163 ceph-mon[34354]: pgmap v861: 129 pgs: 129 active+clean; 563 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 17 MiB/s wr, 964 op/s 2024-05-09T09:35:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:17 smithi103 ceph-mon[34421]: pgmap v861: 129 pgs: 129 active+clean; 563 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 17 MiB/s wr, 964 op/s 2024-05-09T09:35:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:17 smithi069 ceph-mon[30123]: pgmap v861: 129 pgs: 129 active+clean; 563 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 17 MiB/s wr, 964 op/s 2024-05-09T09:35:17.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 109863 30.48 MB/sec execute 75 sec latency 81.442 ms 2024-05-09T09:35:18.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 -- 172.21.15.69:0/594045558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 msgr2=0x7fc85c0fff60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 --2- 172.21.15.69:0/594045558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c0fff60 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7fc84c009930 tx=0x7fc84c02a420 comp rx=0 tx=0).stop 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 -- 172.21.15.69:0/594045558 shutdown_connections 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 --2- 172.21.15.69:0/594045558 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 0x7fc85c109d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 --2- 172.21.15.69:0/594045558 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 0x7fc85c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 --2- 172.21.15.69:0/594045558 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c0fff60 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 -- 172.21.15.69:0/594045558 >> 172.21.15.69:0/594045558 conn(0x7fc85c0fb490 msgr2=0x7fc85c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 -- 172.21.15.69:0/594045558 shutdown_connections 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.325+0000 7fc861942640 1 -- 172.21.15.69:0/594045558 wait complete. 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 Processor -- start 2024-05-09T09:35:18.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 -- start start 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 0x7fc85c100ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 0x7fc85c101000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c10d510 con 0x7fc85c0ffb40 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c10d390 con 0x7fc85c102b10 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc861942640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c10d690 con 0x7fc85c1034d0 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42238/0 (socket says 172.21.15.69:42238) 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 -- 172.21.15.69:0/1588939589 learned_addr learned my addr 172.21.15.69:0/1588939589 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85b7fe640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 0x7fc85c101000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc853fff640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 0x7fc85c100ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 -- 172.21.15.69:0/1588939589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 msgr2=0x7fc85c101000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 0x7fc85c101000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 -- 172.21.15.69:0/1588939589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 msgr2=0x7fc85c100ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 0x7fc85c100ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.326+0000 7fc85affd640 1 -- 172.21.15.69:0/1588939589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc85c101890 con 0x7fc85c0ffb40 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc85affd640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc84c031440 tx=0x7fc84c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc860940640 1 -- 172.21.15.69:0/1588939589 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c03f030 con 0x7fc85c0ffb40 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc860940640 1 -- 172.21.15.69:0/1588939589 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc84c003ad0 con 0x7fc85c0ffb40 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc85c1a1c10 con 0x7fc85c0ffb40 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc860940640 1 -- 172.21.15.69:0/1588939589 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c035070 con 0x7fc85c0ffb40 2024-05-09T09:35:18.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.327+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc85c1a21b0 con 0x7fc85c0ffb40 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 Processor -- start 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc860940640 1 -- 172.21.15.69:0/1588939589 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc84c042030 con 0x7fc85c0ffb40 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 -- start start 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 0x5628efc8d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8dc40 0x5628efc92740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5628efbf1340 con 0x5628efc92db0 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5628efb99290 con 0x5628efc8d250 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.330+0000 7fc86464e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5628efbed300 con 0x5628efc8dc40 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc85b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 0x5628efc8d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc85b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 0x5628efc8d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:34012/0 (socket says 172.21.15.69:34012) 2024-05-09T09:35:18.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42248/0 (socket says 172.21.15.69:42248) 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc85b7fe640 1 -- 172.21.15.69:0/3874408293 learned_addr learned my addr 172.21.15.69:0/3874408293 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 -- 172.21.15.69:0/3874408293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8dc40 msgr2=0x5628efc92740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8dc40 0x5628efc92740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 -- 172.21.15.69:0/3874408293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 msgr2=0x5628efc8d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 0x5628efc8d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 -- 172.21.15.69:0/3874408293 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628efbbd8e0 con 0x5628efc92db0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc853fff640 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc844009bf0 tx=0x7fc844011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a5c56051eb773d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.331+0000 7fc860940640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 0x7fc8480a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.332+0000 7fc851ffb640 1 -- 172.21.15.69:0/3874408293 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844009da0 con 0x5628efc92db0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.332+0000 7fc860940640 1 -- 172.21.15.69:0/1588939589 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc84c0ef890 con 0x7fc85c0ffb40 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.332+0000 7fc851ffb640 1 -- 172.21.15.69:0/3874408293 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844005020 con 0x5628efc92db0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.332+0000 7fc851ffb640 1 -- 172.21.15.69:0/3874408293 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844004bc0 con 0x5628efc92db0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.332+0000 7fc85b7fe640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 0x7fc8480a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.334 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- 172.21.15.69:0/3874408293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 msgr2=0x5628efc95240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7fc844009bf0 tx=0x7fc844011810 comp rx=0 tx=0).stop 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc85b7fe640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 0x7fc8480a5170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x5628efc8e4a0 tx=0x7fc854009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- 172.21.15.69:0/3874408293 shutdown_connections 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc92db0 0x5628efc95240 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8dc40 0x5628efc92740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 --2- 172.21.15.69:0/3874408293 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8d250 0x5628efc8d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- 172.21.15.69:0/3874408293 >> 172.21.15.69:0/3874408293 conn(0x5628efc888c0 msgr2=0x5628efc88cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- 172.21.15.69:0/3874408293 shutdown_connections 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- 172.21.15.69:0/3874408293 wait complete. 2024-05-09T09:35:18.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.333+0000 7fc86464e740 1 -- start start 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.334+0000 7fc86464e740 1 Processor -- start 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.334+0000 7fc86464e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc97560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.334+0000 7fc86464e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.334+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42262/0 (socket says 172.21.15.69:42262) 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc97560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc853fff640 1 -- 172.21.15.69:0/1127326726 learned_addr learned my addr 172.21.15.69:0/1127326726 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5628efc95e00 con 0x5628efc8dc40 2024-05-09T09:35:18.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5628efbac790 con 0x5628efc92db0 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5628efc95f80 con 0x5628efc8d250 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85b7fe640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 10 client.0 _ll_register_callbacks cb 0x5628efbe7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 -- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 msgr2=0x5628efc9ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 -- 172.21.15.69:0/1127326726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 msgr2=0x5628efc97aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628ef905210 con 0x5628efc8d250 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc853fff640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:18.341 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc844005020 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc85affd640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc97560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5628efc8e6b0 tx=0x7fc84c0bba20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1711ee765835eb12 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.335+0000 7fc83affd640 10 client.26560 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc83affd640 1 -- 172.21.15.69:0/1127326726 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c049040 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc83affd640 1 -- 172.21.15.69:0/1127326726 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc84c03b460 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc83affd640 1 -- 172.21.15.69:0/1127326726 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc84c0bbe20 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 10 client.26560 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=47}) -- 0x5628ef9d1620 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc83affd640 1 -- 172.21.15.69:0/1127326726 <== mon.1 v2:172.21.15.163:3300/0 4 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc84c0b8490 con 0x5628efc8d250 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 10 client.26560 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 10 client.26560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 20 client.26560 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efca2de0 0x5628efca51f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 4 client.26560 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:35:18.342 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.336+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5628efbfea70 con 0x5628efca2de0 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.337+0000 7fc853fff640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efca2de0 0x5628efca51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.337+0000 7fc853fff640 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efca2de0 0x5628efca51f0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5edca07a51e67ad0 server_cookie=d80b94786056026e in_seq=0 out_seq=0 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.337+0000 7fc83affd640 10 client.26560 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.338+0000 7fc83affd640 1 -- 172.21.15.69:0/1127326726 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5628efbfea70 con 0x5628efca2de0 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.338+0000 7fc83affd640 10 client.26560 handle_command_reply: tid=0 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.338+0000 7fc86464e740 1 client.26560 shutdown 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 msgr2=0x5628efc97560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc97560 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x5628efc8e6b0 tx=0x7fc84c0bba20 comp rx=0 tx=0).stop 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 shutdown_connections 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc97aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efca2de0 0x5628efca51f0 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 --2- 172.21.15.69:0/1127326726 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc97560 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc83affd640 0 client.26560 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 >> 172.21.15.69:0/1127326726 conn(0x5628efc888c0 msgr2=0x5628efc87a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 shutdown_connections 2024-05-09T09:35:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 1 -- 172.21.15.69:0/1127326726 wait complete. 2024-05-09T09:35:18.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.339+0000 7fc86464e740 20 client.26560 trim_cache size 0 max 16384 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 Processor -- start 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 -- start start 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8dc40 0x5628efc9ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc92db0 0x5628efcaaad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1aee30 con 0x5628efc8d250 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1ac700 con 0x5628efc8dc40 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc86464e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1aa550 con 0x5628efc92db0 2024-05-09T09:35:18.359 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42264/0 (socket says 172.21.15.69:42264) 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 -- 172.21.15.69:0/992671571 learned_addr learned my addr 172.21.15.69:0/992671571 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc853fff640 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8dc40 0x5628efc9ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85b7fe640 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc92db0 0x5628efcaaad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 -- 172.21.15.69:0/992671571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc92db0 msgr2=0x5628efcaaad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc92db0 0x5628efcaaad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 -- 172.21.15.69:0/992671571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8dc40 msgr2=0x5628efc9ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8dc40 0x5628efc9ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.355+0000 7fc85affd640 1 -- 172.21.15.69:0/992671571 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5628efc86f00 con 0x5628efc8d250 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc85affd640 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0bbb20 tx=0x7fc84c0f1000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6aeacb648df4ee77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc8517fa640 1 -- 172.21.15.69:0/992671571 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c049070 con 0x5628efc8d250 2024-05-09T09:35:18.360 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc8517fa640 1 -- 172.21.15.69:0/992671571 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844005020 con 0x5628efc8d250 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc8517fa640 1 -- 172.21.15.69:0/992671571 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc84c0c47b0 con 0x5628efc8d250 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 -- 172.21.15.69:0/992671571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 msgr2=0x5628efc96a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7fc84c0bbb20 tx=0x7fc84c0f1000 comp rx=0 tx=0).stop 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 -- 172.21.15.69:0/992671571 shutdown_connections 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc92db0 0x5628efcaaad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc8dc40 0x5628efc9ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.356+0000 7fc86464e740 1 --2- 172.21.15.69:0/992671571 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8d250 0x5628efc96a60 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- 172.21.15.69:0/992671571 >> 172.21.15.69:0/992671571 conn(0x5628efc888c0 msgr2=0x5628efc96350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- 172.21.15.69:0/992671571 shutdown_connections 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- 172.21.15.69:0/992671571 wait complete. 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- start start 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 Processor -- start 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc9c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.361 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9dba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1aabc0 con 0x5628efc8dc40 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1ac030 con 0x5628efc92db0 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc85c1ab290 con 0x5628efc8d250 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc86464e740 10 client.0 _ll_register_callbacks cb 0x7fc84c009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc85affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc9c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc85b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9dba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.357+0000 7fc853fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42268/0 (socket says 172.21.15.69:42268) 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 -- 172.21.15.69:0/3138071034 learned_addr learned my addr 172.21.15.69:0/3138071034 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 -- 172.21.15.69:0/3138071034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 msgr2=0x5628efc9c6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc9c6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.362 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 -- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 msgr2=0x5628efc9dba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9dba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 -- 172.21.15.69:0/3138071034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5628efc9e190 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc853fff640 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc84401b040 tx=0x7fc8440056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ed2d1c70a4a7de9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc83affd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc844009820 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc84402bb50 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc844005020 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc844038050 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc86464e740 10 client.17355 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5628efd09830 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.358+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc844005020 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc844033030 con 0x5628efc8dc40 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 10 client.17355 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 10 client.17355 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:18.363 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 20 client.17355 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efd0a010 0x5628efd0c430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 4 client.17355 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5628efbf8dc0 con 0x5628efd0a010 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.359+0000 7fc85affd640 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efd0a010 0x5628efd0c430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.360+0000 7fc85affd640 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efd0a010 0x5628efd0c430 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9877b4b25413fb09 server_cookie=428a640a8d339563 in_seq=0 out_seq=0 2024-05-09T09:35:18.364 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.360+0000 7fc83affd640 10 client.17355 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:18.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:18 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:35:18.621 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:18 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:35:18.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:18 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:35:18.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 110352 30.43 MB/sec execute 76 sec latency 361.926 ms 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.946+0000 7fc83affd640 1 -- 172.21.15.69:0/3138071034 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5628efbf8dc0 con 0x5628efd0a010 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.946+0000 7fc83affd640 10 client.17355 handle_command_reply: tid=0 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.946+0000 7fc86464e740 1 client.17355 shutdown 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 msgr2=0x5628efc9cbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7fc84401b040 tx=0x7fc8440056f0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 shutdown_connections 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5628efc92db0 0x5628efc9dba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5628efc8dc40 0x5628efc9cbe0 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5628efd0a010 0x5628efd0c430 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 --2- 172.21.15.69:0/3138071034 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5628efc8d250 0x5628efc9c6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 >> 172.21.15.69:0/3138071034 conn(0x5628efc888c0 msgr2=0x5628efc97100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc83affd640 0 client.17355 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 shutdown_connections 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 1 -- 172.21.15.69:0/3138071034 wait complete. 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.947+0000 7fc86464e740 20 client.17355 trim_cache size 0 max 16384 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:35:18.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 msgr2=0x7fc8480a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 0x7fc8480a5170 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x5628efc8e4a0 tx=0x7fc854009500 comp rx=0 tx=0).stop 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 msgr2=0x7fc85c102480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7fc84c031440 tx=0x7fc84c004270 comp rx=0 tx=0).stop 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 shutdown_connections 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc8480a2cb0 0x7fc8480a5170 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc85c1034d0 0x7fc85c101000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc85c102b10 0x7fc85c100ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 --2- 172.21.15.69:0/1588939589 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc85c0ffb40 0x7fc85c102480 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 >> 172.21.15.69:0/1588939589 conn(0x7fc85c0fb490 msgr2=0x7fc85c105e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 shutdown_connections 2024-05-09T09:35:18.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:18.948+0000 7fc861942640 1 -- 172.21.15.69:0/1588939589 wait complete. 2024-05-09T09:35:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:19 smithi163 ceph-mon[34354]: pgmap v862: 129 pgs: 129 active+clean; 573 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 890 op/s 2024-05-09T09:35:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:19 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:19 smithi103 ceph-mon[34421]: pgmap v862: 129 pgs: 129 active+clean; 573 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 890 op/s 2024-05-09T09:35:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:19 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:19 smithi069 ceph-mon[30123]: pgmap v862: 129 pgs: 129 active+clean; 573 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 10 MiB/s wr, 890 op/s 2024-05-09T09:35:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:19 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:35:19.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 110867 30.49 MB/sec execute 77 sec latency 358.457 ms 2024-05-09T09:35:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:20 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:35:20.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:20 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:35:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:20 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:35:20.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:20 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:35:20.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:20 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi069 to 176.1M 2024-05-09T09:35:20.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:20 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi069 to 184708608: error parsing value: Value '184708608' is below minimum 939524096 2024-05-09T09:35:20.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 111381 30.42 MB/sec execute 78 sec latency 71.681 ms 2024-05-09T09:35:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:21 smithi163 ceph-mon[34354]: pgmap v863: 129 pgs: 129 active+clean; 568 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 784 op/s 2024-05-09T09:35:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:35:21.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:35:21.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:35:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:21 smithi103 ceph-mon[34421]: pgmap v863: 129 pgs: 129 active+clean; 568 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 784 op/s 2024-05-09T09:35:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:35:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:35:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:35:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:21 smithi069 ceph-mon[30123]: pgmap v863: 129 pgs: 129 active+clean; 568 MiB data, 6.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 6.1 MiB/s wr, 784 op/s 2024-05-09T09:35:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:35:21.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:35:21.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:35:21.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 111897 30.35 MB/sec execute 79 sec latency 58.564 ms 2024-05-09T09:35:22.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 112522 30.27 MB/sec execute 80 sec latency 53.514 ms 2024-05-09T09:35:23.561 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:23 smithi069 ceph-mon[30123]: pgmap v864: 129 pgs: 129 active+clean; 568 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2024-05-09T09:35:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:23 smithi103 ceph-mon[34421]: pgmap v864: 129 pgs: 129 active+clean; 568 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2024-05-09T09:35:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:23 smithi163 ceph-mon[34354]: pgmap v864: 129 pgs: 129 active+clean; 568 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.3 MiB/s wr, 571 op/s 2024-05-09T09:35:23.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 113416 30.53 MB/sec execute 81 sec latency 69.559 ms 2024-05-09T09:35:24.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 114126 30.69 MB/sec execute 82 sec latency 62.267 ms 2024-05-09T09:35:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:25 smithi103 ceph-mon[34421]: pgmap v865: 129 pgs: 129 active+clean; 569 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 555 op/s 2024-05-09T09:35:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:25 smithi069 ceph-mon[30123]: pgmap v865: 129 pgs: 129 active+clean; 569 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 555 op/s 2024-05-09T09:35:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:25 smithi163 ceph-mon[34354]: pgmap v865: 129 pgs: 129 active+clean; 569 MiB data, 6.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.6 MiB/s wr, 555 op/s 2024-05-09T09:35:25.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 114654 30.78 MB/sec execute 83 sec latency 70.137 ms 2024-05-09T09:35:26.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 115211 30.65 MB/sec execute 84 sec latency 80.833 ms 2024-05-09T09:35:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:27 smithi103 ceph-mon[34421]: pgmap v866: 129 pgs: 129 active+clean; 573 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.7 MiB/s wr, 552 op/s 2024-05-09T09:35:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:27 smithi069 ceph-mon[30123]: pgmap v866: 129 pgs: 129 active+clean; 573 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.7 MiB/s wr, 552 op/s 2024-05-09T09:35:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:27 smithi163 ceph-mon[34354]: pgmap v866: 129 pgs: 129 active+clean; 573 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.5 MiB/s rd, 3.7 MiB/s wr, 552 op/s 2024-05-09T09:35:27.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 115814 30.66 MB/sec execute 85 sec latency 65.766 ms 2024-05-09T09:35:28.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 116562 30.75 MB/sec execute 86 sec latency 61.924 ms 2024-05-09T09:35:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:29 smithi103 ceph-mon[34421]: pgmap v867: 129 pgs: 129 active+clean; 580 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 597 op/s 2024-05-09T09:35:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:29 smithi069 ceph-mon[30123]: pgmap v867: 129 pgs: 129 active+clean; 580 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 597 op/s 2024-05-09T09:35:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:29 smithi163 ceph-mon[34354]: pgmap v867: 129 pgs: 129 active+clean; 580 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.9 MiB/s wr, 597 op/s 2024-05-09T09:35:29.826 INFO:tasks.workunit.client.0.smithi069.stdout: 10 117367 30.91 MB/sec execute 87 sec latency 60.411 ms 2024-05-09T09:35:30.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 118030 31.05 MB/sec execute 88 sec latency 55.876 ms 2024-05-09T09:35:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:31 smithi103 ceph-mon[34421]: pgmap v868: 129 pgs: 129 active+clean; 585 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.4 MiB/s wr, 510 op/s 2024-05-09T09:35:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:31 smithi069 ceph-mon[30123]: pgmap v868: 129 pgs: 129 active+clean; 585 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.4 MiB/s wr, 510 op/s 2024-05-09T09:35:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:31 smithi163 ceph-mon[34354]: pgmap v868: 129 pgs: 129 active+clean; 585 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 3.4 MiB/s wr, 510 op/s 2024-05-09T09:35:31.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 118575 30.99 MB/sec execute 89 sec latency 181.666 ms 2024-05-09T09:35:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 -- 172.21.15.69:0/1318872048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 msgr2=0x7ff6e4109d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/1318872048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e4109d20 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc009900 tx=0x7ff6dc02a410 comp rx=0 tx=0).stop 2024-05-09T09:35:32.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 -- 172.21.15.69:0/1318872048 shutdown_connections 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/1318872048 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e4109d20 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/1318872048 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e4102ad0 0x7ff6e4102f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/1318872048 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6e41018e0 0x7ff6e4101d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 -- 172.21.15.69:0/1318872048 >> 172.21.15.69:0/1318872048 conn(0x7ff6e40fce50 msgr2=0x7ff6e40ff290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 -- 172.21.15.69:0/1318872048 shutdown_connections 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 -- 172.21.15.69:0/1318872048 wait complete. 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.046+0000 7ff6ea07d640 1 Processor -- start 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 -- start start 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e41018e0 0x7ff6e4197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6e4102ad0 0x7ff6e4198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e410d560 con 0x7ff6e4103490 2024-05-09T09:35:32.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e410d3e0 con 0x7ff6e41018e0 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6ea07d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e410d6e0 con 0x7ff6e4102ad0 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49812/0 (socket says 172.21.15.69:49812) 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 -- 172.21.15.69:0/492700182 learned_addr learned my addr 172.21.15.69:0/492700182 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e41018e0 0x7ff6e4197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 -- 172.21.15.69:0/492700182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6e4102ad0 msgr2=0x7ff6e4198000 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6e4102ad0 0x7ff6e4198000 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 -- 172.21.15.69:0/492700182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e41018e0 msgr2=0x7ff6e4197ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e41018e0 0x7ff6e4197ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.047+0000 7ff6e3fff640 1 -- 172.21.15.69:0/492700182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff6e419f6e0 con 0x7ff6e4103490 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6e3fff640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0314e0 tx=0x7ff6dc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6e0ff9640 1 -- 172.21.15.69:0/492700182 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc03f030 con 0x7ff6e4103490 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6e0ff9640 1 -- 172.21.15.69:0/492700182 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6dc003ce0 con 0x7ff6e4103490 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff6e419f970 con 0x7ff6e4103490 2024-05-09T09:35:32.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6e0ff9640 1 -- 172.21.15.69:0/492700182 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6dc035070 con 0x7ff6e4103490 2024-05-09T09:35:32.052 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.048+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff6e419feb0 con 0x7ff6e4103490 2024-05-09T09:35:32.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e0ff9640 1 -- 172.21.15.69:0/492700182 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff6dc042030 con 0x7ff6e4103490 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 Processor -- start 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 -- start start 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7c40 0x557d869fc740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d869ff240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557d8695b340 con 0x557d869f7250 2024-05-09T09:35:32.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557d86903290 con 0x557d869f7c40 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557d86957300 con 0x557d869fcdb0 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49816/0 (socket says 172.21.15.69:49816) 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e2ffd640 1 -- 172.21.15.69:0/957650328 learned_addr learned my addr 172.21.15.69:0/957650328 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7c40 0x557d869fc740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.051+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d869ff240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 -- 172.21.15.69:0/957650328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 msgr2=0x557d869ff240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d869ff240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 -- 172.21.15.69:0/957650328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7c40 msgr2=0x557d869fc740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7c40 0x557d869fc740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e0ff9640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 0x7ff6d00a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 -- 172.21.15.69:0/957650328 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d869278e0 con 0x557d869f7250 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc012740 tx=0x7ff6cc009760 comp rx=0 tx=0).ready entity=mon.0 client_cookie=664260610683302d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e0ff9640 1 -- 172.21.15.69:0/492700182 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff6dc0ef4b0 con 0x7ff6e4103490 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6e3fff640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 0x7ff6d00a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6d9ffb640 1 -- 172.21.15.69:0/957650328 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc004020 con 0x557d869f7250 2024-05-09T09:35:32.065 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6d9ffb640 1 -- 172.21.15.69:0/957650328 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc010c50 con 0x557d869f7250 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.052+0000 7ff6d9ffb640 1 -- 172.21.15.69:0/957650328 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6cc003910 con 0x557d869f7250 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/957650328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 msgr2=0x557d869f7670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6e3fff640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 0x7ff6d00a5170 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b7df0 tx=0x7ff6dc02a410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7ff6cc012740 tx=0x7ff6cc009760 comp rx=0 tx=0).stop 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/957650328 shutdown_connections 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d869ff240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.066 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7c40 0x557d869fc740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/957650328 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7250 0x557d869f7670 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/957650328 >> 172.21.15.69:0/957650328 conn(0x557d869f28c0 msgr2=0x557d869f2cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/957650328 shutdown_connections 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/957650328 wait complete. 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.053+0000 7ff6ecd8d740 1 -- start start 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 Processor -- start 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a01460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 0x557d86a08970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.067 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x557d869ffd00 con 0x557d869fcdb0 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x557d86916790 con 0x557d869f7250 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x557d869ffe80 con 0x557d869f7c40 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6ecd8d740 10 client.0 _ll_register_callbacks cb 0x557d86951a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 0x557d86a08970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:43238/0 (socket says 172.21.15.69:43238) 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 0x557d86a08970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49818/0 (socket says 172.21.15.69:49818) 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2421091546 learned_addr learned my addr 172.21.15.69:0/2421091546 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a01460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 msgr2=0x557d86a01460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a01460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2421091546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 msgr2=0x557d86a08970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 0x557d86a08970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.054+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2421091546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d8666f210 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4011ce0 tx=0x7ff6d400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a5cd6ada8b1169ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4003970 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff6d4003b10 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff6cc010c50 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff6d4003d10 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 10 client.26575 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557d8673b620 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6cc010c50 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7ff6d401b680 con 0x557d869f7c40 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 10 client.26575 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 10 client.26575 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 20 client.26575 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:32.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a0cfe0 0x557d86a0f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.055+0000 7ff6ecd8d740 4 client.26575 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.056+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x557d86968a70 con 0x557d86a0cfe0 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.056+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a0cfe0 0x557d86a0f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.056+0000 7ff6c2ffd640 10 client.26575 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.056+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a0cfe0 0x557d86a0f420 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dbe8c5442e3e855d server_cookie=c9bdfb518888b97c in_seq=0 out_seq=0 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.061+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/2421091546 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x557d86968a70 con 0x557d86a0cfe0 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.061+0000 7ff6c2ffd640 10 client.26575 handle_command_reply: tid=0 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.061+0000 7ff6ecd8d740 1 client.26575 shutdown 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 msgr2=0x557d86a019a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4011ce0 tx=0x7ff6d400f790 comp rx=0 tx=0).stop 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 shutdown_connections 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869fcdb0 0x557d86a08970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a0cfe0 0x557d86a0f420 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869f7c40 0x557d86a019a0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2421091546 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a01460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6c2ffd640 0 client.26575 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 >> 172.21.15.69:0/2421091546 conn(0x557d869f28c0 msgr2=0x557d869f1a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 shutdown_connections 2024-05-09T09:35:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2421091546 wait complete. 2024-05-09T09:35:32.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.062+0000 7ff6ecd8d740 20 client.26575 trim_cache size 0 max 16384 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 Processor -- start 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 -- start start 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d869f8e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a08930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41acc50 con 0x557d869f7c40 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41aa520 con 0x557d869f7250 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a8370 con 0x557d869fcdb0 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d869f8e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49830/0 (socket says 172.21.15.69:49830) 2024-05-09T09:35:32.078 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2507810942 learned_addr learned my addr 172.21.15.69:0/2507810942 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.077+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a08930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2507810942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 msgr2=0x557d86a08930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a08930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2507810942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 msgr2=0x557d869f8e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d869f8e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 -- 172.21.15.69:0/2507810942 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x557d869f3630 con 0x557d869f7c40 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006010 tx=0x7ff6d4011900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3290d7debc87773 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6d97fa640 1 -- 172.21.15.69:0/2507810942 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d402d030 con 0x557d869f7c40 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6d97fa640 1 -- 172.21.15.69:0/2507810942 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff6d400fe80 con 0x557d869f7c40 2024-05-09T09:35:32.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.078+0000 7ff6d97fa640 1 -- 172.21.15.69:0/2507810942 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4005230 con 0x557d869f7c40 2024-05-09T09:35:32.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2507810942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 msgr2=0x557d869f9360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7ff6d4006010 tx=0x7ff6d4011900 comp rx=0 tx=0).stop 2024-05-09T09:35:32.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2507810942 shutdown_connections 2024-05-09T09:35:32.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a08930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d869f9360 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/2507810942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d869f8e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2507810942 >> 172.21.15.69:0/2507810942 conn(0x557d869f28c0 msgr2=0x557d869f2ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2507810942 shutdown_connections 2024-05-09T09:35:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/2507810942 wait complete. 2024-05-09T09:35:32.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.079+0000 7ff6ecd8d740 1 -- start start 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 Processor -- start 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a061e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a076e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a89e0 con 0x557d869f7c40 2024-05-09T09:35:32.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a9e50 con 0x557d869f7250 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff6e41a90b0 con 0x557d869fcdb0 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6ecd8d740 10 client.0 _ll_register_callbacks cb 0x7ff6d4009cc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a061e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49838/0 (socket says 172.21.15.69:49838) 2024-05-09T09:35:32.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 -- 172.21.15.69:0/3434276656 learned_addr learned my addr 172.21.15.69:0/3434276656 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e2ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a061e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:44318/0 (socket says 172.21.15.69:44318) 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a076e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 -- 172.21.15.69:0/3434276656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 msgr2=0x557d86a076e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a076e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 -- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 msgr2=0x557d86a061e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:32.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a061e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.080+0000 7ff6e37fe640 1 -- 172.21.15.69:0/3434276656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x557d86a07cd0 con 0x557d869f7c40 2024-05-09T09:35:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6e37fe640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400f7d0 tx=0x7ff6d4003ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=af5e1d62da40e8b1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:35:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff6d4035030 con 0x557d869f7c40 2024-05-09T09:35:32.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x557d86962dc0 con 0x557d869f7c40 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff6d4017070 con 0x557d869f7c40 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff6d4041050 con 0x557d869f7c40 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 10 client.17373 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x557d86a736c0 con 0x557d869f7c40 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x557d86962dc0 con 0x557d869f7c40 2024-05-09T09:35:32.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7ff6d4029510 con 0x557d869f7c40 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 10 client.17373 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 10 client.17373 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 20 client.17373 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a73f60 0x557d86a76380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 4 client.17373 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x557d868e08e0 con 0x557d86a73f60 2024-05-09T09:35:32.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.081+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a73f60 0x557d86a76380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.082+0000 7ff6c2ffd640 10 client.17373 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:32.092 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.082+0000 7ff6e2ffd640 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a73f60 0x557d86a76380 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc9f0804e716b5b5 server_cookie=5a2da4dc6c8fe056 in_seq=0 out_seq=0 2024-05-09T09:35:32.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 118809 30.80 MB/sec execute 90 sec latency 648.947 ms 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.872+0000 7ff6c2ffd640 1 -- 172.21.15.69:0/3434276656 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x557d868e08e0 con 0x557d86a73f60 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.872+0000 7ff6c2ffd640 10 client.17373 handle_command_reply: tid=0 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.872+0000 7ff6ecd8d740 1 client.17373 shutdown 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.872+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 msgr2=0x557d86a06720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.872+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7ff6d400f7d0 tx=0x7ff6d4003ba0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 shutdown_connections 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x557d86a73f60 0x557d86a76380 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.875 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6c2ffd640 0 client.17373 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x557d869fcdb0 0x557d86a076e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x557d869f7c40 0x557d86a06720 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 --2- 172.21.15.69:0/3434276656 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x557d869f7250 0x557d86a061e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 >> 172.21.15.69:0/3434276656 conn(0x557d869f28c0 msgr2=0x557d869f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 shutdown_connections 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 1 -- 172.21.15.69:0/3434276656 wait complete. 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.873+0000 7ff6ecd8d740 20 client.17373 trim_cache size 0 max 16384 2024-05-09T09:35:32.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:35:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:35:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:35:32.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 msgr2=0x7ff6d00a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 0x7ff6d00a5170 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0b7df0 tx=0x7ff6dc02a410 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 msgr2=0x7ff6e419f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7ff6dc0314e0 tx=0x7ff6dc0042f0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 shutdown_connections 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff6d00a2cb0 0x7ff6d00a5170 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff6e4103490 0x7ff6e419f070 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff6e4102ad0 0x7ff6e4198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 --2- 172.21.15.69:0/492700182 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff6e41018e0 0x7ff6e4197ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:32.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.874+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 >> 172.21.15.69:0/492700182 conn(0x7ff6e40fce50 msgr2=0x7ff6e4105d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.875+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 shutdown_connections 2024-05-09T09:35:32.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:32.875+0000 7ff6ea07d640 1 -- 172.21.15.69:0/492700182 wait complete. 2024-05-09T09:35:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:33 smithi103 ceph-mon[34421]: pgmap v869: 129 pgs: 129 active+clean; 595 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 566 op/s 2024-05-09T09:35:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:33 smithi069 ceph-mon[30123]: pgmap v869: 129 pgs: 129 active+clean; 595 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 566 op/s 2024-05-09T09:35:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:33 smithi163 ceph-mon[34354]: pgmap v869: 129 pgs: 129 active+clean; 595 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.0 MiB/s wr, 566 op/s 2024-05-09T09:35:33.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 119267 30.71 MB/sec execute 91 sec latency 649.452 ms 2024-05-09T09:35:34.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 119950 30.79 MB/sec execute 92 sec latency 46.682 ms 2024-05-09T09:35:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:35 smithi103 ceph-mon[34421]: pgmap v870: 129 pgs: 129 active+clean; 598 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 519 op/s 2024-05-09T09:35:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:35 smithi069 ceph-mon[30123]: pgmap v870: 129 pgs: 129 active+clean; 598 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 519 op/s 2024-05-09T09:35:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:35 smithi163 ceph-mon[34354]: pgmap v870: 129 pgs: 129 active+clean; 598 MiB data, 6.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.9 MiB/s wr, 519 op/s 2024-05-09T09:35:35.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 120574 30.80 MB/sec execute 93 sec latency 72.828 ms 2024-05-09T09:35:36.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 121270 30.88 MB/sec execute 94 sec latency 109.330 ms 2024-05-09T09:35:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:37 smithi103 ceph-mon[34421]: pgmap v871: 129 pgs: 129 active+clean; 599 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 607 op/s 2024-05-09T09:35:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:37 smithi069 ceph-mon[30123]: pgmap v871: 129 pgs: 129 active+clean; 599 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 607 op/s 2024-05-09T09:35:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:37 smithi163 ceph-mon[34354]: pgmap v871: 129 pgs: 129 active+clean; 599 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.4 MiB/s rd, 4.3 MiB/s wr, 607 op/s 2024-05-09T09:35:37.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 121805 30.86 MB/sec execute 95 sec latency 331.340 ms 2024-05-09T09:35:38.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 121816 30.54 MB/sec execute 96 sec latency 1331.387 ms 2024-05-09T09:35:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:39 smithi103 ceph-mon[34421]: pgmap v872: 129 pgs: 129 active+clean; 594 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 599 op/s 2024-05-09T09:35:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:39 smithi069 ceph-mon[30123]: pgmap v872: 129 pgs: 129 active+clean; 594 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 599 op/s 2024-05-09T09:35:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:39 smithi163 ceph-mon[34354]: pgmap v872: 129 pgs: 129 active+clean; 594 MiB data, 6.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 4.0 MiB/s wr, 599 op/s 2024-05-09T09:35:39.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 121846 30.25 MB/sec execute 97 sec latency 2331.430 ms 2024-05-09T09:35:40.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 122053 30.05 MB/sec execute 98 sec latency 2471.107 ms 2024-05-09T09:35:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:41 smithi103 ceph-mon[34421]: pgmap v873: 129 pgs: 129 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.1 MiB/s wr, 691 op/s 2024-05-09T09:35:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:41 smithi069 ceph-mon[30123]: pgmap v873: 129 pgs: 129 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.1 MiB/s wr, 691 op/s 2024-05-09T09:35:41.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:41 smithi163 ceph-mon[34354]: pgmap v873: 129 pgs: 129 active+clean; 638 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 8.1 MiB/s wr, 691 op/s 2024-05-09T09:35:41.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 122286 29.88 MB/sec execute 99 sec latency 294.555 ms 2024-05-09T09:35:42.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 122548 29.71 MB/sec execute 100 sec latency 124.354 ms 2024-05-09T09:35:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:43 smithi103 ceph-mon[34421]: pgmap v874: 129 pgs: 129 active+clean; 677 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 749 op/s 2024-05-09T09:35:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:43 smithi069 ceph-mon[30123]: pgmap v874: 129 pgs: 129 active+clean; 677 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 749 op/s 2024-05-09T09:35:43.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:43 smithi163 ceph-mon[34354]: pgmap v874: 129 pgs: 129 active+clean; 677 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 12 MiB/s wr, 749 op/s 2024-05-09T09:35:43.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 122858 29.63 MB/sec execute 101 sec latency 132.103 ms 2024-05-09T09:35:44.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 123348 29.56 MB/sec execute 102 sec latency 184.913 ms 2024-05-09T09:35:45.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:45 smithi163 ceph-mon[34354]: pgmap v875: 129 pgs: 129 active+clean; 679 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 685 op/s 2024-05-09T09:35:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:45 smithi103 ceph-mon[34421]: pgmap v875: 129 pgs: 129 active+clean; 679 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 685 op/s 2024-05-09T09:35:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:45 smithi069 ceph-mon[30123]: pgmap v875: 129 pgs: 129 active+clean; 679 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 13 MiB/s wr, 685 op/s 2024-05-09T09:35:45.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 123957 29.59 MB/sec execute 103 sec latency 78.954 ms 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 -- 172.21.15.69:0/2060742590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 msgr2=0x7fedc8102f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2060742590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc8102f20 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7fedb4009900 tx=0x7fedb402a410 comp rx=0 tx=0).stop 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 -- 172.21.15.69:0/2060742590 shutdown_connections 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2060742590 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 0x7fedc8109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2060742590 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc8102f20 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:45.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2060742590 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 -- 172.21.15.69:0/2060742590 >> 172.21.15.69:0/2060742590 conn(0x7fedc80fce20 msgr2=0x7fedc80ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 -- 172.21.15.69:0/2060742590 shutdown_connections 2024-05-09T09:35:45.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.989+0000 7fedcf03f640 1 -- 172.21.15.69:0/2060742590 wait complete. 2024-05-09T09:35:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.990+0000 7fedcf03f640 1 Processor -- start 2024-05-09T09:35:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.990+0000 7fedcf03f640 1 -- start start 2024-05-09T09:35:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.990+0000 7fedcf03f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:45.993 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.990+0000 7fedcf03f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc819a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedcf03f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 0x7fedc81a1320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedcf03f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedc8074070 con 0x7fedc8102aa0 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedcf03f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fedc8073ef0 con 0x7fedc8103460 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedcf03f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fedc80741f0 con 0x7fedc81018b0 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51282/0 (socket says 172.21.15.69:51282) 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 -- 172.21.15.69:0/2934147106 learned_addr learned my addr 172.21.15.69:0/2934147106 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc819a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 0x7fedc81a1320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 -- 172.21.15.69:0/2934147106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 msgr2=0x7fedc81a1320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:45.994 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 0x7fedc81a1320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 -- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 msgr2=0x7fedc819a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc819a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 -- 172.21.15.69:0/2934147106 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fedc81a19c0 con 0x7fedc81018b0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.991+0000 7fedccddb640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00e9c0 tx=0x7fedbc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.992+0000 7fedc5ffb640 1 -- 172.21.15.69:0/2934147106 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc00cd80 con 0x7fedc81018b0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.992+0000 7fedc5ffb640 1 -- 172.21.15.69:0/2934147106 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedbc0035f0 con 0x7fedc81018b0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.992+0000 7fedc5ffb640 1 -- 172.21.15.69:0/2934147106 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc010690 con 0x7fedc81018b0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.992+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fedc81a1cb0 con 0x7fedc81018b0 2024-05-09T09:35:45.995 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.993+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fedc81a21b0 con 0x7fedc81018b0 2024-05-09T09:35:45.996 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.996+0000 7fedc5ffb640 1 -- 172.21.15.69:0/2934147106 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fedbc0037a0 con 0x7fedc81018b0 2024-05-09T09:35:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.997+0000 7fedc5ffb640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 0x7fedb80a5300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.997+0000 7fedc5ffb640 1 -- 172.21.15.69:0/2934147106 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fedbc010430 con 0x7fedc81018b0 2024-05-09T09:35:45.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.997+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 0x7fedb80a5300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.998+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 0x7fedb80a5300 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fedb40314e0 tx=0x7fedb4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 Processor -- start 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 -- start start 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026e690 0x55d23026eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026f080 0x55d230273b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d2301d7340 con 0x55d23026f080 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d23017f290 con 0x55d2302741f0 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d2301d3300 con 0x55d23026e690 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:41684/0 (socket says 172.21.15.69:41684) 2024-05-09T09:35:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:45.999+0000 7fedcd5dc640 1 -- 172.21.15.69:0/168956736 learned_addr learned my addr 172.21.15.69:0/168956736 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 -- 172.21.15.69:0/168956736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026e690 msgr2=0x55d23026eab0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:35:46.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026e690 0x55d23026eab0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 -- 172.21.15.69:0/168956736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026f080 msgr2=0x55d230273b80 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:35:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026f080 0x55d230273b80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 -- 172.21.15.69:0/168956736 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d2301a38e0 con 0x55d2302741f0 2024-05-09T09:35:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.000+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009fd0 tx=0x7fedc00338e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64e4eb8f4eb7f412 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.001+0000 7fedb2ffd640 1 -- 172.21.15.69:0/168956736 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0004d20 con 0x55d2302741f0 2024-05-09T09:35:46.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.001+0000 7fedb2ffd640 1 -- 172.21.15.69:0/168956736 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc0004050 con 0x55d2302741f0 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedb2ffd640 1 -- 172.21.15.69:0/168956736 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc0005280 con 0x55d2302741f0 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 -- 172.21.15.69:0/168956736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 msgr2=0x55d23028c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fedc0009fd0 tx=0x7fedc00338e0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 -- 172.21.15.69:0/168956736 shutdown_connections 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23028c6f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026f080 0x55d230273b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/168956736 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026e690 0x55d23026eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.002+0000 7fedd1e2d740 1 -- 172.21.15.69:0/168956736 >> 172.21.15.69:0/168956736 conn(0x55d2302770d0 msgr2=0x55d2302774d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.003+0000 7fedd1e2d740 1 -- 172.21.15.69:0/168956736 shutdown_connections 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.003+0000 7fedd1e2d740 1 -- 172.21.15.69:0/168956736 wait complete. 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.003+0000 7fedd1e2d740 1 -- start start 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.003+0000 7fedd1e2d740 1 Processor -- start 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027d560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d23027daa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d23027b8e0 con 0x55d23026e690 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d230192790 con 0x55d23026f080 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d23027ba60 con 0x55d2302741f0 2024-05-09T09:35:46.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedd1e2d740 10 client.0 _ll_register_callbacks cb 0x55d2301cda20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51300/0 (socket says 172.21.15.69:51300) 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.004+0000 7fedcd5dc640 1 -- 172.21.15.69:0/2452111231 learned_addr learned my addr 172.21.15.69:0/2452111231 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedccddb640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027d560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 -- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 msgr2=0x55d23027daa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d23027daa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 -- 172.21.15.69:0/2452111231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 msgr2=0x55d23027d560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027d560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 -- 172.21.15.69:0/2452111231 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d22feeb210 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fedc003d040 tx=0x7fedc0003c60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48f26b6023067b2c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.005+0000 7feda7fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedc004a070 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fedc0004410 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedc0004050 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fedc0056050 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 10 client.26590 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d22ffb7620 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedc0004050 con 0x55d2302741f0 2024-05-09T09:35:46.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fedc00464c0 con 0x55d2302741f0 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 10 client.26590 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 10 client.26590 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 20 client.26590 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d230288d60 0x55d23028b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 4 client.26590 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.006+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d2301e4a70 con 0x55d230288d60 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.007+0000 7fedccddb640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d230288d60 0x55d23028b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.007+0000 7feda7fff640 10 client.26590 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:46.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.007+0000 7fedccddb640 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d230288d60 0x55d23028b150 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10226c584764ed49 server_cookie=2aa761487b2ae782 in_seq=0 out_seq=0 2024-05-09T09:35:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.011+0000 7feda7fff640 1 -- 172.21.15.69:0/2452111231 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d2301e4a70 con 0x55d230288d60 2024-05-09T09:35:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7feda7fff640 10 client.26590 handle_command_reply: tid=0 2024-05-09T09:35:46.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 client.26590 shutdown 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 msgr2=0x55d230284a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fedc003d040 tx=0x7fedc0003c60 comp rx=0 tx=0).stop 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 shutdown_connections 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d230288d60 0x55d23028b150 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d230284a70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d23027daa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2452111231 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027d560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7feda7fff640 0 client.26590 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 >> 172.21.15.69:0/2452111231 conn(0x55d2302770d0 msgr2=0x55d23026da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 shutdown_connections 2024-05-09T09:35:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2452111231 wait complete. 2024-05-09T09:35:46.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.012+0000 7fedd1e2d740 20 client.26590 trim_cache size 0 max 16384 2024-05-09T09:35:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 Processor -- start 2024-05-09T09:35:46.035 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 -- start start 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026f080 0x55d230284a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23027c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81aac80 con 0x55d23026e690 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81a8550 con 0x55d2302741f0 2024-05-09T09:35:46.036 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.035+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81a63a0 con 0x55d23026f080 2024-05-09T09:35:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23027c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.037 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60462/0 (socket says 172.21.15.69:60462) 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedc7fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026f080 0x55d230284a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 -- 172.21.15.69:0/3153745729 learned_addr learned my addr 172.21.15.69:0/3153745729 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 -- 172.21.15.69:0/3153745729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026f080 msgr2=0x55d230284a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026f080 0x55d230284a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 -- 172.21.15.69:0/3153745729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 msgr2=0x55d23027c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23027c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedccddb640 1 -- 172.21.15.69:0/3153745729 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d23026f460 con 0x55d23026e690 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.036+0000 7fedcd5dc640 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23027c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.037+0000 7fedccddb640 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c6a0 tx=0x7fedbc0cc4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d487a0eb56907111 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.037+0000 7fedb27fc640 1 -- 172.21.15.69:0/3153745729 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc095e60 con 0x55d23026e690 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.037+0000 7fedb27fc640 1 -- 172.21.15.69:0/3153745729 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedbc09b380 con 0x55d23026e690 2024-05-09T09:35:46.038 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.037+0000 7fedb27fc640 1 -- 172.21.15.69:0/3153745729 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc0d3060 con 0x55d23026e690 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- 172.21.15.69:0/3153745729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 msgr2=0x55d23027be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00c6a0 tx=0x7fedbc0cc4f0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- 172.21.15.69:0/3153745729 shutdown_connections 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d2302741f0 0x55d23027c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d23026f080 0x55d230284a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/3153745729 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d23027be40 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- 172.21.15.69:0/3153745729 >> 172.21.15.69:0/3153745729 conn(0x55d2302770d0 msgr2=0x55d23027d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:46.040 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- 172.21.15.69:0/3153745729 shutdown_connections 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- 172.21.15.69:0/3153745729 wait complete. 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 -- start start 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.038+0000 7fedd1e2d740 1 Processor -- start 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d2302817c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d230281d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d2302ed230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81a6a10 con 0x55d23026e690 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81a7e80 con 0x55d23026f080 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fedc81a70e0 con 0x55d2302741f0 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedd1e2d740 10 client.0 _ll_register_callbacks cb 0x7fedc0009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedccddb640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d2302817c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d2302ed230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedcd5dc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d2302ed230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51314/0 (socket says 172.21.15.69:51314) 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedcd5dc640 1 -- 172.21.15.69:0/2991482713 learned_addr learned my addr 172.21.15.69:0/2991482713 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:46.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.039+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d230281d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 -- 172.21.15.69:0/2991482713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 msgr2=0x55d2302ed230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d2302ed230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 -- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 msgr2=0x55d230281d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d230281d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.043 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 -- 172.21.15.69:0/2991482713 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d2302823a0 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedccddb640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d2302817c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fedbc003ec0 tx=0x7fedbc003d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5a40e7974d72321 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7feda7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fedbc09f730 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fedbc097960 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fedbc09b380 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.040+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fedbc09a7d0 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 10 client.17388 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d2302ee900 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fedbc09b380 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fedbc09a240 con 0x55d23026e690 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 10 client.17388 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 10 client.17388 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 20 client.17388 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:46.044 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2302ef2f0 0x55d2302f1710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 4 client.17388 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:35:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d2301dedc0 con 0x55d2302ef2f0 2024-05-09T09:35:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.041+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2302ef2f0 0x55d2302f1710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.042+0000 7fedc7fff640 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2302ef2f0 0x55d2302f1710 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=92498c7bec287171 server_cookie=1904172a5309cd19 in_seq=0 out_seq=0 2024-05-09T09:35:46.045 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.042+0000 7feda7fff640 10 client.17388 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.323+0000 7feda7fff640 1 -- 172.21.15.69:0/2991482713 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d2301dedc0 con 0x55d2302ef2f0 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.323+0000 7feda7fff640 10 client.17388 handle_command_reply: tid=0 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.323+0000 7fedd1e2d740 1 client.17388 shutdown 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 msgr2=0x55d2302817c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d2302817c0 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7fedbc003ec0 tx=0x7fedbc003d70 comp rx=0 tx=0).stop 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 shutdown_connections 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d2302ef2f0 0x55d2302f1710 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d2302741f0 0x55d2302ed230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d23026f080 0x55d230281d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 --2- 172.21.15.69:0/2991482713 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d23026e690 0x55d2302817c0 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 >> 172.21.15.69:0/2991482713 conn(0x55d2302770d0 msgr2=0x55d23027be40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:46.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7feda7fff640 0 client.17388 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 shutdown_connections 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 1 -- 172.21.15.69:0/2991482713 wait complete. 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.324+0000 7fedd1e2d740 20 client.17388 trim_cache size 0 max 16384 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:35:46.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:35:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 msgr2=0x7fedb80a5300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 0x7fedb80a5300 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fedb40314e0 tx=0x7fedb4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 msgr2=0x7fedc8199db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:46.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7fedbc00e9c0 tx=0x7fedbc00ee80 comp rx=0 tx=0).stop 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 shutdown_connections 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fedb80a2e40 0x7fedb80a5300 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fedc8103460 0x7fedc81a1320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fedc8102aa0 0x7fedc819a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 --2- 172.21.15.69:0/2934147106 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fedc81018b0 0x7fedc8199db0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 >> 172.21.15.69:0/2934147106 conn(0x7fedc80fce20 msgr2=0x7fedc80fe750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 shutdown_connections 2024-05-09T09:35:46.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:46.325+0000 7fedcf03f640 1 -- 172.21.15.69:0/2934147106 wait complete. 2024-05-09T09:35:46.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 124426 29.55 MB/sec execute 104 sec latency 163.187 ms 2024-05-09T09:35:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:47 smithi103 ceph-mon[34421]: pgmap v876: 129 pgs: 129 active+clean; 616 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 17 MiB/s wr, 940 op/s 2024-05-09T09:35:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:47 smithi069 ceph-mon[30123]: pgmap v876: 129 pgs: 129 active+clean; 616 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 17 MiB/s wr, 940 op/s 2024-05-09T09:35:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:47 smithi163 ceph-mon[34354]: pgmap v876: 129 pgs: 129 active+clean; 616 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 17 MiB/s wr, 940 op/s 2024-05-09T09:35:47.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 125155 29.67 MB/sec execute 105 sec latency 127.555 ms 2024-05-09T09:35:48.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 125865 29.68 MB/sec execute 106 sec latency 49.100 ms 2024-05-09T09:35:49.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:49 smithi103 ceph-mon[34421]: pgmap v877: 129 pgs: 129 active+clean; 601 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 903 op/s 2024-05-09T09:35:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:49 smithi069 ceph-mon[30123]: pgmap v877: 129 pgs: 129 active+clean; 601 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 903 op/s 2024-05-09T09:35:49.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:49 smithi163 ceph-mon[34354]: pgmap v877: 129 pgs: 129 active+clean; 601 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 16 MiB/s wr, 903 op/s 2024-05-09T09:35:49.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 126477 29.77 MB/sec execute 107 sec latency 96.531 ms 2024-05-09T09:35:50.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 127044 29.80 MB/sec execute 108 sec latency 129.158 ms 2024-05-09T09:35:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:51 smithi103 ceph-mon[34421]: pgmap v878: 129 pgs: 129 active+clean; 588 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 920 op/s 2024-05-09T09:35:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:51 smithi069 ceph-mon[30123]: pgmap v878: 129 pgs: 129 active+clean; 588 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 920 op/s 2024-05-09T09:35:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:51 smithi163 ceph-mon[34354]: pgmap v878: 129 pgs: 129 active+clean; 588 MiB data, 6.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 920 op/s 2024-05-09T09:35:51.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 127589 29.72 MB/sec execute 109 sec latency 107.623 ms 2024-05-09T09:35:52.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 128193 29.77 MB/sec execute 110 sec latency 120.374 ms 2024-05-09T09:35:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:53 smithi103 ceph-mon[34421]: pgmap v879: 129 pgs: 129 active+clean; 588 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 778 op/s 2024-05-09T09:35:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:53 smithi069 ceph-mon[30123]: pgmap v879: 129 pgs: 129 active+clean; 588 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 778 op/s 2024-05-09T09:35:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:53 smithi163 ceph-mon[34354]: pgmap v879: 129 pgs: 129 active+clean; 588 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 778 op/s 2024-05-09T09:35:53.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 128995 29.81 MB/sec execute 111 sec latency 54.001 ms 2024-05-09T09:35:54.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 129811 30.04 MB/sec execute 112 sec latency 44.793 ms 2024-05-09T09:35:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:55 smithi103 ceph-mon[34421]: pgmap v880: 129 pgs: 129 active+clean; 590 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 7.8 MiB/s wr, 710 op/s 2024-05-09T09:35:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:55 smithi069 ceph-mon[30123]: pgmap v880: 129 pgs: 129 active+clean; 590 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 7.8 MiB/s wr, 710 op/s 2024-05-09T09:35:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:55 smithi163 ceph-mon[34354]: pgmap v880: 129 pgs: 129 active+clean; 590 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 7.8 MiB/s wr, 710 op/s 2024-05-09T09:35:55.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 130531 30.12 MB/sec execute 113 sec latency 58.820 ms 2024-05-09T09:35:56.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 131126 30.15 MB/sec execute 114 sec latency 61.434 ms 2024-05-09T09:35:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:57 smithi103 ceph-mon[34421]: pgmap v881: 129 pgs: 129 active+clean; 603 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.4 MiB/s wr, 786 op/s 2024-05-09T09:35:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:57 smithi069 ceph-mon[30123]: pgmap v881: 129 pgs: 129 active+clean; 603 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.4 MiB/s wr, 786 op/s 2024-05-09T09:35:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:57 smithi163 ceph-mon[34354]: pgmap v881: 129 pgs: 129 active+clean; 603 MiB data, 6.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 7.4 MiB/s wr, 786 op/s 2024-05-09T09:35:57.827 INFO:tasks.workunit.client.0.smithi069.stdout: 10 131691 30.13 MB/sec execute 115 sec latency 57.823 ms 2024-05-09T09:35:58.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 132281 30.11 MB/sec execute 116 sec latency 68.026 ms 2024-05-09T09:35:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 -- 172.21.15.69:0/4186338881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84101980 msgr2=0x7f6e84101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 --2- 172.21.15.69:0/4186338881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84101980 0x7f6e84101e00 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c009900 tx=0x7f6e6c02a410 comp rx=0 tx=0).stop 2024-05-09T09:35:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 -- 172.21.15.69:0/4186338881 shutdown_connections 2024-05-09T09:35:59.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 --2- 172.21.15.69:0/4186338881 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84102340 0x7f6e84108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 --2- 172.21.15.69:0/4186338881 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84101980 0x7f6e84101e00 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 --2- 172.21.15.69:0/4186338881 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.436+0000 7f6e899b0640 1 -- 172.21.15.69:0/4186338881 >> 172.21.15.69:0/4186338881 conn(0x7f6e840fbd00 msgr2=0x7f6e840fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.437+0000 7f6e899b0640 1 -- 172.21.15.69:0/4186338881 shutdown_connections 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.437+0000 7f6e899b0640 1 -- 172.21.15.69:0/4186338881 wait complete. 2024-05-09T09:35:59.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.437+0000 7f6e899b0640 1 Processor -- start 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.437+0000 7f6e899b0640 1 -- start start 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84101980 0x7f6e84102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410c410 con 0x7f6e84102340 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410c290 con 0x7f6e84100790 2024-05-09T09:35:59.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e8410c590 con 0x7f6e84101980 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52328/0 (socket says 172.21.15.69:52328) 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e827fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84101980 0x7f6e84102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e837fe640 1 -- 172.21.15.69:0/3712535157 learned_addr learned my addr 172.21.15.69:0/3712535157 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 -- 172.21.15.69:0/3712535157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84101980 msgr2=0x7f6e84102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84101980 0x7f6e84102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 -- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 msgr2=0x7f6e841032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 -- 172.21.15.69:0/3712535157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e84103b30 con 0x7f6e84100790 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e837fe640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84104720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e9c0 tx=0x7f6e7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e889ae640 1 -- 172.21.15.69:0/3712535157 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400ccc0 con 0x7f6e84100790 2024-05-09T09:35:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.438+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e841a1c10 con 0x7f6e84100790 2024-05-09T09:35:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.439+0000 7f6e889ae640 1 -- 172.21.15.69:0/3712535157 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e740035f0 con 0x7f6e84100790 2024-05-09T09:35:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.439+0000 7f6e889ae640 1 -- 172.21.15.69:0/3712535157 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7400ce60 con 0x7f6e84100790 2024-05-09T09:35:59.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.439+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e841a2180 con 0x7f6e84100790 2024-05-09T09:35:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e889ae640 1 -- 172.21.15.69:0/3712535157 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e740037a0 con 0x7f6e84100790 2024-05-09T09:35:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 Processor -- start 2024-05-09T09:35:59.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 -- start start 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b72740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b72db0 0x55ffe6b75240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6ad1340 con 0x55ffe6b6d250 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6a79290 con 0x55ffe6b6dc40 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6acd300 con 0x55ffe6b72db0 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52330/0 (socket says 172.21.15.69:52330) 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e837fe640 1 -- 172.21.15.69:0/2553563210 learned_addr learned my addr 172.21.15.69:0/2553563210 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e827fc640 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b72db0 0x55ffe6b75240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.441+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b72740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 -- 172.21.15.69:0/2553563210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b72db0 msgr2=0x55ffe6b75240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b72db0 0x55ffe6b75240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 -- 172.21.15.69:0/2553563210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 msgr2=0x55ffe6b72740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b72740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 -- 172.21.15.69:0/2553563210 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe6a9d8e0 con 0x55ffe6b6d250 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e889ae640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 0x7f6e700a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0118b0 tx=0x7f6e7c011d70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=448087919287f16c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 0x7f6e700a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e79ffb640 1 -- 172.21.15.69:0/2553563210 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c01e070 con 0x55ffe6b6d250 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e889ae640 1 -- 172.21.15.69:0/3712535157 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6e740cf820 con 0x7f6e84100790 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e79ffb640 1 -- 172.21.15.69:0/2553563210 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c011310 con 0x55ffe6b6d250 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.442+0000 7f6e79ffb640 1 -- 172.21.15.69:0/2553563210 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c019e60 con 0x55ffe6b6d250 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2553563210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 msgr2=0x55ffe6b6d670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c0118b0 tx=0x7f6e7c011d70 comp rx=0 tx=0).stop 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e82ffd640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 0x7f6e700a50f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6e740cd310 tx=0x7f6e74019f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2553563210 shutdown_connections 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b72db0 0x55ffe6b75240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b72740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2553563210 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6d250 0x55ffe6b6d670 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2553563210 >> 172.21.15.69:0/2553563210 conn(0x55ffe6b688c0 msgr2=0x55ffe6b68cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2553563210 shutdown_connections 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2553563210 wait complete. 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 -- start start 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.443+0000 7f6e8c7a0740 1 Processor -- start 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b773d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6b75d60 con 0x55ffe6b6dc40 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6a8c790 con 0x55ffe6b72db0 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ffe6b75ee0 con 0x55ffe6b6d250 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 10 client.0 _ll_register_callbacks cb 0x55ffe6ac7a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b773d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52342/0 (socket says 172.21.15.69:52342) 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 -- 172.21.15.69:0/1606738771 learned_addr learned my addr 172.21.15.69:0/1606738771 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e837fe640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 -- 172.21.15.69:0/1606738771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 msgr2=0x55ffe6b773d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b773d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 -- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 msgr2=0x55ffe6b7e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 -- 172.21.15.69:0/1606738771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe67e5210 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e827fc640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c02a930 tx=0x7f6e6c005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=942aecb04b89200c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e6c035040 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e7c011310 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.444+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6e6c0319b0 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6e6c04f050 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 10 client.17400 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ffe68b1620 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c011310 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f6e6c004810 con 0x55ffe6b6dc40 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 10 client.17400 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 10 client.17400 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:59.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 20 client.17400 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6b82ef0 0x55ffe6b85300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 4 client.17400 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.445+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ffe6adea70 con 0x55ffe6b82ef0 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.446+0000 7f6e837fe640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6b82ef0 0x55ffe6b85300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.446+0000 7f6e62ffd640 10 client.17400 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.446+0000 7f6e837fe640 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6b82ef0 0x55ffe6b85300 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=39425f3a83eb3471 server_cookie=36ac7e731292219f in_seq=0 out_seq=0 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.447+0000 7f6e62ffd640 1 -- 172.21.15.69:0/1606738771 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ffe6adea70 con 0x55ffe6b82ef0 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.447+0000 7f6e62ffd640 10 client.17400 handle_command_reply: tid=0 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.447+0000 7f6e8c7a0740 1 client.17400 shutdown 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 msgr2=0x55ffe6b77910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f6e6c02a930 tx=0x7f6e6c005e50 comp rx=0 tx=0).stop 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 shutdown_connections 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b77910 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6b82ef0 0x55ffe6b85300 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/1606738771 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b773d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e62ffd640 0 client.17400 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 >> 172.21.15.69:0/1606738771 conn(0x55ffe6b688c0 msgr2=0x55ffe6b67a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 shutdown_connections 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/1606738771 wait complete. 2024-05-09T09:35:59.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.448+0000 7f6e8c7a0740 20 client.17400 trim_cache size 0 max 16384 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 Processor -- start 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 -- start start 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b723f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b8ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841aec90 con 0x55ffe6b6dc40 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841ac560 con 0x55ffe6b72db0 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.466+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841aa460 con 0x55ffe6b6d250 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52344/0 (socket says 172.21.15.69:52344) 2024-05-09T09:35:59.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b8ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e827fc640 1 -- 172.21.15.69:0/2504464579 learned_addr learned my addr 172.21.15.69:0/2504464579 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b723f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 -- 172.21.15.69:0/2504464579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 msgr2=0x55ffe6b723f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b723f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 -- 172.21.15.69:0/2504464579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 msgr2=0x55ffe6b7e910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 -- 172.21.15.69:0/2504464579 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ffe6b6e910 con 0x55ffe6b72db0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e827fc640 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.467+0000 7f6e837fe640 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b8ac10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c011880 tx=0x7f6e7c011d70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9086636f8bbd2c85 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e797fa640 1 -- 172.21.15.69:0/2504464579 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c01e040 con 0x55ffe6b72db0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e797fa640 1 -- 172.21.15.69:0/2504464579 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c01f040 con 0x55ffe6b72db0 2024-05-09T09:35:59.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e797fa640 1 -- 172.21.15.69:0/2504464579 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c0199f0 con 0x55ffe6b72db0 2024-05-09T09:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2504464579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 msgr2=0x55ffe6b8ac10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b8ac10 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c011880 tx=0x7f6e7c011d70 comp rx=0 tx=0).stop 2024-05-09T09:35:59.471 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2504464579 shutdown_connections 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b72db0 0x55ffe6b8ac10 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7e910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/2504464579 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b723f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.468+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2504464579 >> 172.21.15.69:0/2504464579 conn(0x55ffe6b688c0 msgr2=0x55ffe6b68ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.469+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2504464579 shutdown_connections 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.469+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/2504464579 wait complete. 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.469+0000 7f6e8c7a0740 1 -- start start 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.469+0000 7f6e8c7a0740 1 Processor -- start 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841aaa20 con 0x55ffe6b72db0 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841abe90 con 0x55ffe6b6dc40 2024-05-09T09:35:59.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e841ab0f0 con 0x55ffe6b6d250 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e8c7a0740 10 client.0 _ll_register_callbacks cb 0x7f6e6c032c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:43828/0 (socket says 172.21.15.69:43828) 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 -- 172.21.15.69:0/756257251 learned_addr learned my addr 172.21.15.69:0/756257251 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e827fc640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 -- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 msgr2=0x55ffe6b7c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 -- 172.21.15.69:0/756257251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b72db0 msgr2=0x55ffe6b7d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.470+0000 7f6e837fe640 1 -- 172.21.15.69:0/756257251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ffe6b7d6c0 con 0x55ffe6b6d250 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e837fe640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009560 tx=0x7f6e7c011d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c403656067a093 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e62ffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:35:59.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e7c01e030 con 0x55ffe6b6d250 2024-05-09T09:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f6e7c01f040 con 0x55ffe6b6d250 2024-05-09T09:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6e7c004bd0 con 0x55ffe6b6d250 2024-05-09T09:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6e7c004dd0 con 0x55ffe6b6d250 2024-05-09T09:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e8c7a0740 10 client.26614 fetch_fsmap learned FSMap version 47 2024-05-09T09:35:59.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ffe6b7e9d0 con 0x55ffe6b6d250 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.471+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e7c01f040 con 0x55ffe6b6d250 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f6e7c03a040 con 0x55ffe6b6d250 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 10 client.26614 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 10 client.26614 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 20 client.26614 populate_metadata read hostname 'smithi069' 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6be9770 0x55ffe6bebb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 4 client.26614 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ffe6ad8dc0 con 0x55ffe6be9770 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.472+0000 7f6e827fc640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6be9770 0x55ffe6bebb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:35:59.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.473+0000 7f6e827fc640 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6be9770 0x55ffe6bebb90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4335652dc063306 server_cookie=b68f39c00ff8c73 in_seq=0 out_seq=0 2024-05-09T09:35:59.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:35:59.473+0000 7f6e62ffd640 10 client.26614 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:35:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:35:59 smithi069 ceph-mon[30123]: pgmap v882: 129 pgs: 129 active+clean; 605 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 505 op/s 2024-05-09T09:35:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:35:59 smithi163 ceph-mon[34354]: pgmap v882: 129 pgs: 129 active+clean; 605 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 505 op/s 2024-05-09T09:35:59.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 132910 30.14 MB/sec execute 117 sec latency 334.437 ms 2024-05-09T09:35:59.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:35:59 smithi103 ceph-mon[34421]: pgmap v882: 129 pgs: 129 active+clean; 605 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 505 op/s 2024-05-09T09:36:00.189 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e62ffd640 1 -- 172.21.15.69:0/756257251 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ffe6ad8dc0 con 0x55ffe6be9770 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e62ffd640 10 client.26614 handle_command_reply: tid=0 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 client.26614 shutdown 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 msgr2=0x55ffe6b7bbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6e7c009560 tx=0x7f6e7c011d70 comp rx=0 tx=0).stop 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 shutdown_connections 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ffe6b72db0 0x55ffe6b7d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ffe6b6dc40 0x55ffe6b7c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ffe6be9770 0x55ffe6bebb90 crc :-1 s=READY pgs=213 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 --2- 172.21.15.69:0/756257251 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ffe6b6d250 0x55ffe6b7bbb0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e62ffd640 0 client.26614 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 >> 172.21.15.69:0/756257251 conn(0x55ffe6b688c0 msgr2=0x55ffe6b6ee20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 shutdown_connections 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.188+0000 7f6e8c7a0740 1 -- 172.21.15.69:0/756257251 wait complete. 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.189+0000 7f6e8c7a0740 20 client.26614 trim_cache size 0 max 16384 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:36:00.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.190+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 msgr2=0x7f6e700a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.190+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 0x7f6e700a50f0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f6e740cd310 tx=0x7f6e74019f90 comp rx=0 tx=0).stop 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.190+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 msgr2=0x7f6e84104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.190+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84104720 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6e7400e9c0 tx=0x7f6e7400ee80 comp rx=0 tx=0).stop 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 shutdown_connections 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e700a2c30 0x7f6e700a50f0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e84102340 0x7f6e841032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e84101980 0x7f6e84102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 --2- 172.21.15.69:0/3712535157 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e84100790 0x7f6e84104720 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 >> 172.21.15.69:0/3712535157 conn(0x7f6e840fbd00 msgr2=0x7f6e840fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 shutdown_connections 2024-05-09T09:36:00.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:00.191+0000 7f6e899b0640 1 -- 172.21.15.69:0/3712535157 wait complete. 2024-05-09T09:36:00.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 133555 30.11 MB/sec execute 118 sec latency 551.473 ms 2024-05-09T09:36:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:01 smithi069 ceph-mon[30123]: pgmap v883: 129 pgs: 129 active+clean; 614 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.1 MiB/s wr, 508 op/s 2024-05-09T09:36:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:01 smithi163 ceph-mon[34354]: pgmap v883: 129 pgs: 129 active+clean; 614 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.1 MiB/s wr, 508 op/s 2024-05-09T09:36:01.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 134239 30.33 MB/sec execute 119 sec latency 58.488 ms 2024-05-09T09:36:01.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:01 smithi103 ceph-mon[34421]: pgmap v883: 129 pgs: 129 active+clean; 614 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.1 MiB/s wr, 508 op/s 2024-05-09T09:36:02.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 134917 30.39 MB/sec execute 120 sec latency 44.200 ms 2024-05-09T09:36:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:03 smithi069 ceph-mon[30123]: pgmap v884: 129 pgs: 129 active+clean; 613 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-05-09T09:36:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:03 smithi163 ceph-mon[34354]: pgmap v884: 129 pgs: 129 active+clean; 613 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-05-09T09:36:03.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 135049 30.17 MB/sec execute 121 sec latency 704.935 ms 2024-05-09T09:36:03.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:03 smithi103 ceph-mon[34421]: pgmap v884: 129 pgs: 129 active+clean; 613 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.7 MiB/s wr, 446 op/s 2024-05-09T09:36:04.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 135063 29.92 MB/sec execute 122 sec latency 1682.814 ms 2024-05-09T09:36:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:05 smithi069 ceph-mon[30123]: pgmap v885: 129 pgs: 129 active+clean; 610 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-05-09T09:36:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:05 smithi163 ceph-mon[34354]: pgmap v885: 129 pgs: 129 active+clean; 610 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-05-09T09:36:05.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 135269 29.75 MB/sec execute 123 sec latency 1506.638 ms 2024-05-09T09:36:05.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:05 smithi103 ceph-mon[34421]: pgmap v885: 129 pgs: 129 active+clean; 610 MiB data, 7.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 3.5 MiB/s wr, 446 op/s 2024-05-09T09:36:06.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 135496 29.60 MB/sec execute 124 sec latency 139.151 ms 2024-05-09T09:36:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:07 smithi069 ceph-mon[30123]: pgmap v886: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 9.7 MiB/s wr, 695 op/s 2024-05-09T09:36:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:07 smithi163 ceph-mon[34354]: pgmap v886: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 9.7 MiB/s wr, 695 op/s 2024-05-09T09:36:07.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 135809 29.50 MB/sec execute 125 sec latency 99.554 ms 2024-05-09T09:36:07.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:07 smithi103 ceph-mon[34421]: pgmap v886: 129 pgs: 129 active+clean; 664 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 9.7 MiB/s wr, 695 op/s 2024-05-09T09:36:08.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 136565 29.50 MB/sec execute 126 sec latency 53.201 ms 2024-05-09T09:36:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:09 smithi069 ceph-mon[30123]: pgmap v887: 129 pgs: 129 active+clean; 679 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-05-09T09:36:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:09 smithi163 ceph-mon[34354]: pgmap v887: 129 pgs: 129 active+clean; 679 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-05-09T09:36:09.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 137432 29.68 MB/sec execute 127 sec latency 75.362 ms 2024-05-09T09:36:09.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:09 smithi103 ceph-mon[34421]: pgmap v887: 129 pgs: 129 active+clean; 679 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 11 MiB/s wr, 678 op/s 2024-05-09T09:36:10.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 138141 29.83 MB/sec execute 128 sec latency 52.630 ms 2024-05-09T09:36:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:11 smithi069 ceph-mon[30123]: pgmap v888: 129 pgs: 129 active+clean; 643 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 788 op/s 2024-05-09T09:36:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:11 smithi163 ceph-mon[34354]: pgmap v888: 129 pgs: 129 active+clean; 643 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 788 op/s 2024-05-09T09:36:11.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 138688 29.82 MB/sec execute 129 sec latency 48.611 ms 2024-05-09T09:36:11.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:11 smithi103 ceph-mon[34421]: pgmap v888: 129 pgs: 129 active+clean; 643 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 12 MiB/s wr, 788 op/s 2024-05-09T09:36:12.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:12 smithi069 ceph-mon[30123]: pgmap v889: 129 pgs: 129 active+clean; 602 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 784 op/s 2024-05-09T09:36:12.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:12 smithi163 ceph-mon[34354]: pgmap v889: 129 pgs: 129 active+clean; 602 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 784 op/s 2024-05-09T09:36:12.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 139201 29.80 MB/sec execute 130 sec latency 52.562 ms 2024-05-09T09:36:12.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:12 smithi103 ceph-mon[34421]: pgmap v889: 129 pgs: 129 active+clean; 602 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 784 op/s 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 -- 172.21.15.69:0/92488350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240101980 msgr2=0x7ff240101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 --2- 172.21.15.69:0/92488350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240101980 0x7ff240101e00 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff228009900 tx=0x7ff22802a410 comp rx=0 tx=0).stop 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 -- 172.21.15.69:0/92488350 shutdown_connections 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 --2- 172.21.15.69:0/92488350 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240102340 0x7ff240108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 --2- 172.21.15.69:0/92488350 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240101980 0x7ff240101e00 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 --2- 172.21.15.69:0/92488350 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240100790 0x7ff240100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 -- 172.21.15.69:0/92488350 >> 172.21.15.69:0/92488350 conn(0x7ff2400fbd00 msgr2=0x7ff2400fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.316 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 -- 172.21.15.69:0/92488350 shutdown_connections 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.314+0000 7ff246714640 1 -- 172.21.15.69:0/92488350 wait complete. 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 Processor -- start 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 -- start start 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 0x7ff240197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240101980 0x7ff240198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 0x7ff24019f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010c410 con 0x7ff240102340 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010c290 con 0x7ff240100790 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff246714640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff24010c590 con 0x7ff240101980 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 0x7ff24019f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 0x7ff240197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff237fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240101980 0x7ff240198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 0x7ff240197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35564/0 (socket says 172.21.15.69:35564) 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff23ffff640 1 -- 172.21.15.69:0/4128671512 learned_addr learned my addr 172.21.15.69:0/4128671512 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:13.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 -- 172.21.15.69:0/4128671512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240101980 msgr2=0x7ff240198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240101980 0x7ff240198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 -- 172.21.15.69:0/4128671512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 msgr2=0x7ff240197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 0x7ff240197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.315+0000 7ff244cb1640 1 -- 172.21.15.69:0/4128671512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ff24019f730 con 0x7ff240102340 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff244cb1640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 0x7ff24019f0c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff238009fd0 tx=0x7ff23800ef10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff23dffb640 1 -- 172.21.15.69:0/4128671512 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff23800ce20 con 0x7ff240102340 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff23dffb640 1 -- 172.21.15.69:0/4128671512 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff238005020 con 0x7ff240102340 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ff24019fa20 con 0x7ff240102340 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff23dffb640 1 -- 172.21.15.69:0/4128671512 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff238010720 con 0x7ff240102340 2024-05-09T09:36:13.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.316+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ff24019fef0 con 0x7ff240102340 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff23dffb640 1 -- 172.21.15.69:0/4128671512 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ff238005540 con 0x7ff240102340 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 Processor -- start 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 -- start start 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782d250 0x563b3782d670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563b37791340 con 0x563b37832db0 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563b37739290 con 0x563b3782dc40 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff249428740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563b3778d300 con 0x563b3782d250 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff23ffff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:35572/0 (socket says 172.21.15.69:35572) 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59664/0 (socket says 172.21.15.69:59664) 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.319+0000 7ff23ffff640 1 -- 172.21.15.69:0/4035116107 learned_addr learned my addr 172.21.15.69:0/4035116107 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782d250 0x563b3782d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff23dffb640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 0x7ff22c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 -- 172.21.15.69:0/4035116107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782d250 msgr2=0x563b3782d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782d250 0x563b3782d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 -- 172.21.15.69:0/4035116107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 msgr2=0x563b37832740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 -- 172.21.15.69:0/4035116107 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b3775d8e0 con 0x563b37832db0 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff23ffff640 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff244cb1640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 0x7ff22c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff237fff640 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff228031440 tx=0x7ff228005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6c0cc1be42d5f1ff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff23dffb640 1 -- 172.21.15.69:0/4128671512 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff2380cfad0 con 0x7ff240102340 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff2367fc640 1 -- 172.21.15.69:0/4035116107 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228035070 con 0x563b37832db0 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff2367fc640 1 -- 172.21.15.69:0/4035116107 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff228005030 con 0x563b37832db0 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.320+0000 7ff2367fc640 1 -- 172.21.15.69:0/4035116107 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff2280049e0 con 0x563b37832db0 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- 172.21.15.69:0/4035116107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 msgr2=0x563b37835240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff244cb1640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 0x7ff22c0a50f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff23809bcb0 tx=0x7ff238066c30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff228031440 tx=0x7ff228005fd0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- 172.21.15.69:0/4035116107 shutdown_connections 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b37832db0 0x563b37835240 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37832740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 --2- 172.21.15.69:0/4035116107 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782d250 0x563b3782d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- 172.21.15.69:0/4035116107 >> 172.21.15.69:0/4035116107 conn(0x563b378288c0 msgr2=0x563b37828cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- 172.21.15.69:0/4035116107 shutdown_connections 2024-05-09T09:36:13.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- 172.21.15.69:0/4035116107 wait complete. 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.321+0000 7ff249428740 1 -- start start 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 Processor -- start 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b3783bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b37842db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563b37835ca0 con 0x563b3782d250 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563b3774c790 con 0x563b37832db0 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563b37835e20 con 0x563b3782dc40 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff249428740 10 client.0 _ll_register_callbacks cb 0x563b37787a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59680/0 (socket says 172.21.15.69:59680) 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 -- 172.21.15.69:0/1841010261 learned_addr learned my addr 172.21.15.69:0/1841010261 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:13.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff23ffff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b3783bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b37842db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 -- 172.21.15.69:0/1841010261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 msgr2=0x563b3783bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b3783bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 -- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 msgr2=0x563b37842db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b37842db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.322+0000 7ff237fff640 1 -- 172.21.15.69:0/1841010261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b374a5210 con 0x563b3782d250 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff237fff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff22802a450 tx=0x7ff2280058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f720df0641a49b02 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228035030 con 0x563b3782d250 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff228004240 con 0x563b3782d250 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff228005030 con 0x563b3782d250 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff228004440 con 0x563b3782d250 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 10 client.17418 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:13.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563b37571620 con 0x563b3782d250 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff228005030 con 0x563b3782d250 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7ff228003920 con 0x563b3782d250 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 10 client.17418 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 10 client.17418 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 20 client.17418 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.323+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b37846f10 0x563b37849300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.324+0000 7ff249428740 4 client.17418 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.324+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563b3779ea70 con 0x563b37846f10 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.324+0000 7ff23ffff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b37846f10 0x563b37849300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.324+0000 7ff23ffff640 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b37846f10 0x563b37849300 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ede626e78e4cb9 server_cookie=a7944ec95533e42 in_seq=0 out_seq=0 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.324+0000 7ff21f7fe640 10 client.17418 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.325+0000 7ff21f7fe640 1 -- 172.21.15.69:0/1841010261 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563b3779ea70 con 0x563b37846f10 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.325+0000 7ff21f7fe640 10 client.17418 handle_command_reply: tid=0 2024-05-09T09:36:13.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.325+0000 7ff249428740 1 client.17418 shutdown 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 msgr2=0x563b3783b830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff22802a450 tx=0x7ff2280058b0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 shutdown_connections 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b37842db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b37846f10 0x563b37849300 crc :-1 s=READY pgs=214 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b3783bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 --2- 172.21.15.69:0/1841010261 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b3783b830 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff21f7fe640 0 client.17418 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 >> 172.21.15.69:0/1841010261 conn(0x563b378288c0 msgr2=0x563b37827a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 shutdown_connections 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 1 -- 172.21.15.69:0/1841010261 wait complete. 2024-05-09T09:36:13.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.326+0000 7ff249428740 20 client.17418 trim_cache size 0 max 16384 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 Processor -- start 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 -- start start 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b37842d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b3783b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401aa950 con 0x563b3782d250 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401a8220 con 0x563b37832db0 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff249428740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401a6070 con 0x563b3782dc40 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.344 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59682/0 (socket says 172.21.15.69:59682) 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff237fff640 1 -- 172.21.15.69:0/1870917805 learned_addr learned my addr 172.21.15.69:0/1870917805 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff23ffff640 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b37842d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.342+0000 7ff237fff640 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b3783b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 -- 172.21.15.69:0/1870917805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 msgr2=0x563b37842d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b37842d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 -- 172.21.15.69:0/1870917805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 msgr2=0x563b3783b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b3783b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 -- 172.21.15.69:0/1870917805 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563b378498e0 con 0x563b3782d250 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff237fff640 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff2280057c0 tx=0x7ff228005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9002e6171c96a6e9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff235ffb640 1 -- 172.21.15.69:0/1870917805 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228035070 con 0x563b3782d250 2024-05-09T09:36:13.345 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff235ffb640 1 -- 172.21.15.69:0/1870917805 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff228003ad0 con 0x563b3782d250 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.343+0000 7ff235ffb640 1 -- 172.21.15.69:0/1870917805 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff22804ca00 con 0x563b3782d250 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- 172.21.15.69:0/1870917805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 msgr2=0x563b37828180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff2280057c0 tx=0x7ff228005c90 comp rx=0 tx=0).stop 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- 172.21.15.69:0/1870917805 shutdown_connections 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b37832db0 0x563b3783b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b3782dc40 0x563b37842d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 --2- 172.21.15.69:0/1870917805 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37828180 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- 172.21.15.69:0/1870917805 >> 172.21.15.69:0/1870917805 conn(0x563b378288c0 msgr2=0x563b3782b870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- 172.21.15.69:0/1870917805 shutdown_connections 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- 172.21.15.69:0/1870917805 wait complete. 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 -- start start 2024-05-09T09:36:13.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.344+0000 7ff249428740 1 Processor -- start 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37840b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b37832db0 0x563b37841b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401a66e0 con 0x563b3782d250 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401a7b50 con 0x563b3782dc40 2024-05-09T09:36:13.350 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ff2401a6db0 con 0x563b37832db0 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 10 client.0 _ll_register_callbacks cb 0x7ff228009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:59684/0 (socket says 172.21.15.69:59684) 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 -- 172.21.15.69:0/3252008409 learned_addr learned my addr 172.21.15.69:0/3252008409 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b37832db0 0x563b37841b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 -- 172.21.15.69:0/3252008409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b37832db0 msgr2=0x563b37841b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b37832db0 0x563b37841b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 -- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 msgr2=0x563b37840b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37840b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 -- 172.21.15.69:0/3252008409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563b37842170 con 0x563b3782d250 2024-05-09T09:36:13.351 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff237fff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff228031610 tx=0x7ff228005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=645dae4f7348c0ad server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff21f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ff228035070 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ff22804c9a0 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ff228003ad0 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7ff228005d50 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 10 client.17424 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.345+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563b37842170 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ff228003ad0 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7ff228004c60 con 0x563b3782d250 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 10 client.17424 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:13.352 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 10 client.17424 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 20 client.17424 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b378adb80 0x563b378aff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 4 client.17424 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563b37798dc0 con 0x563b378adb80 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff23ffff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b378adb80 0x563b378aff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff21f7fe640 10 client.17424 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:13.353 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.346+0000 7ff23ffff640 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b378adb80 0x563b378aff70 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8132090573167a82 server_cookie=86fa41b913fccc1 in_seq=0 out_seq=0 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.734+0000 7ff21f7fe640 1 -- 172.21.15.69:0/3252008409 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563b37798dc0 con 0x563b378adb80 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.734+0000 7ff21f7fe640 10 client.17424 handle_command_reply: tid=0 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.734+0000 7ff249428740 1 client.17424 shutdown 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 msgr2=0x563b37840630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff228031610 tx=0x7ff228005fd0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 shutdown_connections 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563b378adb80 0x563b378aff70 crc :-1 s=READY pgs=215 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563b37832db0 0x563b37841b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff21f7fe640 0 client.17424 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:13.735 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563b3782dc40 0x563b37840b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 --2- 172.21.15.69:0/3252008409 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563b3782d250 0x563b37840630 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 >> 172.21.15.69:0/3252008409 conn(0x563b378288c0 msgr2=0x563b3783b440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 shutdown_connections 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 1 -- 172.21.15.69:0/3252008409 wait complete. 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.735+0000 7ff249428740 20 client.17424 trim_cache size 0 max 16384 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:36:13.736 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:36:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 msgr2=0x7ff22c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 0x7ff22c0a50f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff23809bcb0 tx=0x7ff238066c30 comp rx=0 tx=0).stop 2024-05-09T09:36:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 msgr2=0x7ff24019f0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:13.737 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 0x7ff24019f0c0 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff238009fd0 tx=0x7ff23800ef10 comp rx=0 tx=0).stop 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 shutdown_connections 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ff22c0a2c30 0x7ff22c0a50f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ff240102340 0x7ff24019f0c0 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ff240101980 0x7ff240198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 --2- 172.21.15.69:0/4128671512 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ff240100790 0x7ff240197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 >> 172.21.15.69:0/4128671512 conn(0x7ff2400fbd00 msgr2=0x7ff2400fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 shutdown_connections 2024-05-09T09:36:13.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:13.736+0000 7ff246714640 1 -- 172.21.15.69:0/4128671512 wait complete. 2024-05-09T09:36:13.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 139701 29.68 MB/sec execute 131 sec latency 258.027 ms 2024-05-09T09:36:14.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:14 smithi069 ceph-mon[30123]: pgmap v890: 129 pgs: 129 active+clean; 603 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 767 op/s 2024-05-09T09:36:14.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:14 smithi163 ceph-mon[34354]: pgmap v890: 129 pgs: 129 active+clean; 603 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 767 op/s 2024-05-09T09:36:14.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 140455 29.79 MB/sec execute 132 sec latency 53.633 ms 2024-05-09T09:36:14.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:14 smithi103 ceph-mon[34421]: pgmap v890: 129 pgs: 129 active+clean; 603 MiB data, 7.1 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 12 MiB/s wr, 767 op/s 2024-05-09T09:36:15.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 141307 29.94 MB/sec execute 133 sec latency 54.549 ms 2024-05-09T09:36:16.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:16 smithi069 ceph-mon[30123]: pgmap v891: 129 pgs: 129 active+clean; 607 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 903 op/s 2024-05-09T09:36:16.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:16 smithi163 ceph-mon[34354]: pgmap v891: 129 pgs: 129 active+clean; 607 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 903 op/s 2024-05-09T09:36:16.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 141940 30.00 MB/sec execute 134 sec latency 66.532 ms 2024-05-09T09:36:16.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:16 smithi103 ceph-mon[34421]: pgmap v891: 129 pgs: 129 active+clean; 607 MiB data, 7.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 903 op/s 2024-05-09T09:36:17.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 142602 30.04 MB/sec execute 135 sec latency 39.081 ms 2024-05-09T09:36:18.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:18 smithi069 ceph-mon[30123]: pgmap v892: 129 pgs: 129 active+clean; 611 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 739 op/s 2024-05-09T09:36:18.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:18 smithi163 ceph-mon[34354]: pgmap v892: 129 pgs: 129 active+clean; 611 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 739 op/s 2024-05-09T09:36:18.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 143074 30.01 MB/sec execute 136 sec latency 77.450 ms 2024-05-09T09:36:18.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:18 smithi103 ceph-mon[34421]: pgmap v892: 129 pgs: 129 active+clean; 611 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 7.4 MiB/s wr, 739 op/s 2024-05-09T09:36:19.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 143708 29.99 MB/sec execute 137 sec latency 52.247 ms 2024-05-09T09:36:20.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:20 smithi069 ceph-mon[30123]: pgmap v893: 129 pgs: 129 active+clean; 613 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.8 MiB/s wr, 637 op/s 2024-05-09T09:36:20.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:20 smithi163 ceph-mon[34354]: pgmap v893: 129 pgs: 129 active+clean; 613 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.8 MiB/s wr, 637 op/s 2024-05-09T09:36:20.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 144456 30.10 MB/sec execute 138 sec latency 48.941 ms 2024-05-09T09:36:20.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:20 smithi103 ceph-mon[34421]: pgmap v893: 129 pgs: 129 active+clean; 613 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 4.8 MiB/s wr, 637 op/s 2024-05-09T09:36:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:21 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:36:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:21 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:36:21.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 145131 30.14 MB/sec execute 139 sec latency 52.316 ms 2024-05-09T09:36:21.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:21 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:36:22.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: pgmap v894: 129 pgs: 129 active+clean; 621 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:36:22.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:36:22.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:36:22.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:36:22.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:36:22.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 145786 30.17 MB/sec execute 140 sec latency 57.271 ms 2024-05-09T09:36:22.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: pgmap v894: 129 pgs: 129 active+clean; 621 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:36:22.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:36:22.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:36:22.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:36:23.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: pgmap v894: 129 pgs: 129 active+clean; 621 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.8 MiB/s rd, 4.2 MiB/s wr, 614 op/s 2024-05-09T09:36:23.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi163 to 176.2M 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi163 to 184780288: error parsing value: Value '184780288' is below minimum 939524096 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: Adjusting osd_memory_target on smithi103 to 176.2M 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: Unable to set osd_memory_target on smithi103 to 184789606: error parsing value: Value '184789606' is below minimum 939524096 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:36:23.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:36:23.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 146490 30.27 MB/sec execute 141 sec latency 46.005 ms 2024-05-09T09:36:24.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:24 smithi163 ceph-mon[34354]: pgmap v895: 129 pgs: 129 active+clean; 625 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-05-09T09:36:24.812 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:24 smithi069 ceph-mon[30123]: pgmap v895: 129 pgs: 129 active+clean; 625 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-05-09T09:36:24.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 147005 30.21 MB/sec execute 142 sec latency 64.403 ms 2024-05-09T09:36:24.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:24 smithi103 ceph-mon[34421]: pgmap v895: 129 pgs: 129 active+clean; 625 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.7 MiB/s wr, 522 op/s 2024-05-09T09:36:25.828 INFO:tasks.workunit.client.0.smithi069.stdout: 10 147628 30.26 MB/sec execute 143 sec latency 58.563 ms 2024-05-09T09:36:26.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:26 smithi163 ceph-mon[34354]: pgmap v896: 129 pgs: 129 active+clean; 632 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 552 op/s 2024-05-09T09:36:26.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 148132 30.24 MB/sec execute 144 sec latency 176.950 ms 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/1864334435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0101980 msgr2=0x7f1bd0101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/1864334435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0101980 0x7f1bd0101e00 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f1bb8009900 tx=0x7f1bb802a410 comp rx=0 tx=0).stop 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/1864334435 shutdown_connections 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/1864334435 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0102340 0x7f1bd0108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/1864334435 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0101980 0x7f1bd0101e00 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.840 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/1864334435 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 0x7f1bd0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/1864334435 >> 172.21.15.69:0/1864334435 conn(0x7f1bd00fbd00 msgr2=0x7f1bd00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/1864334435 shutdown_connections 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.835+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/1864334435 wait complete. 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 Processor -- start 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 -- start start 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 0x7f1bd01958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0101980 0x7f1bd0195e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd010b360 con 0x7f1bd0102340 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd010b1e0 con 0x7f1bd0100790 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bd4f6a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd010b4e0 con 0x7f1bd0101980 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bced9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bced9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33962/0 (socket says 172.21.15.69:33962) 2024-05-09T09:36:26.841 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0101980 0x7f1bd0195e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bced9d640 1 -- 172.21.15.69:0/318327893 learned_addr learned my addr 172.21.15.69:0/318327893 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.836+0000 7f1bce59c640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 0x7f1bd01958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 -- 172.21.15.69:0/318327893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0101980 msgr2=0x7f1bd0195e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0101980 0x7f1bd0195e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 -- 172.21.15.69:0/318327893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 msgr2=0x7f1bd01958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 0x7f1bd01958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 -- 172.21.15.69:0/318327893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1bd019d510 con 0x7f1bd0102340 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bced9d640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800eaa0 tx=0x7f1bc800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bc7fff640 1 -- 172.21.15.69:0/318327893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc800cdb0 con 0x7f1bd0102340 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bc7fff640 1 -- 172.21.15.69:0/318327893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc8005020 con 0x7f1bd0102340 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.837+0000 7f1bc7fff640 1 -- 172.21.15.69:0/318327893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc80116b0 con 0x7f1bd0102340 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.838+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1bd019d800 con 0x7f1bd0102340 2024-05-09T09:36:26.842 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.838+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1bd019dc70 con 0x7f1bd0102340 2024-05-09T09:36:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 Processor -- start 2024-05-09T09:36:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bc7fff640 1 -- 172.21.15.69:0/318327893 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1bc8005560 con 0x7f1bd0102340 2024-05-09T09:36:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 -- start start 2024-05-09T09:36:26.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c2a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 0x55ae61c2f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ae61b8e340 con 0x55ae61c2fdb0 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae61b36290 con 0x55ae61c2ac40 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.841+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae61b8a300 con 0x55ae61c2a250 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33970/0 (socket says 172.21.15.69:33970) 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc5d9b640 1 -- 172.21.15.69:0/1892997658 learned_addr learned my addr 172.21.15.69:0/1892997658 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 0x55ae61c2f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c2a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc7fff640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 0x7f1bbc0a95a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 -- 172.21.15.69:0/1892997658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 msgr2=0x55ae61c2a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c2a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 -- 172.21.15.69:0/1892997658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 msgr2=0x55ae61c32240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bce59c640 1 -- 172.21.15.69:0/1892997658 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae61b5a8e0 con 0x55ae61c2ac40 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.842+0000 7f1bced9d640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 0x7f1bbc0a95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bc7fff640 1 -- 172.21.15.69:0/318327893 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1bc80cf530 con 0x7f1bd0102340 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 0x55ae61c2f740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009bd0 tx=0x7f1bc0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6d3a7fa08239be2e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bced9d640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 0x7f1bbc0a95a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1bc809b7c0 tx=0x7f1bc8066740 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bc4d99640 1 -- 172.21.15.69:0/1892997658 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0004230 con 0x55ae61c2ac40 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bc4d99640 1 -- 172.21.15.69:0/1892997658 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc0009e40 con 0x55ae61c2ac40 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.843+0000 7f1bc4d99640 1 -- 172.21.15.69:0/1892997658 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0019820 con 0x55ae61c2ac40 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1892997658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 msgr2=0x55ae61c2f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 0x55ae61c2f740 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0009bd0 tx=0x7f1bc0011810 comp rx=0 tx=0).stop 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1892997658 shutdown_connections 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2fdb0 0x55ae61c32240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.848 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2ac40 0x55ae61c2f740 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1892997658 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c2a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1892997658 >> 172.21.15.69:0/1892997658 conn(0x55ae61c258c0 msgr2=0x55ae61c25cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.844+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1892997658 shutdown_connections 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.845+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1892997658 wait complete. 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.845+0000 7f1bd7c71740 1 -- start start 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.845+0000 7f1bd7c71740 1 Processor -- start 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c441f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55ae61c32db0 con 0x55ae61c2ac40 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55ae61b49790 con 0x55ae61c2a250 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55ae61c32f30 con 0x55ae61c2fdb0 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bd7c71740 10 client.0 _ll_register_callbacks cb 0x55ae61b84a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:26.849 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33976/0 (socket says 172.21.15.69:33976) 2024-05-09T09:36:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bce59c640 1 -- 172.21.15.69:0/1570841555 learned_addr learned my addr 172.21.15.69:0/1570841555 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c441f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.850 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 -- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 msgr2=0x55ae61c3cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 -- 172.21.15.69:0/1570841555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 msgr2=0x55ae61c3d1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.851 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bc5d9b640 1 -- 172.21.15.69:0/1570841555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae618a2210 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.846+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.847+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c441f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55ae61c2b4a0 tx=0x7f1bb802a410 comp rx=0 tx=0).ready entity=mon.1 client_cookie=86e0c75402fbf02d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.847+0000 7f1bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.848+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bb8035070 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.848+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1bb80037f0 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.848+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x55ae61b9ba70 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1bb80039f0 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 10 client.26644 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ae6196e620 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x55ae61b9ba70 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f1bb803da30 con 0x55ae61c2fdb0 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 10 client.26644 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:26.852 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 10 client.26644 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 20 client.26644 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61c4ab20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 4 client.26644 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.849+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55ae61b95dc0 con 0x55ae61c486f0 2024-05-09T09:36:26.853 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.850+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61c4ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.855+0000 7f1bce59c640 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61c4ab20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77d1fa3f15fd0fe server_cookie=ff48c1ca06367c08 in_seq=0 out_seq=0 2024-05-09T09:36:26.855 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.855+0000 7f1bb5ffb640 10 client.26644 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.857+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/1570841555 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55ae61b95dc0 con 0x55ae61c486f0 2024-05-09T09:36:26.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.857+0000 7f1bb5ffb640 10 client.26644 handle_command_reply: tid=0 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.857+0000 7f1bd7c71740 1 client.26644 shutdown 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.857+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 msgr2=0x55ae61c441f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.857+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c441f0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x55ae61c2b4a0 tx=0x7f1bb802a410 comp rx=0 tx=0).stop 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 shutdown_connections 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61c4ab20 crc :-1 s=READY pgs=216 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c441f0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bb5ffb640 0 client.26644 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/1570841555 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 >> 172.21.15.69:0/1570841555 conn(0x55ae61c258c0 msgr2=0x55ae61c24a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 shutdown_connections 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 1 -- 172.21.15.69:0/1570841555 wait complete. 2024-05-09T09:36:26.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.858+0000 7f1bd7c71740 20 client.26644 trim_cache size 0 max 16384 2024-05-09T09:36:26.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:26 smithi103 ceph-mon[34421]: pgmap v896: 129 pgs: 129 active+clean; 632 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 552 op/s 2024-05-09T09:36:26.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.886+0000 7f1bd7c71740 1 Processor -- start 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.886+0000 7f1bd7c71740 1 -- start start 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.886+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2fdb0 0x55ae61c28840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01aaac0 con 0x55ae61c2ac40 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a8390 con 0x55ae61c2fdb0 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a61e0 con 0x55ae61c2a250 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33992/0 (socket says 172.21.15.69:33992) 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 -- 172.21.15.69:0/3708851991 learned_addr learned my addr 172.21.15.69:0/3708851991 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.888 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 -- 172.21.15.69:0/3708851991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 msgr2=0x55ae61c3cc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 -- 172.21.15.69:0/3708851991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2fdb0 msgr2=0x55ae61c28840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2fdb0 0x55ae61c28840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 -- 172.21.15.69:0/3708851991 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55ae61c2fac0 con 0x55ae61c2ac40 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0003b60 tx=0x7f1bc0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b50828db8146e8d5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.887+0000 7f1bb7fff640 1 -- 172.21.15.69:0/3708851991 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0031070 con 0x55ae61c2ac40 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bb7fff640 1 -- 172.21.15.69:0/3708851991 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc0025420 con 0x55ae61c2ac40 2024-05-09T09:36:26.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bb7fff640 1 -- 172.21.15.69:0/3708851991 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc0019660 con 0x55ae61c2ac40 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3708851991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 msgr2=0x55ae61c3d1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0003b60 tx=0x7f1bc0005f90 comp rx=0 tx=0).stop 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3708851991 shutdown_connections 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2fdb0 0x55ae61c28840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c3d1b0 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3708851991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2a250 0x55ae61c3cc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.888+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3708851991 >> 172.21.15.69:0/3708851991 conn(0x55ae61c258c0 msgr2=0x55ae61c25ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3708851991 shutdown_connections 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3708851991 wait complete. 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 -- start start 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 Processor -- start 2024-05-09T09:36:26.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c40a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c40f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c41f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.889+0000 7f1bd7c71740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a6850 con 0x55ae61c2ac40 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c40a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bc5d9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c40a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:39784/0 (socket says 172.21.15.69:39784) 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bc5d9b640 1 -- 172.21.15.69:0/3632291873 learned_addr learned my addr 172.21.15.69:0/3632291873 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c40f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a7cc0 con 0x55ae61c2a250 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1bd01a6f20 con 0x55ae61c2fdb0 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bd7c71740 10 client.0 _ll_register_callbacks cb 0x7f1bb8032880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 -- 172.21.15.69:0/3632291873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 msgr2=0x55ae61c41f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c41f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 -- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 msgr2=0x55ae61c40a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:26.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c40a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55ae61c42560 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bce59c640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c40f60 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0025d90 tx=0x7f1bc0009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=700ebc586ea57645 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bb5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1bc003a060 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.890+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1bc0025420 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1bc0005220 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f1bc0041050 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 10 client.17442 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55ae61c43830 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1bc0025420 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f1bc002c050 con 0x55ae61c2ac40 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 10 client.17442 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 10 client.17442 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 20 client.17442 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:26.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61cb1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 4 client.17442 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:36:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.891+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55ae61b138e0 con 0x55ae61c486f0 2024-05-09T09:36:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.893+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61cb1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.894+0000 7f1bb5ffb640 10 client.17442 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:26.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:26.894+0000 7f1bc5d9b640 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61cb1040 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1b02dd3e4636e6 server_cookie=c64fea68b810979d in_seq=0 out_seq=0 2024-05-09T09:36:27.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:26 smithi069 ceph-mon[30123]: pgmap v896: 129 pgs: 129 active+clean; 632 MiB data, 7.3 GiB used, 1.0 TiB / 1.0 TiB avail; 10 MiB/s rd, 4.0 MiB/s wr, 552 op/s 2024-05-09T09:36:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bb5ffb640 1 -- 172.21.15.69:0/3632291873 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55ae61b138e0 con 0x55ae61c486f0 2024-05-09T09:36:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bb5ffb640 10 client.17442 handle_command_reply: tid=0 2024-05-09T09:36:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 client.17442 shutdown 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 msgr2=0x55ae61c40f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c40f60 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f1bc0025d90 tx=0x7f1bc0009780 comp rx=0 tx=0).stop 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 shutdown_connections 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55ae61c486f0 0x55ae61cb1040 crc :-1 s=READY pgs=217 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55ae61c2fdb0 0x55ae61c41f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bb5ffb640 0 client.17442 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55ae61c2ac40 0x55ae61c40f60 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 --2- 172.21.15.69:0/3632291873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55ae61c2a250 0x55ae61c40a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 >> 172.21.15.69:0/3632291873 conn(0x55ae61c258c0 msgr2=0x55ae61c2be80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 shutdown_connections 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.528+0000 7f1bd7c71740 1 -- 172.21.15.69:0/3632291873 wait complete. 2024-05-09T09:36:27.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd7c71740 20 client.17442 trim_cache size 0 max 16384 2024-05-09T09:36:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:36:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:36:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:36:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 msgr2=0x7f1bbc0a95a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 0x7f1bbc0a95a0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f1bc809b7c0 tx=0x7f1bc8066740 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 msgr2=0x7f1bd019cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f1bc800eaa0 tx=0x7f1bc800ef60 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 shutdown_connections 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1bbc0a70e0 0x7f1bbc0a95a0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1bd0102340 0x7f1bd019cea0 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1bd0101980 0x7f1bd0195e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 --2- 172.21.15.69:0/318327893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1bd0100790 0x7f1bd01958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 >> 172.21.15.69:0/318327893 conn(0x7f1bd00fbd00 msgr2=0x7f1bd0104c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 shutdown_connections 2024-05-09T09:36:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:27.529+0000 7f1bd4f6a640 1 -- 172.21.15.69:0/318327893 wait complete. 2024-05-09T09:36:27.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 148434 30.11 MB/sec execute 145 sec latency 369.128 ms 2024-05-09T09:36:28.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:28 smithi163 ceph-mon[34354]: pgmap v897: 129 pgs: 129 active+clean; 639 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 480 op/s 2024-05-09T09:36:28.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149042 30.14 MB/sec execute 146 sec latency 131.131 ms 2024-05-09T09:36:28.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:28 smithi103 ceph-mon[34421]: pgmap v897: 129 pgs: 129 active+clean; 639 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 480 op/s 2024-05-09T09:36:29.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:28 smithi069 ceph-mon[30123]: pgmap v897: 129 pgs: 129 active+clean; 639 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.6 MiB/s rd, 3.7 MiB/s wr, 480 op/s 2024-05-09T09:36:29.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149074 29.94 MB/sec execute 147 sec latency 1113.482 ms 2024-05-09T09:36:30.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:30 smithi163 ceph-mon[34354]: pgmap v898: 129 pgs: 129 active+clean; 646 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 448 op/s 2024-05-09T09:36:30.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149108 29.74 MB/sec execute 148 sec latency 1946.753 ms 2024-05-09T09:36:30.870 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:30 smithi103 ceph-mon[34421]: pgmap v898: 129 pgs: 129 active+clean; 646 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 448 op/s 2024-05-09T09:36:31.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:30 smithi069 ceph-mon[30123]: pgmap v898: 129 pgs: 129 active+clean; 646 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.8 MiB/s wr, 448 op/s 2024-05-09T09:36:31.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149340 29.62 MB/sec execute 149 sec latency 2188.980 ms 2024-05-09T09:36:32.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149584 29.52 MB/sec execute 150 sec latency 121.182 ms 2024-05-09T09:36:33.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:32 smithi069 ceph-mon[30123]: pgmap v899: 129 pgs: 129 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-05-09T09:36:33.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:32 smithi163 ceph-mon[34354]: pgmap v899: 129 pgs: 129 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-05-09T09:36:33.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:32 smithi103 ceph-mon[34421]: pgmap v899: 129 pgs: 129 active+clean; 702 MiB data, 7.4 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 9.3 MiB/s wr, 651 op/s 2024-05-09T09:36:33.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 149921 29.49 MB/sec execute 151 sec latency 94.923 ms 2024-05-09T09:36:34.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 150531 29.43 MB/sec execute 152 sec latency 84.558 ms 2024-05-09T09:36:35.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:34 smithi069 ceph-mon[30123]: pgmap v900: 129 pgs: 129 active+clean; 694 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 607 op/s 2024-05-09T09:36:35.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:34 smithi163 ceph-mon[34354]: pgmap v900: 129 pgs: 129 active+clean; 694 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 607 op/s 2024-05-09T09:36:35.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:34 smithi103 ceph-mon[34421]: pgmap v900: 129 pgs: 129 active+clean; 694 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 9.7 MiB/s wr, 607 op/s 2024-05-09T09:36:35.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 151012 29.47 MB/sec execute 153 sec latency 365.816 ms 2024-05-09T09:36:36.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 151580 29.50 MB/sec execute 154 sec latency 187.155 ms 2024-05-09T09:36:37.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:36 smithi069 ceph-mon[30123]: pgmap v901: 129 pgs: 129 active+clean; 646 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 13 MiB/s wr, 814 op/s 2024-05-09T09:36:37.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:36 smithi163 ceph-mon[34354]: pgmap v901: 129 pgs: 129 active+clean; 646 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 13 MiB/s wr, 814 op/s 2024-05-09T09:36:37.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:36 smithi103 ceph-mon[34421]: pgmap v901: 129 pgs: 129 active+clean; 646 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 13 MiB/s wr, 814 op/s 2024-05-09T09:36:37.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 152136 29.46 MB/sec execute 155 sec latency 204.923 ms 2024-05-09T09:36:38.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 152635 29.43 MB/sec execute 156 sec latency 128.915 ms 2024-05-09T09:36:39.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:38 smithi069 ceph-mon[30123]: pgmap v902: 129 pgs: 129 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 846 op/s 2024-05-09T09:36:39.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:38 smithi163 ceph-mon[34354]: pgmap v902: 129 pgs: 129 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 846 op/s 2024-05-09T09:36:39.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:38 smithi103 ceph-mon[34421]: pgmap v902: 129 pgs: 129 active+clean; 633 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 13 MiB/s wr, 846 op/s 2024-05-09T09:36:39.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 153279 29.49 MB/sec execute 157 sec latency 67.021 ms 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/3543404032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 msgr2=0x7f7e88108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/3543404032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e88108bd0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7e80009900 tx=0x7f7e8002a410 comp rx=0 tx=0).stop 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/3543404032 shutdown_connections 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/3543404032 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e88108bd0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/3543404032 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88101980 0x7f7e88101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/3543404032 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88100790 0x7f7e88100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/3543404032 >> 172.21.15.69:0/3543404032 conn(0x7f7e880fbd00 msgr2=0x7f7e880fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/3543404032 shutdown_connections 2024-05-09T09:36:40.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/3543404032 wait complete. 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 Processor -- start 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.628+0000 7f7e8fd9b640 1 -- start start 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88100790 0x7f7e88197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88101980 0x7f7e88198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8810d530 con 0x7f7e88102340 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8810d3b0 con 0x7f7e88101980 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8fd9b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7e8810d6b0 con 0x7f7e88100790 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51826/0 (socket says 172.21.15.69:51826) 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 -- 172.21.15.69:0/524240572 learned_addr learned my addr 172.21.15.69:0/524240572 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8d336640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88101980 0x7f7e88198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8db37640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88100790 0x7f7e88197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 -- 172.21.15.69:0/524240572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88100790 msgr2=0x7f7e88197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88100790 0x7f7e88197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 -- 172.21.15.69:0/524240572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88101980 msgr2=0x7f7e88198070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88101980 0x7f7e88198070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 -- 172.21.15.69:0/524240572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7e8819f700 con 0x7f7e88102340 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.629+0000 7f7e8e338640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7e800314e0 tx=0x7f7e800042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.630+0000 7f7e86ffd640 1 -- 172.21.15.69:0/524240572 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e8003f030 con 0x7f7e88102340 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.630+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7e8819f990 con 0x7f7e88102340 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.630+0000 7f7e86ffd640 1 -- 172.21.15.69:0/524240572 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e80003ce0 con 0x7f7e88102340 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.630+0000 7f7e86ffd640 1 -- 172.21.15.69:0/524240572 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80035070 con 0x7f7e88102340 2024-05-09T09:36:40.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.630+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7e8819fe80 con 0x7f7e88102340 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.632+0000 7f7e86ffd640 1 -- 172.21.15.69:0/524240572 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7e80042030 con 0x7f7e88102340 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.632+0000 7f7e86ffd640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 0x7f7e700a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8db37640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 0x7f7e700a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e86ffd640 1 -- 172.21.15.69:0/524240572 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7e800ef800 con 0x7f7e88102340 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 Processor -- start 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 -- start start 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a011c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8db37640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 0x7f7e700a50f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7e780099a0 tx=0x7f7e78009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.641 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0121740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 0x55b5a0124240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0080340 con 0x55b5a0121db0 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0028290 con 0x55b5a011cc40 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e92b83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b5a007c300 con 0x55b5a011c250 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8db37640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a011c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8e338640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 0x55b5a0124240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8db37640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a011c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:46308/0 (socket says 172.21.15.69:46308) 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8db37640 1 -- 172.21.15.69:0/894589181 learned_addr learned my addr 172.21.15.69:0/894589181 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.633+0000 7f7e8d336640 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0121740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 -- 172.21.15.69:0/894589181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 msgr2=0x55b5a011c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.642 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a011c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 -- 172.21.15.69:0/894589181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 msgr2=0x55b5a0121740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0121740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 -- 172.21.15.69:0/894589181 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a004c8e0 con 0x55b5a0121db0 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e8e338640 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 0x55b5a0124240 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7e800ecf80 tx=0x7f7e8003b670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69deb2fd0ff8e8f9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e6ffff640 1 -- 172.21.15.69:0/894589181 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e80049040 con 0x55b5a0121db0 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e6ffff640 1 -- 172.21.15.69:0/894589181 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e800bbd90 con 0x55b5a0121db0 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.634+0000 7f7e6ffff640 1 -- 172.21.15.69:0/894589181 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e800b8400 con 0x55b5a0121db0 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- 172.21.15.69:0/894589181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 msgr2=0x55b5a0124240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 0x55b5a0124240 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f7e800ecf80 tx=0x7f7e8003b670 comp rx=0 tx=0).stop 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- 172.21.15.69:0/894589181 shutdown_connections 2024-05-09T09:36:40.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a0121db0 0x55b5a0124240 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0121740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 --2- 172.21.15.69:0/894589181 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a011c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- 172.21.15.69:0/894589181 >> 172.21.15.69:0/894589181 conn(0x55b5a01178c0 msgr2=0x55b5a0117cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- 172.21.15.69:0/894589181 shutdown_connections 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- 172.21.15.69:0/894589181 wait complete. 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 -- start start 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.635+0000 7f7e92b83740 1 Processor -- start 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a01261f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a0121db0 0x55b5a012d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0124b20 con 0x55b5a011cc40 2024-05-09T09:36:40.644 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5a003b790 con 0x55b5a0121db0 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b5a0124ca0 con 0x55b5a011c250 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 10 client.0 _ll_register_callbacks cb 0x55b5a0076a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51832/0 (socket says 172.21.15.69:51832) 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8db37640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a01261f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 -- 172.21.15.69:0/1162321224 learned_addr learned my addr 172.21.15.69:0/1162321224 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8e338640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a0121db0 0x55b5a012d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.645 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 -- 172.21.15.69:0/1162321224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 msgr2=0x55b5a01261f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a01261f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 -- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a0121db0 msgr2=0x55b5a012d700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a0121db0 0x55b5a012d700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 -- 172.21.15.69:0/1162321224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b59fd94210 con 0x55b5a011cc40 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8db37640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a01261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e8d336640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0126e0 tx=0x7f7e7c012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9f6fd82b51f52e29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c004020 con 0x55b5a011cc40 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7e7c0041c0 con 0x55b5a011cc40 2024-05-09T09:36:40.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.636+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e800bbd90 con 0x55b5a011cc40 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7e7c025050 con 0x55b5a011cc40 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 10 client.17454 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b59fe60620 con 0x55b5a011cc40 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e800bbd90 con 0x55b5a011cc40 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f7e7c003a60 con 0x55b5a011cc40 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 10 client.17454 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 10 client.17454 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:40.647 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 20 client.17454 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0131cb0 0x55b5a0134120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 4 client.17454 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5a008da70 con 0x55b5a0131cb0 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.637+0000 7f7e8db37640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0131cb0 0x55b5a0134120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.638+0000 7f7e8db37640 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0131cb0 0x55b5a0134120 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45bf17baf262db32 server_cookie=f3c64d85f101c6ec in_seq=0 out_seq=0 2024-05-09T09:36:40.648 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.638+0000 7f7e6d7fa640 10 client.17454 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/1162321224 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5a008da70 con 0x55b5a0131cb0 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e6d7fa640 10 client.17454 handle_command_reply: tid=0 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 client.17454 shutdown 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 msgr2=0x55b5a0126730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f7e7c0126e0 tx=0x7f7e7c012ba0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.649 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 shutdown_connections 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a0121db0 0x55b5a012d700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a0126730 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0131cb0 0x55b5a0134120 crc :-1 s=READY pgs=218 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 --2- 172.21.15.69:0/1162321224 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a011c250 0x55b5a01261f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 >> 172.21.15.69:0/1162321224 conn(0x55b5a01178c0 msgr2=0x55b5a0116a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e6d7fa640 0 client.17454 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:40.650 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 shutdown_connections 2024-05-09T09:36:40.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 1 -- 172.21.15.69:0/1162321224 wait complete. 2024-05-09T09:36:40.651 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.639+0000 7f7e92b83740 20 client.17454 trim_cache size 0 max 16384 2024-05-09T09:36:40.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 Processor -- start 2024-05-09T09:36:40.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 -- start start 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0139f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881acc50 con 0x55b5a011c250 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881aa520 con 0x55b5a011cc40 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.657+0000 7f7e92b83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881a8370 con 0x55b5a0121db0 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51846/0 (socket says 172.21.15.69:51846) 2024-05-09T09:36:40.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 -- 172.21.15.69:0/427435295 learned_addr learned my addr 172.21.15.69:0/427435295 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8d336640 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0139f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8e338640 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 -- 172.21.15.69:0/427435295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 msgr2=0x55b5a012d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 -- 172.21.15.69:0/427435295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 msgr2=0x55b5a0139f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0139f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 -- 172.21.15.69:0/427435295 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5a0127090 con 0x55b5a011c250 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e8db37640 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f7e78005fd0 tx=0x7f7e78014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c13506fe7beb97bd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e6f7fe640 1 -- 172.21.15.69:0/427435295 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7802c260 con 0x55b5a011c250 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e6f7fe640 1 -- 172.21.15.69:0/427435295 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e7802c8e0 con 0x55b5a011c250 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.658+0000 7f7e6f7fe640 1 -- 172.21.15.69:0/427435295 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7802a400 con 0x55b5a011c250 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- 172.21.15.69:0/427435295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 msgr2=0x55b5a0139a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f7e78005fd0 tx=0x7f7e78014000 comp rx=0 tx=0).stop 2024-05-09T09:36:40.670 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- 172.21.15.69:0/427435295 shutdown_connections 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011cc40 0x55b5a0139f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 --2- 172.21.15.69:0/427435295 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011c250 0x55b5a0139a30 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- 172.21.15.69:0/427435295 >> 172.21.15.69:0/427435295 conn(0x55b5a01178c0 msgr2=0x55b5a0117ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- 172.21.15.69:0/427435295 shutdown_connections 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- 172.21.15.69:0/427435295 wait complete. 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.659+0000 7f7e92b83740 1 -- start start 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 Processor -- start 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011c250 0x55b5a012b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.671 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012c550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881a89e0 con 0x55b5a011cc40 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881a9e50 con 0x55b5a011c250 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7e881a90b0 con 0x55b5a0121db0 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e92b83740 10 client.0 _ll_register_callbacks cb 0x7f7e7c009910 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51850/0 (socket says 172.21.15.69:51850) 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 -- 172.21.15.69:0/515290544 learned_addr learned my addr 172.21.15.69:0/515290544 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8db37640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011c250 0x55b5a012b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 -- 172.21.15.69:0/515290544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 msgr2=0x55b5a012c550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012c550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 -- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011c250 msgr2=0x55b5a012b000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011c250 0x55b5a012b000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.660+0000 7f7e8d336640 1 -- 172.21.15.69:0/515290544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5a01347a0 con 0x55b5a011cc40 2024-05-09T09:36:40.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e8d336640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x55b5a0127660 tx=0x7f7e7c003820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a2f339f090f7855b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7e7c022040 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7e7c003c20 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7e7802c8e0 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7e7c036050 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 10 client.17460 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b5a01982d0 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7e7802c8e0 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f7e7c031030 con 0x55b5a011cc40 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 10 client.17460 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 10 client.17460 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 20 client.17460 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0198cf0 0x55b5a019b110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:40.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 4 client.17460 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:36:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5a0087dc0 con 0x55b5a0198cf0 2024-05-09T09:36:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.661+0000 7f7e8db37640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0198cf0 0x55b5a019b110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.662+0000 7f7e8db37640 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0198cf0 0x55b5a019b110 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cfc967f7206ed806 server_cookie=561c7e1978c4decc in_seq=0 out_seq=0 2024-05-09T09:36:40.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:40.662+0000 7f7e6d7fa640 10 client.17460 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:40.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 153922 29.48 MB/sec execute 158 sec latency 105.535 ms 2024-05-09T09:36:41.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:40 smithi069 ceph-mon[30123]: pgmap v903: 129 pgs: 129 active+clean; 630 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 798 op/s 2024-05-09T09:36:41.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:40 smithi163 ceph-mon[34354]: pgmap v903: 129 pgs: 129 active+clean; 630 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 798 op/s 2024-05-09T09:36:41.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:40 smithi103 ceph-mon[34421]: pgmap v903: 129 pgs: 129 active+clean; 630 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 12 MiB/s wr, 798 op/s 2024-05-09T09:36:41.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.208+0000 7f7e6d7fa640 1 -- 172.21.15.69:0/515290544 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5a0087dc0 con 0x55b5a0198cf0 2024-05-09T09:36:41.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.208+0000 7f7e6d7fa640 10 client.17460 handle_command_reply: tid=0 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.208+0000 7f7e92b83740 1 client.17460 shutdown 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.208+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 msgr2=0x55b5a012b540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.208+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x55b5a0127660 tx=0x7f7e7c003820 comp rx=0 tx=0).stop 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 shutdown_connections 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5a0198cf0 0x55b5a019b110 crc :-1 s=READY pgs=219 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e6d7fa640 0 client.17460 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5a0121db0 0x55b5a012c550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5a011cc40 0x55b5a012b540 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 --2- 172.21.15.69:0/515290544 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5a011c250 0x55b5a012b000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 >> 172.21.15.69:0/515290544 conn(0x55b5a01178c0 msgr2=0x55b5a0116170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 shutdown_connections 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 1 -- 172.21.15.69:0/515290544 wait complete. 2024-05-09T09:36:41.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e92b83740 20 client.17460 trim_cache size 0 max 16384 2024-05-09T09:36:41.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:36:41.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:36:41.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:36:41.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 msgr2=0x7f7e700a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 0x7f7e700a50f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f7e780099a0 tx=0x7f7e78009500 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 msgr2=0x7f7e8819f090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f7e800314e0 tx=0x7f7e800042f0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 shutdown_connections 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7e700a2c30 0x7f7e700a50f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7e88102340 0x7f7e8819f090 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7e88101980 0x7f7e88198070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 --2- 172.21.15.69:0/524240572 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7e88100790 0x7f7e88197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.209+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 >> 172.21.15.69:0/524240572 conn(0x7f7e880fbd00 msgr2=0x7f7e88104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:41.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.210+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 shutdown_connections 2024-05-09T09:36:41.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:41.210+0000 7f7e8fd9b640 1 -- 172.21.15.69:0/524240572 wait complete. 2024-05-09T09:36:41.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 154417 29.48 MB/sec execute 159 sec latency 426.669 ms 2024-05-09T09:36:42.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 155022 29.56 MB/sec execute 160 sec latency 162.509 ms 2024-05-09T09:36:43.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:42 smithi069 ceph-mon[30123]: pgmap v904: 129 pgs: 129 active+clean; 626 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 769 op/s 2024-05-09T09:36:43.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:42 smithi163 ceph-mon[34354]: pgmap v904: 129 pgs: 129 active+clean; 626 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 769 op/s 2024-05-09T09:36:43.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:42 smithi103 ceph-mon[34421]: pgmap v904: 129 pgs: 129 active+clean; 626 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 12 MiB/s wr, 769 op/s 2024-05-09T09:36:43.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 155572 29.55 MB/sec execute 161 sec latency 67.164 ms 2024-05-09T09:36:44.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 156127 29.50 MB/sec execute 162 sec latency 50.169 ms 2024-05-09T09:36:45.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:44 smithi069 ceph-mon[30123]: pgmap v905: 129 pgs: 129 active+clean; 623 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.8 MiB/s wr, 613 op/s 2024-05-09T09:36:45.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:44 smithi163 ceph-mon[34354]: pgmap v905: 129 pgs: 129 active+clean; 623 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.8 MiB/s wr, 613 op/s 2024-05-09T09:36:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:44 smithi103 ceph-mon[34421]: pgmap v905: 129 pgs: 129 active+clean; 623 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 6.8 MiB/s wr, 613 op/s 2024-05-09T09:36:45.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 156796 29.55 MB/sec execute 163 sec latency 54.392 ms 2024-05-09T09:36:46.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 157658 29.60 MB/sec execute 164 sec latency 67.728 ms 2024-05-09T09:36:47.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:46 smithi069 ceph-mon[30123]: pgmap v906: 129 pgs: 129 active+clean; 628 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-05-09T09:36:47.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:46 smithi163 ceph-mon[34354]: pgmap v906: 129 pgs: 129 active+clean; 628 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-05-09T09:36:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:46 smithi103 ceph-mon[34421]: pgmap v906: 129 pgs: 129 active+clean; 628 MiB data, 7.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 6.4 MiB/s wr, 730 op/s 2024-05-09T09:36:47.829 INFO:tasks.workunit.client.0.smithi069.stdout: 10 157978 29.59 MB/sec execute 165 sec latency 614.789 ms 2024-05-09T09:36:48.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 158570 29.65 MB/sec execute 166 sec latency 52.023 ms 2024-05-09T09:36:49.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:48 smithi069 ceph-mon[30123]: pgmap v907: 129 pgs: 129 active+clean; 632 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 535 op/s 2024-05-09T09:36:49.058 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:48 smithi163 ceph-mon[34354]: pgmap v907: 129 pgs: 129 active+clean; 632 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 535 op/s 2024-05-09T09:36:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:48 smithi103 ceph-mon[34421]: pgmap v907: 129 pgs: 129 active+clean; 632 MiB data, 7.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 3.2 MiB/s wr, 535 op/s 2024-05-09T09:36:49.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 159039 29.59 MB/sec execute 167 sec latency 66.321 ms 2024-05-09T09:36:50.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 159656 29.58 MB/sec execute 168 sec latency 124.195 ms 2024-05-09T09:36:51.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:50 smithi069 ceph-mon[30123]: pgmap v908: 129 pgs: 129 active+clean; 639 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-05-09T09:36:51.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:50 smithi163 ceph-mon[34354]: pgmap v908: 129 pgs: 129 active+clean; 639 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-05-09T09:36:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:50 smithi103 ceph-mon[34421]: pgmap v908: 129 pgs: 129 active+clean; 639 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.3 MiB/s wr, 500 op/s 2024-05-09T09:36:51.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 160296 29.63 MB/sec execute 169 sec latency 61.542 ms 2024-05-09T09:36:52.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 160994 29.66 MB/sec execute 170 sec latency 92.657 ms 2024-05-09T09:36:53.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:52 smithi069 ceph-mon[30123]: pgmap v909: 129 pgs: 129 active+clean; 644 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 519 op/s 2024-05-09T09:36:53.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:52 smithi163 ceph-mon[34354]: pgmap v909: 129 pgs: 129 active+clean; 644 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 519 op/s 2024-05-09T09:36:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:52 smithi103 ceph-mon[34421]: pgmap v909: 129 pgs: 129 active+clean; 644 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.5 MiB/s wr, 519 op/s 2024-05-09T09:36:53.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 161666 29.71 MB/sec execute 171 sec latency 52.785 ms 2024-05-09T09:36:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.371+0000 7f5be27a9640 1 -- 172.21.15.69:0/2853121353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc108830 msgr2=0x7f5bdc10ac60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:54.371 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.371+0000 7f5be27a9640 1 --2- 172.21.15.69:0/2853121353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc108830 0x7f5bdc10ac60 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7f5bd4009ca0 tx=0x7f5bd402a8f0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.376+0000 7f5be27a9640 1 -- 172.21.15.69:0/2853121353 shutdown_connections 2024-05-09T09:36:54.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.376+0000 7f5be27a9640 1 --2- 172.21.15.69:0/2853121353 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc10db50 0x7f5bdc10dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.376+0000 7f5be27a9640 1 --2- 172.21.15.69:0/2853121353 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10b1a0 0x7f5bdc10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.376+0000 7f5be27a9640 1 --2- 172.21.15.69:0/2853121353 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc108830 0x7f5bdc10ac60 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.376+0000 7f5be27a9640 1 -- 172.21.15.69:0/2853121353 >> 172.21.15.69:0/2853121353 conn(0x7f5bdc06e8a0 msgr2=0x7f5bdc070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:54.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.383+0000 7f5be27a9640 1 -- 172.21.15.69:0/2853121353 shutdown_connections 2024-05-09T09:36:54.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.384+0000 7f5be27a9640 1 -- 172.21.15.69:0/2853121353 wait complete. 2024-05-09T09:36:54.386 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.384+0000 7f5be27a9640 1 Processor -- start 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 -- start start 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc10b1a0 0x7f5bdc1a6f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10db50 0x7f5bdc1adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc11c3b0 con 0x7f5bdc10b1a0 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc11c230 con 0x7f5bdc108830 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.385+0000 7f5be27a9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5bdc11c530 con 0x7f5bdc10db50 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.386+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.386+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10db50 0x7f5bdc1adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.386+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58800/0 (socket says 172.21.15.69:58800) 2024-05-09T09:36:54.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.386+0000 7f5bdbfff640 1 -- 172.21.15.69:0/373546183 learned_addr learned my addr 172.21.15.69:0/373546183 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.389+0000 7f5bdbfff640 1 -- 172.21.15.69:0/373546183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10db50 msgr2=0x7f5bdc1adfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.389+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10db50 0x7f5bdc1adfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.389+0000 7f5bdbfff640 1 -- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc10b1a0 msgr2=0x7f5bdc1a6f40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.389+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc10b1a0 0x7f5bdc1a6f40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.389+0000 7f5bdbfff640 1 -- 172.21.15.69:0/373546183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5bdc1ae620 con 0x7f5bdc108830 2024-05-09T09:36:54.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.390+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40319e0 tx=0x7f5bd4030820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.391+0000 7f5bd97fa640 1 -- 172.21.15.69:0/373546183 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd4030da0 con 0x7f5bdc108830 2024-05-09T09:36:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.391+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5bdc1ae8b0 con 0x7f5bdc108830 2024-05-09T09:36:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.391+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5bdc1aee80 con 0x7f5bdc108830 2024-05-09T09:36:54.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.392+0000 7f5bd97fa640 1 -- 172.21.15.69:0/373546183 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd402d070 con 0x7f5bdc108830 2024-05-09T09:36:54.392 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.392+0000 7f5bd97fa640 1 -- 172.21.15.69:0/373546183 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd403b780 con 0x7f5bdc108830 2024-05-09T09:36:54.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.395+0000 7f5be5599740 1 Processor -- start 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.398+0000 7f5be5599740 1 -- start start 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0714740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d0714db0 0x5596d0717240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5596d0673340 con 0x5596d0714db0 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5596d061b290 con 0x5596d070f250 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5be5599740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5596d066f300 con 0x5596d070fc40 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.399+0000 7f5bd97fa640 1 -- 172.21.15.69:0/373546183 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5bd403b920 con 0x7f5bdc108830 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58804/0 (socket says 172.21.15.69:58804) 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5be0d46640 1 -- 172.21.15.69:0/1816990952 learned_addr learned my addr 172.21.15.69:0/1816990952 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0714740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5bd97fa640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 0x7f5bc40a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.400+0000 7f5bd97fa640 1 -- 172.21.15.69:0/373546183 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5bd40f10c0 con 0x7f5bdc108830 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 0x7f5bc40a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 -- 172.21.15.69:0/1816990952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 msgr2=0x5596d0714740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0714740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 -- 172.21.15.69:0/1816990952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d0714db0 msgr2=0x5596d0717240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d0714db0 0x5596d0717240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.401+0000 7f5be0d46640 1 -- 172.21.15.69:0/1816990952 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596d063f8e0 con 0x5596d070f250 2024-05-09T09:36:54.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.402+0000 7f5be0d46640 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009fd0 tx=0x7f5bd00158e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=883c202e25de841 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.405+0000 7f5bc27fc640 1 -- 172.21.15.69:0/1816990952 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0039900 con 0x5596d070f250 2024-05-09T09:36:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.405+0000 7f5bc27fc640 1 -- 172.21.15.69:0/1816990952 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd0041400 con 0x5596d070f250 2024-05-09T09:36:54.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.406+0000 7f5be5599740 1 -- 172.21.15.69:0/1816990952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 msgr2=0x5596d070f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:54.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.406+0000 7f5be5599740 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009fd0 tx=0x7f5bd00158e0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.410+0000 7f5be5599740 1 -- 172.21.15.69:0/1816990952 shutdown_connections 2024-05-09T09:36:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.410+0000 7f5be5599740 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d0714db0 0x5596d0717240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.410+0000 7f5be5599740 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0714740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.410+0000 7f5be5599740 1 --2- 172.21.15.69:0/1816990952 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d070f250 0x5596d070f670 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.410+0000 7f5be5599740 1 -- 172.21.15.69:0/1816990952 >> 172.21.15.69:0/1816990952 conn(0x5596d070a8c0 msgr2=0x5596d070acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:54.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.415+0000 7f5be0d46640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 0x7f5bc40a5330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0041000 tx=0x7f5bd003c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.420+0000 7f5be5599740 1 -- 172.21.15.69:0/1816990952 shutdown_connections 2024-05-09T09:36:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.422+0000 7f5be5599740 1 -- 172.21.15.69:0/1816990952 wait complete. 2024-05-09T09:36:54.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.422+0000 7f5be5599740 1 -- start start 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.422+0000 7f5be5599740 1 Processor -- start 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d07192f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0719830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0720870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5596d0717bf0 con 0x5596d070f250 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5596d062e790 con 0x5596d0714db0 2024-05-09T09:36:54.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5596d0717d70 con 0x5596d070fc40 2024-05-09T09:36:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.424+0000 7f5be5599740 10 client.0 _ll_register_callbacks cb 0x5596d0669a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.426+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d07192f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.426+0000 7f5bdbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d07192f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34112/0 (socket says 172.21.15.69:34112) 2024-05-09T09:36:54.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.426+0000 7f5bdbfff640 1 -- 172.21.15.69:0/810332177 learned_addr learned my addr 172.21.15.69:0/810332177 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:54.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.428+0000 7f5be0d46640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0720870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.429+0000 7f5bdb7fe640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0719830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.435+0000 7f5be0d46640 1 -- 172.21.15.69:0/810332177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 msgr2=0x5596d0719830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.435+0000 7f5be0d46640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0719830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.435+0000 7f5be0d46640 1 -- 172.21.15.69:0/810332177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 msgr2=0x5596d07192f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.435+0000 7f5be0d46640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d07192f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.435+0000 7f5be0d46640 1 -- 172.21.15.69:0/810332177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596d0387210 con 0x5596d0714db0 2024-05-09T09:36:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.442+0000 7f5be0d46640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0720870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5596d07104a0 tx=0x7f5bd003b000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a18d4e520200bcc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.443+0000 7f5bbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:36:54.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.444+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0009c80 con 0x5596d0714db0 2024-05-09T09:36:54.444 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.444+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5bd0041620 con 0x5596d0714db0 2024-05-09T09:36:54.448 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.447+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd0041400 con 0x5596d0714db0 2024-05-09T09:36:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.449+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5bd0041820 con 0x5596d0714db0 2024-05-09T09:36:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.450+0000 7f5be5599740 10 client.26747 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:54.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.450+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5596d0453620 con 0x5596d0714db0 2024-05-09T09:36:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.458+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd0041400 con 0x5596d0714db0 2024-05-09T09:36:54.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.458+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f5bd0006030 con 0x5596d0714db0 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 10 client.26747 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 10 client.26747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 20 client.26747 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d0724b20 0x5596d0726f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 4 client.26747 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:36:54.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.461+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5596d0680a70 con 0x5596d0724b20 2024-05-09T09:36:54.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.464+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d0724b20 0x5596d0726f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.475+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d0724b20 0x5596d0726f70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4a68877effca7c6 server_cookie=37c1c7d0c74471e8 in_seq=0 out_seq=0 2024-05-09T09:36:54.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.475+0000 7f5bbb7fe640 10 client.26747 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.479+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/810332177 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5596d0680a70 con 0x5596d0724b20 2024-05-09T09:36:54.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.479+0000 7f5bbb7fe640 10 client.26747 handle_command_reply: tid=0 2024-05-09T09:36:54.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.482+0000 7f5be5599740 1 client.26747 shutdown 2024-05-09T09:36:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.491+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 msgr2=0x5596d0720870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:54.490 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.491+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0720870 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x5596d07104a0 tx=0x7f5bd003b000 comp rx=0 tx=0).stop 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 shutdown_connections 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0720870 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d0724b20 0x5596d0726f70 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0719830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 --2- 172.21.15.69:0/810332177 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d07192f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 >> 172.21.15.69:0/810332177 conn(0x5596d070a8c0 msgr2=0x5596d0709a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.494+0000 7f5bbb7fe640 0 client.26747 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:54.494 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.495+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 shutdown_connections 2024-05-09T09:36:54.498 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.497+0000 7f5be5599740 1 -- 172.21.15.69:0/810332177 wait complete. 2024-05-09T09:36:54.500 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.499+0000 7f5be5599740 20 client.26747 trim_cache size 0 max 16384 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 Processor -- start 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 -- start start 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d0710dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0720830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc8012570 con 0x5596d070f250 2024-05-09T09:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc800fe40 con 0x5596d0714db0 2024-05-09T09:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.532+0000 7f5be5599740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc800dc90 con 0x5596d070fc40 2024-05-09T09:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58820/0 (socket says 172.21.15.69:58820) 2024-05-09T09:36:54.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 -- 172.21.15.69:0/2428878047 learned_addr learned my addr 172.21.15.69:0/2428878047 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 -- 172.21.15.69:0/2428878047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 msgr2=0x5596d0720830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0720830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 -- 172.21.15.69:0/2428878047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 msgr2=0x5596d0710dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d0710dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 -- 172.21.15.69:0/2428878047 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5596d0710020 con 0x5596d0714db0 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5be0d46640 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0006400 tx=0x7f5bd00098f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9a5670d5d8bc89b7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.533+0000 7f5bc1ffb640 1 -- 172.21.15.69:0/2428878047 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0009c80 con 0x5596d0714db0 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.534+0000 7f5bc1ffb640 1 -- 172.21.15.69:0/2428878047 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd0064070 con 0x5596d0714db0 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.534+0000 7f5bc1ffb640 1 -- 172.21.15.69:0/2428878047 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0041400 con 0x5596d0714db0 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 -- 172.21.15.69:0/2428878047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 msgr2=0x5596d0711300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0006400 tx=0x7f5bd00098f0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 -- 172.21.15.69:0/2428878047 shutdown_connections 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d0711300 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070fc40 0x5596d0720830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 --2- 172.21.15.69:0/2428878047 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d070f250 0x5596d0710dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.535+0000 7f5be5599740 1 -- 172.21.15.69:0/2428878047 >> 172.21.15.69:0/2428878047 conn(0x5596d070a8c0 msgr2=0x5596d07192f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:54.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.540+0000 7f5be5599740 1 -- 172.21.15.69:0/2428878047 shutdown_connections 2024-05-09T09:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.543+0000 7f5be5599740 1 -- 172.21.15.69:0/2428878047 wait complete. 2024-05-09T09:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.543+0000 7f5be5599740 1 -- start start 2024-05-09T09:36:54.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 Processor -- start 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070f250 0x5596d071ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d071cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc800e300 con 0x5596d071fb60 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc800f770 con 0x5596d0714db0 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5bc800e9d0 con 0x5596d070f250 2024-05-09T09:36:54.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.544+0000 7f5be5599740 10 client.0 _ll_register_callbacks cb 0x7f5bd0009f20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.545+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.546+0000 7f5be0d46640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34130/0 (socket says 172.21.15.69:34130) 2024-05-09T09:36:54.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.546+0000 7f5be0d46640 1 -- 172.21.15.69:0/4169979942 learned_addr learned my addr 172.21.15.69:0/4169979942 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:36:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.547+0000 7f5be0d46640 1 -- 172.21.15.69:0/4169979942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070f250 msgr2=0x5596d071ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:36:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.547+0000 7f5be0d46640 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070f250 0x5596d071ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.547+0000 7f5be0d46640 1 -- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 msgr2=0x5596d071cfa0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:36:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.547+0000 7f5be0d46640 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d071cfa0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:54.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.547+0000 7f5be0d46640 1 -- 172.21.15.69:0/4169979942 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5596d0720550 con 0x5596d071fb60 2024-05-09T09:36:54.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.549+0000 7f5be0d46640 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009230 tx=0x7f5bd00158e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=80858d70fdfe050a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:36:54.549 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.549+0000 7f5bbb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:36:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.550+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5bd0041600 con 0x5596d071fb60 2024-05-09T09:36:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.551+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5bd0064070 con 0x5596d071fb60 2024-05-09T09:36:54.550 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.551+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5bd0041b70 con 0x5596d071fb60 2024-05-09T09:36:54.551 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.552+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f5bd0041d90 con 0x5596d071fb60 2024-05-09T09:36:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.552+0000 7f5be5599740 10 client.17469 fetch_fsmap learned FSMap version 47 2024-05-09T09:36:54.553 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.552+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5596d078aeb0 con 0x5596d071fb60 2024-05-09T09:36:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.556+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5bd0064070 con 0x5596d071fb60 2024-05-09T09:36:54.555 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.556+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f5bd0054430 con 0x5596d071fb60 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 10 client.17469 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 10 client.17469 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 20 client.17469 populate_metadata read hostname 'smithi069' 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d078b7a0 0x5596d078dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 4 client.17469 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:36:54.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.557+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5596d067adc0 con 0x5596d078b7a0 2024-05-09T09:36:54.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.561+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d078b7a0 0x5596d078dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:36:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.574+0000 7f5bdbfff640 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d078b7a0 0x5596d078dbc0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f707b31a590a18b server_cookie=76c2d133055f53ce in_seq=0 out_seq=0 2024-05-09T09:36:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:54.575+0000 7f5bbb7fe640 10 client.17469 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:54.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 161911 29.63 MB/sec execute 172 sec latency 499.148 ms 2024-05-09T09:36:55.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:54 smithi069 ceph-mon[30123]: pgmap v910: 129 pgs: 129 active+clean; 645 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 471 op/s 2024-05-09T09:36:55.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:54 smithi163 ceph-mon[34354]: pgmap v910: 129 pgs: 129 active+clean; 645 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 471 op/s 2024-05-09T09:36:55.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:54 smithi103 ceph-mon[34421]: pgmap v910: 129 pgs: 129 active+clean; 645 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 471 op/s 2024-05-09T09:36:55.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 161922 29.46 MB/sec execute 173 sec latency 1499.191 ms 2024-05-09T09:36:56.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 161983 29.31 MB/sec execute 174 sec latency 1998.260 ms 2024-05-09T09:36:57.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:56 smithi069 ceph-mon[30123]: pgmap v911: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 502 op/s 2024-05-09T09:36:57.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:56 smithi163 ceph-mon[34354]: pgmap v911: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 502 op/s 2024-05-09T09:36:57.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:56 smithi103 ceph-mon[34421]: pgmap v911: 129 pgs: 129 active+clean; 653 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 4.7 MiB/s wr, 502 op/s 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5bbb7fe640 1 -- 172.21.15.69:0/4169979942 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5596d067adc0 con 0x5596d078b7a0 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5bbb7fe640 10 client.17469 handle_command_reply: tid=0 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 client.17469 shutdown 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 msgr2=0x5596d071ffe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0009230 tx=0x7f5bd00158e0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 shutdown_connections 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5596d071fb60 0x5596d071ffe0 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5596d0714db0 0x5596d071cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5596d078b7a0 0x5596d078dbc0 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 --2- 172.21.15.69:0/4169979942 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5596d070f250 0x5596d071ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.309+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 >> 172.21.15.69:0/4169979942 conn(0x5596d070a8c0 msgr2=0x5596d0719210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.310+0000 7f5bbb7fe640 0 client.17469 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.310+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 shutdown_connections 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.310+0000 7f5be5599740 1 -- 172.21.15.69:0/4169979942 wait complete. 2024-05-09T09:36:57.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.310+0000 7f5be5599740 20 client.17469 trim_cache size 0 max 16384 2024-05-09T09:36:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:36:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:36:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:36:57.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 msgr2=0x7f5bc40a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 0x7f5bc40a5330 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f5bd0041000 tx=0x7f5bd003c000 comp rx=0 tx=0).stop 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 msgr2=0x7f5bdc1a6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f5bd40319e0 tx=0x7f5bd4030820 comp rx=0 tx=0).stop 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 shutdown_connections 2024-05-09T09:36:57.312 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5bc40a2e70 0x7f5bc40a5330 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5bdc10db50 0x7f5bdc1adfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5bdc10b1a0 0x7f5bdc1a6f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 --2- 172.21.15.69:0/373546183 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5bdc108830 0x7f5bdc1a6a00 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.311+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 >> 172.21.15.69:0/373546183 conn(0x7f5bdc06e8a0 msgr2=0x7f5bdc10d360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.312+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 shutdown_connections 2024-05-09T09:36:57.313 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:36:57.312+0000 7f5be27a9640 1 -- 172.21.15.69:0/373546183 wait complete. 2024-05-09T09:36:57.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 162163 29.19 MB/sec execute 175 sec latency 519.449 ms 2024-05-09T09:36:58.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 162390 29.10 MB/sec execute 176 sec latency 165.288 ms 2024-05-09T09:36:59.010 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:36:58 smithi069 ceph-mon[30123]: pgmap v912: 129 pgs: 129 active+clean; 698 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 482 op/s 2024-05-09T09:36:59.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:36:58 smithi163 ceph-mon[34354]: pgmap v912: 129 pgs: 129 active+clean; 698 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 482 op/s 2024-05-09T09:36:59.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:36:58 smithi103 ceph-mon[34421]: pgmap v912: 129 pgs: 129 active+clean; 698 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.6 MiB/s wr, 482 op/s 2024-05-09T09:36:59.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 162882 29.06 MB/sec execute 177 sec latency 111.409 ms 2024-05-09T09:37:00.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 163499 29.11 MB/sec execute 178 sec latency 71.886 ms 2024-05-09T09:37:01.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:00 smithi163 ceph-mon[34354]: pgmap v913: 129 pgs: 129 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.6 MiB/s wr, 494 op/s 2024-05-09T09:37:01.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:00 smithi103 ceph-mon[34421]: pgmap v913: 129 pgs: 129 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.6 MiB/s wr, 494 op/s 2024-05-09T09:37:01.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:00 smithi069 ceph-mon[30123]: pgmap v913: 129 pgs: 129 active+clean; 701 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 9.6 MiB/s wr, 494 op/s 2024-05-09T09:37:01.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 164258 29.15 MB/sec execute 179 sec latency 67.989 ms 2024-05-09T09:37:02.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 164733 29.16 MB/sec execute 180 sec latency 137.256 ms 2024-05-09T09:37:03.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:02 smithi163 ceph-mon[34354]: pgmap v914: 129 pgs: 129 active+clean; 649 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 757 op/s 2024-05-09T09:37:03.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:02 smithi103 ceph-mon[34421]: pgmap v914: 129 pgs: 129 active+clean; 649 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 757 op/s 2024-05-09T09:37:03.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:02 smithi069 ceph-mon[30123]: pgmap v914: 129 pgs: 129 active+clean; 649 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 757 op/s 2024-05-09T09:37:03.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 165293 29.18 MB/sec execute 181 sec latency 62.968 ms 2024-05-09T09:37:04.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 165927 29.16 MB/sec execute 182 sec latency 48.189 ms 2024-05-09T09:37:05.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:04 smithi163 ceph-mon[34354]: pgmap v915: 129 pgs: 129 active+clean; 637 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 717 op/s 2024-05-09T09:37:05.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:04 smithi103 ceph-mon[34421]: pgmap v915: 129 pgs: 129 active+clean; 637 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 717 op/s 2024-05-09T09:37:05.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:04 smithi069 ceph-mon[30123]: pgmap v915: 129 pgs: 129 active+clean; 637 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 12 MiB/s wr, 717 op/s 2024-05-09T09:37:05.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 166457 29.13 MB/sec execute 183 sec latency 80.277 ms 2024-05-09T09:37:06.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 167299 29.30 MB/sec execute 184 sec latency 87.466 ms 2024-05-09T09:37:07.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:06 smithi163 ceph-mon[34354]: pgmap v916: 129 pgs: 129 active+clean; 634 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 795 op/s 2024-05-09T09:37:07.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:06 smithi103 ceph-mon[34421]: pgmap v916: 129 pgs: 129 active+clean; 634 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 795 op/s 2024-05-09T09:37:07.432 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:06 smithi069 ceph-mon[30123]: pgmap v916: 129 pgs: 129 active+clean; 634 MiB data, 7.7 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 12 MiB/s wr, 795 op/s 2024-05-09T09:37:07.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 167876 29.26 MB/sec execute 185 sec latency 70.458 ms 2024-05-09T09:37:08.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 168437 29.31 MB/sec execute 186 sec latency 108.119 ms 2024-05-09T09:37:09.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:08 smithi163 ceph-mon[34354]: pgmap v917: 129 pgs: 129 active+clean; 643 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-09T09:37:09.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:08 smithi103 ceph-mon[34421]: pgmap v917: 129 pgs: 129 active+clean; 643 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-09T09:37:09.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:08 smithi069 ceph-mon[30123]: pgmap v917: 129 pgs: 129 active+clean; 643 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 801 op/s 2024-05-09T09:37:09.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 168903 29.30 MB/sec execute 187 sec latency 99.466 ms 2024-05-09T09:37:10.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 -- 172.21.15.69:0/3149957846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc104ce0 msgr2=0x7fc0cc1070d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3149957846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc1070d0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4009860 tx=0x7fc0b402a340 comp rx=0 tx=0).stop 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 -- 172.21.15.69:0/3149957846 shutdown_connections 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3149957846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc107610 0x7fc0cc109a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3149957846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc1070d0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3149957846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 -- 172.21.15.69:0/3149957846 >> 172.21.15.69:0/3149957846 conn(0x7fc0cc0fbd40 msgr2=0x7fc0cc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 -- 172.21.15.69:0/3149957846 shutdown_connections 2024-05-09T09:37:10.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.437+0000 7fc0d0970640 1 -- 172.21.15.69:0/3149957846 wait complete. 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 Processor -- start 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 -- start start 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc198050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc107610 0x7fc0cc19f0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:45474/0 (socket says 172.21.15.69:45474) 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3471175858 learned_addr learned my addr 172.21.15.69:0/3471175858 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d550 con 0x7fc0cc107610 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d3d0 con 0x7fc0cc1023b0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc10d6d0 con 0x7fc0cc104ce0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc198050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.438+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc107610 0x7fc0cc19f0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3471175858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc104ce0 msgr2=0x7fc0cc198050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc198050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc107610 msgr2=0x7fc0cc19f0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc107610 0x7fc0cc19f0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc0cc19f730 con 0x7fc0cc1023b0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00e9c0 tx=0x7fc0bc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0c37fe640 1 -- 172.21.15.69:0/3471175858 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc00ccc0 con 0x7fc0cc1023b0 2024-05-09T09:37:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc0cc19fa20 con 0x7fc0cc1023b0 2024-05-09T09:37:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0c37fe640 1 -- 172.21.15.69:0/3471175858 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0bc005500 con 0x7fc0cc1023b0 2024-05-09T09:37:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0c37fe640 1 -- 172.21.15.69:0/3471175858 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc010430 con 0x7fc0cc1023b0 2024-05-09T09:37:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.439+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc0cc19ff90 con 0x7fc0cc1023b0 2024-05-09T09:37:10.442 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.442+0000 7fc0c37fe640 1 -- 172.21.15.69:0/3471175858 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc0bc0046c0 con 0x7fc0cc1023b0 2024-05-09T09:37:10.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0c37fe640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 0x7fc0b80a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0c37fe640 1 -- 172.21.15.69:0/3471175858 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc0bc0cf690 con 0x7fc0cc1023b0 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 0x7fc0b80a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 Processor -- start 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 -- start start 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2c40 0x555c52bc7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.445 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555c52b26340 con 0x555c52bc7db0 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555c52ace290 con 0x555c52bc2c40 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.443+0000 7fc0d3681740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555c52b22300 con 0x555c52bc2250 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57562/0 (socket says 172.21.15.69:57562) 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3001293401 learned_addr learned my addr 172.21.15.69:0/3001293401 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34854/0 (socket says 172.21.15.69:34854) 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2c40 0x555c52bc7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 -- 172.21.15.69:0/3001293401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 msgr2=0x555c52bc2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 -- 172.21.15.69:0/3001293401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2c40 msgr2=0x555c52bc7740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2c40 0x555c52bc7740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.446 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 -- 172.21.15.69:0/3001293401 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c52af28e0 con 0x555c52bc7db0 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 0x7fc0b80a50f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4030410 tx=0x7fc0b4030550 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.444+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009f00 tx=0x7fc0c4011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d61183ddd78e243e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.445+0000 7fc0c0ff9640 1 -- 172.21.15.69:0/3001293401 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4004d20 con 0x555c52bc7db0 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.445+0000 7fc0c0ff9640 1 -- 172.21.15.69:0/3001293401 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0c4004030 con 0x555c52bc7db0 2024-05-09T09:37:10.447 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.445+0000 7fc0c0ff9640 1 -- 172.21.15.69:0/3001293401 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0c4005280 con 0x555c52bc7db0 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.445+0000 7fc0d3681740 1 -- 172.21.15.69:0/3001293401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 msgr2=0x555c52bca240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.445+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fc0c4009f00 tx=0x7fc0c4011810 comp rx=0 tx=0).stop 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 -- 172.21.15.69:0/3001293401 shutdown_connections 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc7db0 0x555c52bca240 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2c40 0x555c52bc7740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3001293401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2250 0x555c52bc2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 -- 172.21.15.69:0/3001293401 >> 172.21.15.69:0/3001293401 conn(0x555c52bbd8c0 msgr2=0x555c52bbdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 -- 172.21.15.69:0/3001293401 shutdown_connections 2024-05-09T09:37:10.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 -- 172.21.15.69:0/3001293401 wait complete. 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 -- start start 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.446+0000 7fc0d3681740 1 Processor -- start 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bcc920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52bd3960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555c52bcad60 con 0x555c52bc2250 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0c3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bcc920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34860/0 (socket says 172.21.15.69:34860) 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0c3fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bcc920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57574/0 (socket says 172.21.15.69:57574) 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 -- 172.21.15.69:0/4193549206 learned_addr learned my addr 172.21.15.69:0/4193549206 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555c52ae1790 con 0x555c52bc7db0 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555c52bcaee0 con 0x555c52bc2c40 2024-05-09T09:37:10.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52bd3960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0d3681740 10 client.0 _ll_register_callbacks cb 0x555c52b1ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 -- 172.21.15.69:0/4193549206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 msgr2=0x555c52bcc920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bcc920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 -- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 msgr2=0x555c52bd3960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52bd3960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c5283a210 con 0x555c52bc2250 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00cf30 tx=0x7fc0bc09c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d406f0e9d74ccb9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.447+0000 7fc0b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc0b9060 con 0x555c52bc2250 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc0bc0cf3b0 con 0x555c52bc2250 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0c4004030 con 0x555c52bc2250 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc0bc0a47e0 con 0x555c52bc2250 2024-05-09T09:37:10.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0d3681740 10 client.17481 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x555c52906620 con 0x555c52bc2250 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0c4004030 con 0x555c52bc2250 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.448+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc0bc005000 con 0x555c52bc2250 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 10 client.17481 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 10 client.17481 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 20 client.17481 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52bda320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 4 client.17481 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.449+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555c52b33a70 con 0x555c52bd7ed0 2024-05-09T09:37:10.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.450+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52bda320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.451+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52bda320 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=871d11596e4a2682 server_cookie=92fe9ba2473f86cc in_seq=0 out_seq=0 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.451+0000 7fc0b1ffb640 10 client.17481 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.453+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/4193549206 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555c52b33a70 con 0x555c52bd7ed0 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.453+0000 7fc0b1ffb640 10 client.17481 handle_command_reply: tid=0 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.453+0000 7fc0d3681740 1 client.17481 shutdown 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 msgr2=0x555c52bcc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00cf30 tx=0x7fc0bc09c330 comp rx=0 tx=0).stop 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 shutdown_connections 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52bd3960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52bda320 crc :-1 s=READY pgs=222 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bcc920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0b1ffb640 0 client.17481 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:10.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 --2- 172.21.15.69:0/4193549206 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bcc3e0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 >> 172.21.15.69:0/4193549206 conn(0x555c52bbd8c0 msgr2=0x555c52bbca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 shutdown_connections 2024-05-09T09:37:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 1 -- 172.21.15.69:0/4193549206 wait complete. 2024-05-09T09:37:10.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.454+0000 7fc0d3681740 20 client.17481 trim_cache size 0 max 16384 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 Processor -- start 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 -- start start 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bdfbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52be0120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1accf0 con 0x555c52bc2250 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1aa5c0 con 0x555c52bc7db0 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0d3681740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1a8410 con 0x555c52bc2c40 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34866/0 (socket says 172.21.15.69:34866) 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3449451316 learned_addr learned my addr 172.21.15.69:0/3449451316 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bdfbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52be0120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3449451316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 msgr2=0x555c52bdfbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bdfbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3449451316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 msgr2=0x555c52be0120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52be0120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0ca59c640 1 -- 172.21.15.69:0/3449451316 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555c52bbe630 con 0x555c52bc2250 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.478+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52be0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc09bb00 tx=0x7fc0bc09a080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69ea02095ce471e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0b3fff640 1 -- 172.21.15.69:0/3449451316 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc097be0 con 0x555c52bc2250 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0b3fff640 1 -- 172.21.15.69:0/3449451316 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0bc0cf3b0 con 0x555c52bc2250 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0b3fff640 1 -- 172.21.15.69:0/3449451316 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0bc005000 con 0x555c52bc2250 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 -- 172.21.15.69:0/3449451316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 msgr2=0x555c52bc0f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc09bb00 tx=0x7fc0bc09a080 comp rx=0 tx=0).stop 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 -- 172.21.15.69:0/3449451316 shutdown_connections 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc7db0 0x555c52be0120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc2c40 0x555c52bdfbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 --2- 172.21.15.69:0/3449451316 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2250 0x555c52bc0f80 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.479+0000 7fc0d3681740 1 -- 172.21.15.69:0/3449451316 >> 172.21.15.69:0/3449451316 conn(0x555c52bbd8c0 msgr2=0x555c52bc0870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- 172.21.15.69:0/3449451316 shutdown_connections 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- 172.21.15.69:0/3449451316 wait complete. 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- start start 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 Processor -- start 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 0x555c52c3a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc7db0 0x555c52c416b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1a8a80 con 0x555c52bc2c40 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1a9ef0 con 0x555c52bc2250 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc0cc1a9150 con 0x555c52bc7db0 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0d3681740 10 client.0 _ll_register_callbacks cb 0x7fc0bc0ccd50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:10.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0c3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0ca59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 0x555c52c3a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0c3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34880/0 (socket says 172.21.15.69:34880) 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0c3fff640 1 -- 172.21.15.69:0/475845856 learned_addr learned my addr 172.21.15.69:0/475845856 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.480+0000 7fc0cad9d640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc7db0 0x555c52c416b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 -- 172.21.15.69:0/475845856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc7db0 msgr2=0x555c52c416b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc7db0 0x555c52c416b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 -- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 msgr2=0x555c52c3a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 0x555c52c3a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 -- 172.21.15.69:0/475845856 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555c52c41ca0 con 0x555c52bc2c40 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 0x555c52c3a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0c3fff640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40095f0 tx=0x7fc0b402a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa6081e8426359ab server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc0b404ee60 con 0x555c52bc2c40 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc0b4005040 con 0x555c52bc2c40 2024-05-09T09:37:10.486 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc0bc0cf3b0 con 0x555c52bc2c40 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc0b405e050 con 0x555c52bc2c40 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0d3681740 10 client.17487 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x555c52c43050 con 0x555c52bc2c40 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc0bc0cf3b0 con 0x555c52bc2c40 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.481+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc0b404e160 con 0x555c52bc2c40 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 10 client.17487 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 10 client.17487 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 20 client.17487 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52c45b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 4 client.17487 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555c52b2ddc0 con 0x555c52bd7ed0 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52c45b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0ca59c640 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52c45b10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8665b196e347ae67 server_cookie=9d215af07d2fc7bd in_seq=0 out_seq=0 2024-05-09T09:37:10.487 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:10.482+0000 7fc0b1ffb640 10 client.17487 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:10.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 169298 29.21 MB/sec execute 188 sec latency 287.577 ms 2024-05-09T09:37:11.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0b1ffb640 1 -- 172.21.15.69:0/475845856 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555c52b2ddc0 con 0x555c52bd7ed0 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0b1ffb640 10 client.17487 handle_command_reply: tid=0 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 client.17487 shutdown 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 msgr2=0x555c52c3a670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7fc0b40095f0 tx=0x7fc0b402a340 comp rx=0 tx=0).stop 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 shutdown_connections 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555c52bd7ed0 0x555c52c45b10 crc :-1 s=READY pgs=223 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555c52bc7db0 0x555c52c416b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0b1ffb640 0 client.17487 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555c52bc2c40 0x555c52c3a670 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 --2- 172.21.15.69:0/475845856 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555c52bc2250 0x555c52c3a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.019+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 >> 172.21.15.69:0/475845856 conn(0x555c52bbd8c0 msgr2=0x555c52bbc170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 shutdown_connections 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d3681740 1 -- 172.21.15.69:0/475845856 wait complete. 2024-05-09T09:37:11.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d3681740 20 client.17487 trim_cache size 0 max 16384 2024-05-09T09:37:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:37:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:37:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:37:11.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 msgr2=0x7fc0b80a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 0x7fc0b80a50f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7fc0b4030410 tx=0x7fc0b4030550 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 msgr2=0x7fc0cc197b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.020+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc0bc00e9c0 tx=0x7fc0bc00ee80 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 shutdown_connections 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc0b80a2c30 0x7fc0b80a50f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc0cc107610 0x7fc0cc19f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc0cc104ce0 0x7fc0cc198050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 --2- 172.21.15.69:0/3471175858 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc0cc1023b0 0x7fc0cc197b10 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:11.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 >> 172.21.15.69:0/3471175858 conn(0x7fc0cc0fbd40 msgr2=0x7fc0cc102d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 shutdown_connections 2024-05-09T09:37:11.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:11.021+0000 7fc0d0970640 1 -- 172.21.15.69:0/3471175858 wait complete. 2024-05-09T09:37:11.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:10 smithi163 ceph-mon[34354]: pgmap v918: 129 pgs: 129 active+clean; 644 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-05-09T09:37:11.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:10 smithi103 ceph-mon[34421]: pgmap v918: 129 pgs: 129 active+clean; 644 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-05-09T09:37:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:10 smithi069 ceph-mon[30123]: pgmap v918: 129 pgs: 129 active+clean; 644 MiB data, 7.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.4 MiB/s wr, 646 op/s 2024-05-09T09:37:11.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 169861 29.20 MB/sec execute 189 sec latency 415.940 ms 2024-05-09T09:37:12.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 170535 29.25 MB/sec execute 190 sec latency 53.334 ms 2024-05-09T09:37:13.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:12 smithi163 ceph-mon[34354]: pgmap v919: 129 pgs: 129 active+clean; 644 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2024-05-09T09:37:13.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:12 smithi103 ceph-mon[34421]: pgmap v919: 129 pgs: 129 active+clean; 644 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2024-05-09T09:37:13.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:12 smithi069 ceph-mon[30123]: pgmap v919: 129 pgs: 129 active+clean; 644 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.8 MiB/s wr, 710 op/s 2024-05-09T09:37:13.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 171194 29.27 MB/sec execute 191 sec latency 58.835 ms 2024-05-09T09:37:14.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 171936 29.36 MB/sec execute 192 sec latency 46.159 ms 2024-05-09T09:37:15.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:14 smithi163 ceph-mon[34354]: pgmap v920: 129 pgs: 129 active+clean; 647 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.9 MiB/s wr, 415 op/s 2024-05-09T09:37:15.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:14 smithi103 ceph-mon[34421]: pgmap v920: 129 pgs: 129 active+clean; 647 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.9 MiB/s wr, 415 op/s 2024-05-09T09:37:15.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:14 smithi069 ceph-mon[30123]: pgmap v920: 129 pgs: 129 active+clean; 647 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.9 MiB/s wr, 415 op/s 2024-05-09T09:37:15.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 172547 29.40 MB/sec execute 193 sec latency 103.565 ms 2024-05-09T09:37:16.830 INFO:tasks.workunit.client.0.smithi069.stdout: 10 173025 29.37 MB/sec execute 194 sec latency 82.293 ms 2024-05-09T09:37:17.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:16 smithi163 ceph-mon[34354]: pgmap v921: 129 pgs: 129 active+clean; 651 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 504 op/s 2024-05-09T09:37:17.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:16 smithi103 ceph-mon[34421]: pgmap v921: 129 pgs: 129 active+clean; 651 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 504 op/s 2024-05-09T09:37:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:16 smithi069 ceph-mon[30123]: pgmap v921: 129 pgs: 129 active+clean; 651 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 504 op/s 2024-05-09T09:37:17.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 173504 29.36 MB/sec execute 195 sec latency 93.697 ms 2024-05-09T09:37:18.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 174066 29.32 MB/sec execute 196 sec latency 68.931 ms 2024-05-09T09:37:19.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:18 smithi163 ceph-mon[34354]: pgmap v922: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.8 MiB/s wr, 555 op/s 2024-05-09T09:37:19.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:18 smithi103 ceph-mon[34421]: pgmap v922: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.8 MiB/s wr, 555 op/s 2024-05-09T09:37:19.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:18 smithi069 ceph-mon[30123]: pgmap v922: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 9.0 MiB/s rd, 3.8 MiB/s wr, 555 op/s 2024-05-09T09:37:19.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 174897 29.37 MB/sec execute 197 sec latency 94.676 ms 2024-05-09T09:37:20.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 175700 29.49 MB/sec execute 198 sec latency 44.470 ms 2024-05-09T09:37:21.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:20 smithi163 ceph-mon[34354]: pgmap v923: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.9 MiB/s wr, 456 op/s 2024-05-09T09:37:21.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:20 smithi103 ceph-mon[34421]: pgmap v923: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.9 MiB/s wr, 456 op/s 2024-05-09T09:37:21.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:20 smithi069 ceph-mon[30123]: pgmap v923: 129 pgs: 129 active+clean; 662 MiB data, 7.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 2.9 MiB/s wr, 456 op/s 2024-05-09T09:37:21.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 176304 29.52 MB/sec execute 199 sec latency 82.246 ms 2024-05-09T09:37:22.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 176742 29.49 MB/sec execute 200 sec latency 103.949 ms 2024-05-09T09:37:23.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:22 smithi163 ceph-mon[34354]: pgmap v924: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.2 MiB/s wr, 607 op/s 2024-05-09T09:37:23.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:22 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:37:23.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:22 smithi103 ceph-mon[34421]: pgmap v924: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.2 MiB/s wr, 607 op/s 2024-05-09T09:37:23.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:22 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:37:23.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:22 smithi069 ceph-mon[30123]: pgmap v924: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.2 MiB/s wr, 607 op/s 2024-05-09T09:37:23.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:22 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:37:23.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 177357 29.49 MB/sec execute 201 sec latency 57.851 ms 2024-05-09T09:37:24.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:37:24.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:37:24.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:23 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:37:24.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:37:24.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:37:24.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:23 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:37:24.161 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:37:24.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:37:24.162 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:23 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:37:24.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.170+0000 7f966bc5d640 1 -- 172.21.15.69:0/1289304162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109040 msgr2=0x7f96641094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.170+0000 7f966bc5d640 1 --2- 172.21.15.69:0/1289304162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109040 0x7f96641094c0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9660009900 tx=0x7f966002a420 comp rx=0 tx=0).stop 2024-05-09T09:37:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.171+0000 7f966bc5d640 1 -- 172.21.15.69:0/1289304162 shutdown_connections 2024-05-09T09:37:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.171+0000 7f966bc5d640 1 --2- 172.21.15.69:0/1289304162 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9664109a00 0x7f9664110320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.171+0000 7f966bc5d640 1 --2- 172.21.15.69:0/1289304162 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109040 0x7f96641094c0 secure :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f9660009900 tx=0x7f966002a420 comp rx=0 tx=0).stop 2024-05-09T09:37:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.171+0000 7f966bc5d640 1 --2- 172.21.15.69:0/1289304162 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664108280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.171+0000 7f966bc5d640 1 -- 172.21.15.69:0/1289304162 >> 172.21.15.69:0/1289304162 conn(0x7f9664076e50 msgr2=0x7f9664079290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.173 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.173+0000 7f966bc5d640 1 -- 172.21.15.69:0/1289304162 shutdown_connections 2024-05-09T09:37:24.174 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.173+0000 7f966bc5d640 1 -- 172.21.15.69:0/1289304162 wait complete. 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.174+0000 7f966bc5d640 1 Processor -- start 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 -- start start 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664070370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 0x7f96640708b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96641ac390 0x7f96641ae780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9664113c90 con 0x7f9664109a00 2024-05-09T09:37:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f9664113b10 con 0x7f96641ac390 2024-05-09T09:37:24.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.178+0000 7f966bc5d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f9664113e10 con 0x7f9664107e60 2024-05-09T09:37:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.180+0000 7f96691f8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 0x7f96640708b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.180+0000 7f96691f8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 0x7f96640708b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51070/0 (socket says 172.21.15.69:51070) 2024-05-09T09:37:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.180+0000 7f96691f8640 1 -- 172.21.15.69:0/4107144355 learned_addr learned my addr 172.21.15.69:0/4107144355 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:24.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.181+0000 7f96699f9640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664070370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.181+0000 7f966a1fa640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96641ac390 0x7f96641ae780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.182+0000 7f96699f9640 1 -- 172.21.15.69:0/4107144355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96641ac390 msgr2=0x7f96641ae780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.182 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.182+0000 7f96699f9640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96641ac390 0x7f96641ae780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.182+0000 7f96699f9640 1 -- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 msgr2=0x7f96640708b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.182+0000 7f96699f9640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 0x7f96640708b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.182+0000 7f96699f9640 1 -- 172.21.15.69:0/4107144355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f966407a080 con 0x7f9664107e60 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.183+0000 7f96699f9640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664070370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f965400b3e0 tx=0x7f965400b8a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.184+0000 7f965affd640 1 -- 172.21.15.69:0/4107144355 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9654015020 con 0x7f9664107e60 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.184+0000 7f965affd640 1 -- 172.21.15.69:0/4107144355 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9654011d10 con 0x7f9664107e60 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.184+0000 7f965affd640 1 -- 172.21.15.69:0/4107144355 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f965400f2e0 con 0x7f9664107e60 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.184+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f966407a370 con 0x7f9664107e60 2024-05-09T09:37:24.185 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.184+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f966407a8c0 con 0x7f9664107e60 2024-05-09T09:37:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.188+0000 7f966e969740 1 Processor -- start 2024-05-09T09:37:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 -- start start 2024-05-09T09:37:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x558027400670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x558027405740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027408240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558027364340 con 0x558027400c40 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55802730c290 con 0x558027400250 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966e969740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558027360300 con 0x558027405db0 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f96699f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x558027405740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f965affd640 1 -- 172.21.15.69:0/4107144355 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f9654018020 con 0x7f9664107e60 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f96699f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x558027405740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51082/0 (socket says 172.21.15.69:51082) 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f96699f9640 1 -- 172.21.15.69:0/2690515125 learned_addr learned my addr 172.21.15.69:0/2690515125 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.190+0000 7f966a1fa640 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x558027400670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.192 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027408240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 -- 172.21.15.69:0/2690515125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 msgr2=0x558027400670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.193 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x558027400670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 -- 172.21.15.69:0/2690515125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 msgr2=0x558027405740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x558027405740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.191+0000 7f96691f8640 1 -- 172.21.15.69:0/2690515125 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580273308e0 con 0x558027405db0 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.192+0000 7f965affd640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 0x7f96500a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.192+0000 7f965affd640 1 -- 172.21.15.69:0/4107144355 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f96540cebc0 con 0x7f9664107e60 2024-05-09T09:37:24.194 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.192+0000 7f96691f8640 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027408240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f966002a990 tx=0x7f96600040f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=afdee87de76a7003 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.195+0000 7f964ffff640 1 -- 172.21.15.69:0/2690515125 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f966003f070 con 0x558027405db0 2024-05-09T09:37:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.195+0000 7f964ffff640 1 -- 172.21.15.69:0/2690515125 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660005500 con 0x558027405db0 2024-05-09T09:37:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.195+0000 7f964ffff640 1 -- 172.21.15.69:0/2690515125 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660035070 con 0x558027405db0 2024-05-09T09:37:24.196 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.196+0000 7f966a1fa640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 0x7f96500a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 -- 172.21.15.69:0/2690515125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 msgr2=0x558027408240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027408240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f966002a990 tx=0x7f96600040f0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966a1fa640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 0x7f96500a5240 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x5580274014a0 tx=0x7f965c00d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 -- 172.21.15.69:0/2690515125 shutdown_connections 2024-05-09T09:37:24.200 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027408240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x558027405740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 --2- 172.21.15.69:0/2690515125 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x558027400670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.198+0000 7f966e969740 1 -- 172.21.15.69:0/2690515125 >> 172.21.15.69:0/2690515125 conn(0x5580273fb8c0 msgr2=0x5580273fbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.200+0000 7f966e969740 1 -- 172.21.15.69:0/2690515125 shutdown_connections 2024-05-09T09:37:24.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.200+0000 7f966e969740 1 -- 172.21.15.69:0/2690515125 wait complete. 2024-05-09T09:37:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.200+0000 7f966e969740 1 -- start start 2024-05-09T09:37:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.200+0000 7f966e969740 1 Processor -- start 2024-05-09T09:37:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 0x55802740a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 0x55802740aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.204 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558027408d00 con 0x558027400250 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55802731f790 con 0x558027400c40 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558027408e80 con 0x558027405db0 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966e969740 10 client.0 _ll_register_callbacks cb 0x55802735aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96699f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 0x55802740a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96699f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 0x55802740a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:51090/0 (socket says 172.21.15.69:51090) 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96699f9640 1 -- 172.21.15.69:0/2926168024 learned_addr learned my addr 172.21.15.69:0/2926168024 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 0x55802740aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f966a1fa640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 -- 172.21.15.69:0/2926168024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 msgr2=0x558027411a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 -- 172.21.15.69:0/2926168024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 msgr2=0x55802740a4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 0x55802740a4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.201+0000 7f96691f8640 1 -- 172.21.15.69:0/2926168024 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558027078210 con 0x558027400c40 2024-05-09T09:37:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.203+0000 7f96691f8640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 0x55802740aa30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f96600049d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49b28777a8eea13c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.203+0000 7f964d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:37:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.204+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660035070 con 0x558027400c40 2024-05-09T09:37:24.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.204+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f9660005200 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.204+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9660005500 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.205+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f966003f040 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.205+0000 7f966e969740 10 client.26777 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.205+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558027144620 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660009520 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f9660003940 con 0x558027400c40 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 10 client.26777 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 10 client.26777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 20 client.26777 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558027415bd0 0x558027417fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 4 client.26777 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:37:24.208 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f9660005500 con 0x558027415bd0 2024-05-09T09:37:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.206+0000 7f96699f9640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558027415bd0 0x558027417fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.207+0000 7f964d7fa640 10 client.26777 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:24.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.207+0000 7f96699f9640 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558027415bd0 0x558027417fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5ae4e1ece5c6f957 server_cookie=558886e89681570a in_seq=0 out_seq=0 2024-05-09T09:37:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.210+0000 7f964d7fa640 1 -- 172.21.15.69:0/2926168024 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f9660005500 con 0x558027415bd0 2024-05-09T09:37:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.210+0000 7f964d7fa640 10 client.26777 handle_command_reply: tid=0 2024-05-09T09:37:24.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.210+0000 7f966e969740 1 client.26777 shutdown 2024-05-09T09:37:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.210+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 msgr2=0x55802740aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.210+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 0x55802740aa30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9660006010 tx=0x7f96600049d0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 shutdown_connections 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558027415bd0 0x558027417fc0 crc :-1 s=READY pgs=224 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400c40 0x55802740aa30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 --2- 172.21.15.69:0/2926168024 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400250 0x55802740a4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.212+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 >> 172.21.15.69:0/2926168024 conn(0x5580273fb8c0 msgr2=0x5580273faa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.213+0000 7f964d7fa640 0 client.26777 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.213+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 shutdown_connections 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.213+0000 7f966e969740 1 -- 172.21.15.69:0/2926168024 wait complete. 2024-05-09T09:37:24.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.214+0000 7f966e969740 20 client.26777 trim_cache size 0 max 16384 2024-05-09T09:37:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.240+0000 7f966e969740 1 Processor -- start 2024-05-09T09:37:24.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 -- start start 2024-05-09T09:37:24.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x55802741d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802741dce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f966419e180 con 0x558027400c40 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f96699f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x55802741d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f96691f8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802741dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f96699f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x55802741d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:45388/0 (socket says 172.21.15.69:45388) 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f96699f9640 1 -- 172.21.15.69:0/557626722 learned_addr learned my addr 172.21.15.69:0/557626722 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f966419ba50 con 0x558027400250 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.241+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f96640824a0 con 0x558027405db0 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f966a1fa640 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 -- 172.21.15.69:0/557626722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 msgr2=0x558027411a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 -- 172.21.15.69:0/557626722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 msgr2=0x55802741d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x55802741d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 -- 172.21.15.69:0/557626722 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5580273f9f00 con 0x558027400c40 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f96691f8640 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802741dce0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9660003d10 tx=0x7f9660000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe090eae9125a24d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f964f7fe640 1 -- 172.21.15.69:0/557626722 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f9660035070 con 0x558027400c40 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f964f7fe640 1 -- 172.21.15.69:0/557626722 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660009520 con 0x558027400c40 2024-05-09T09:37:24.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.242+0000 7f964f7fe640 1 -- 172.21.15.69:0/557626722 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96600316b0 con 0x558027400c40 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 msgr2=0x55802741dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802741dce0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7f9660003d10 tx=0x7f9660000bc0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 shutdown_connections 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027405db0 0x558027411a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802741dce0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.243+0000 7f966e969740 1 --2- 172.21.15.69:0/557626722 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027400250 0x55802741d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.244+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 >> 172.21.15.69:0/557626722 conn(0x5580273fb8c0 msgr2=0x5580273fbca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.244+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 shutdown_connections 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.244+0000 7f966e969740 1 -- 172.21.15.69:0/557626722 wait complete. 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.244+0000 7f966e969740 1 -- start start 2024-05-09T09:37:24.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.244+0000 7f966e969740 1 Processor -- start 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802740ec40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027405db0 0x55802747a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f9664199f10 con 0x558027400c40 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f966419b380 con 0x558027405db0 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f966419a5e0 con 0x558027400250 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57702/0 (socket says 172.21.15.69:57702) 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966e969740 10 client.0 _ll_register_callbacks cb 0x7f96540d2450 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 -- 172.21.15.69:0/398927102 learned_addr learned my addr 172.21.15.69:0/398927102 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96691f8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802740ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f966a1fa640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027405db0 0x55802747a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 -- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027405db0 msgr2=0x55802747a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027405db0 0x55802747a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 -- 172.21.15.69:0/398927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 msgr2=0x55802740ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802740ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.245+0000 7f96699f9640 1 -- 172.21.15.69:0/398927102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55802740f250 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.246+0000 7f96699f9640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f96540974a0 tx=0x7f96540d2580 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8cfd7a69f6ebfcce server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.246+0000 7f964d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.246+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f96540d3040 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.246+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f9660009520 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.246+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f96540bb0b0 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.247+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f965409b040 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.247+0000 7f966e969740 10 client.26707 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.247+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55802747b6a0 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f9660009520 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f96540b7070 con 0x558027400250 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 10 client.26707 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:24.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 10 client.26707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 20 client.26707 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55802747c000 0x55802747e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 4 client.26707 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558027371a70 con 0x55802747c000 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.248+0000 7f96691f8640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55802747c000 0x55802747e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.249+0000 7f96691f8640 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55802747c000 0x55802747e420 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b73c7db6ebdd83b server_cookie=dda28ca13ef6bf04 in_seq=0 out_seq=0 2024-05-09T09:37:24.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.249+0000 7f964d7fa640 10 client.26707 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:24.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 177585 29.41 MB/sec execute 202 sec latency 431.546 ms 2024-05-09T09:37:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f964d7fa640 1 -- 172.21.15.69:0/398927102 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558027371a70 con 0x55802747c000 2024-05-09T09:37:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f964d7fa640 10 client.26707 handle_command_reply: tid=0 2024-05-09T09:37:24.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 client.26707 shutdown 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 msgr2=0x55802740e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f96540974a0 tx=0x7f96540d2580 comp rx=0 tx=0).stop 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 shutdown_connections 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558027405db0 0x55802747a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558027400c40 0x55802740ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55802747c000 0x55802747e420 crc :-1 s=READY pgs=225 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 --2- 172.21.15.69:0/398927102 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558027400250 0x55802740e700 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f964d7fa640 0 client.26707 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 >> 172.21.15.69:0/398927102 conn(0x5580273fb8c0 msgr2=0x558027417be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 shutdown_connections 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.961+0000 7f966e969740 1 -- 172.21.15.69:0/398927102 wait complete. 2024-05-09T09:37:24.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966e969740 20 client.26707 trim_cache size 0 max 16384 2024-05-09T09:37:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:37:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:37:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:37:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:37:24.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 msgr2=0x7f96500a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 0x7f96500a5240 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x5580274014a0 tx=0x7f965c00d040 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 msgr2=0x7f9664070370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664070370 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f965400b3e0 tx=0x7f965400b8a0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 shutdown_connections 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f96500a2d80 0x7f96500a5240 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f96641ac390 0x7f96641ae780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9664109a00 0x7f96640708b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 --2- 172.21.15.69:0/4107144355 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9664107e60 0x7f9664070370 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 >> 172.21.15.69:0/4107144355 conn(0x7f9664076e50 msgr2=0x7f966410c450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.962+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 shutdown_connections 2024-05-09T09:37:24.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:24.963+0000 7f966bc5d640 1 -- 172.21.15.69:0/4107144355 wait complete. 2024-05-09T09:37:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:25 smithi163 ceph-mon[34354]: pgmap v925: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-09T09:37:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:25 smithi103 ceph-mon[34421]: pgmap v925: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-09T09:37:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:25 smithi069 ceph-mon[30123]: pgmap v925: 129 pgs: 129 active+clean; 676 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 479 op/s 2024-05-09T09:37:25.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 178221 29.42 MB/sec execute 203 sec latency 505.222 ms 2024-05-09T09:37:26.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 178791 29.42 MB/sec execute 204 sec latency 53.353 ms 2024-05-09T09:37:27.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:27 smithi163 ceph-mon[34354]: pgmap v926: 129 pgs: 129 active+clean; 671 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2024-05-09T09:37:27.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:27 smithi103 ceph-mon[34421]: pgmap v926: 129 pgs: 129 active+clean; 671 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2024-05-09T09:37:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:27 smithi069 ceph-mon[30123]: pgmap v926: 129 pgs: 129 active+clean; 671 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.5 MiB/s wr, 517 op/s 2024-05-09T09:37:27.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 179381 29.45 MB/sec execute 205 sec latency 77.820 ms 2024-05-09T09:37:28.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 179978 29.45 MB/sec execute 206 sec latency 143.884 ms 2024-05-09T09:37:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:29 smithi163 ceph-mon[34354]: pgmap v927: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.2 MiB/s wr, 569 op/s 2024-05-09T09:37:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:29 smithi103 ceph-mon[34421]: pgmap v927: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.2 MiB/s wr, 569 op/s 2024-05-09T09:37:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:29 smithi069 ceph-mon[30123]: pgmap v927: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.2 MiB/s wr, 569 op/s 2024-05-09T09:37:29.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 180544 29.48 MB/sec execute 207 sec latency 85.718 ms 2024-05-09T09:37:30.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 180750 29.37 MB/sec execute 208 sec latency 679.992 ms 2024-05-09T09:37:31.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:31 smithi103 ceph-mon[34421]: pgmap v928: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.1 MiB/s wr, 454 op/s 2024-05-09T09:37:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:31 smithi069 ceph-mon[30123]: pgmap v928: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.1 MiB/s wr, 454 op/s 2024-05-09T09:37:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:31 smithi163 ceph-mon[34354]: pgmap v928: 129 pgs: 129 active+clean; 669 MiB data, 8.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.1 MiB/s wr, 454 op/s 2024-05-09T09:37:31.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 180751 29.23 MB/sec execute 209 sec latency 1679.993 ms 2024-05-09T09:37:32.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 180842 29.12 MB/sec execute 210 sec latency 2191.125 ms 2024-05-09T09:37:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:33 smithi103 ceph-mon[34421]: pgmap v929: 129 pgs: 129 active+clean; 681 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.6 MiB/s wr, 530 op/s 2024-05-09T09:37:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:33 smithi069 ceph-mon[30123]: pgmap v929: 129 pgs: 129 active+clean; 681 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.6 MiB/s wr, 530 op/s 2024-05-09T09:37:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:33 smithi163 ceph-mon[34354]: pgmap v929: 129 pgs: 129 active+clean; 681 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 4.6 MiB/s wr, 530 op/s 2024-05-09T09:37:33.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 181051 29.03 MB/sec execute 211 sec latency 666.517 ms 2024-05-09T09:37:34.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 181368 29.00 MB/sec execute 212 sec latency 105.244 ms 2024-05-09T09:37:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:35 smithi103 ceph-mon[34421]: pgmap v930: 129 pgs: 129 active+clean; 726 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 514 op/s 2024-05-09T09:37:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:35 smithi069 ceph-mon[30123]: pgmap v930: 129 pgs: 129 active+clean; 726 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 514 op/s 2024-05-09T09:37:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:35 smithi163 ceph-mon[34354]: pgmap v930: 129 pgs: 129 active+clean; 726 MiB data, 8.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 7.5 MiB/s wr, 514 op/s 2024-05-09T09:37:35.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 181590 28.90 MB/sec execute 213 sec latency 88.055 ms 2024-05-09T09:37:36.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 181879 28.85 MB/sec execute 214 sec latency 135.853 ms 2024-05-09T09:37:37.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:37 smithi103 ceph-mon[34421]: pgmap v931: 129 pgs: 129 active+clean; 760 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 13 MiB/s wr, 661 op/s 2024-05-09T09:37:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:37 smithi069 ceph-mon[30123]: pgmap v931: 129 pgs: 129 active+clean; 760 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 13 MiB/s wr, 661 op/s 2024-05-09T09:37:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:37 smithi163 ceph-mon[34354]: pgmap v931: 129 pgs: 129 active+clean; 760 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 13 MiB/s wr, 661 op/s 2024-05-09T09:37:37.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 182159 28.78 MB/sec execute 215 sec latency 129.193 ms 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1117391725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 msgr2=0x7fa9f8108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1117391725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f8108bd0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009ca0 tx=0x7fa9f002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1117391725 shutdown_connections 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1117391725 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f8108bd0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1117391725 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1117391725 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1117391725 >> 172.21.15.69:0/1117391725 conn(0x7fa9f80fbce0 msgr2=0x7fa9f80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1117391725 shutdown_connections 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.086+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1117391725 wait complete. 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 Processor -- start 2024-05-09T09:37:38.087 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 -- start start 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f81032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810c460 con 0x7fa9f8102340 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810c2e0 con 0x7fa9f8101980 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fec9a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f810c5e0 con 0x7fa9f8100790 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fca36640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fca36640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:35874/0 (socket says 172.21.15.69:35874) 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fca36640 1 -- 172.21.15.69:0/1210793750 learned_addr learned my addr 172.21.15.69:0/1210793750 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9fd237640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f81032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.088 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.087+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 -- 172.21.15.69:0/1210793750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 msgr2=0x7fa9f8102d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8102d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 -- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 msgr2=0x7fa9f81032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f81032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 -- 172.21.15.69:0/1210793750 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9f8103b30 con 0x7fa9f8100790 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.088+0000 7fa9fca36640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00e9c0 tx=0x7fa9ec00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.089+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/1210793750 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec00cd80 con 0x7fa9f8100790 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.089+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/1210793750 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9ec0035f0 con 0x7fa9f8100790 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.089+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/1210793750 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9ec010690 con 0x7fa9f8100790 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.089+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9f81a1bd0 con 0x7fa9f8100790 2024-05-09T09:37:38.090 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.089+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9f81a20d0 con 0x7fa9f8100790 2024-05-09T09:37:38.091 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.092+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/1210793750 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa9ec0037a0 con 0x7fa9f8100790 2024-05-09T09:37:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.092+0000 7faa019a6740 1 Processor -- start 2024-05-09T09:37:38.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 -- start start 2024-05-09T09:37:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6250 0x55b5ac3a6670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f5ffb640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 0x7fa9e80a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3ae240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac30a340 con 0x55b5ac3a6c40 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac2b2290 con 0x55b5ac3a6250 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7faa019a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac306300 con 0x55b5ac3abdb0 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44398/0 (socket says 172.21.15.69:44398) 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 -- 172.21.15.69:0/251688323 learned_addr learned my addr 172.21.15.69:0/251688323 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9fd237640 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3ae240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9fca36640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 0x7fa9e80a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f5ffb640 1 -- 172.21.15.69:0/1210793750 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9ec0cf6e0 con 0x7fa9f8100790 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 -- 172.21.15.69:0/251688323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 msgr2=0x55b5ac3ae240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3ae240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 -- 172.21.15.69:0/251688323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6250 msgr2=0x55b5ac3a6670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.093+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6250 0x55b5ac3a6670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9f7fff640 1 -- 172.21.15.69:0/251688323 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ac2d68e0 con 0x55b5ac3a6c40 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9fca36640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 0x7fa9e80a5260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0a42a0 tx=0x7fa9ec003e10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4011470 tx=0x7fa9e4011930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3e0ad8417c00d91 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9e2ffd640 1 -- 172.21.15.69:0/251688323 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e401e070 con 0x55b5ac3a6c40 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9e2ffd640 1 -- 172.21.15.69:0/251688323 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e40035f0 con 0x55b5ac3a6c40 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.094+0000 7fa9e2ffd640 1 -- 172.21.15.69:0/251688323 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9e400f740 con 0x55b5ac3a6c40 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- 172.21.15.69:0/251688323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 msgr2=0x55b5ac3ab740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fa9e4011470 tx=0x7fa9e4011930 comp rx=0 tx=0).stop 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- 172.21.15.69:0/251688323 shutdown_connections 2024-05-09T09:37:38.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3ae240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3ab740 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 --2- 172.21.15.69:0/251688323 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6250 0x55b5ac3a6670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- 172.21.15.69:0/251688323 >> 172.21.15.69:0/251688323 conn(0x55b5ac3a18c0 msgr2=0x55b5ac3a1cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- 172.21.15.69:0/251688323 shutdown_connections 2024-05-09T09:37:38.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- 172.21.15.69:0/251688323 wait complete. 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.095+0000 7faa019a6740 1 -- start start 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.096+0000 7faa019a6740 1 Processor -- start 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.096+0000 7faa019a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.096+0000 7faa019a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b09d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.096+0000 7faa019a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7faa019a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac3aed60 con 0x55b5ac3a6250 2024-05-09T09:37:38.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7faa019a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac2c5790 con 0x55b5ac3abdb0 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7faa019a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b5ac3aeee0 con 0x55b5ac3a6c40 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7faa019a6740 10 client.0 _ll_register_callbacks cb 0x55b5ac300a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44406/0 (socket says 172.21.15.69:44406) 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 -- 172.21.15.69:0/3420285814 learned_addr learned my addr 172.21.15.69:0/3420285814 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.230 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 -- 172.21.15.69:0/3420285814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6c40 msgr2=0x55b5ac3b09d0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b09d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 -- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3abdb0 msgr2=0x55b5ac3b79a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b79a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.097+0000 7fa9fd237640 1 -- 172.21.15.69:0/3420285814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ac01e210 con 0x55b5ac3a6250 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7fa9fd237640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009430 tx=0x7fa9f003f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d5eec8cd07e509a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7fa9d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0004b40 con 0x55b5ac3a6250 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9e40035f0 con 0x55b5ac3a6250 2024-05-09T09:37:38.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9f0003970 con 0x55b5ac3a6250 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa9f0053050 con 0x55b5ac3a6250 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7faa019a6740 10 client.17517 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.098+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b5ac0ea620 con 0x55b5ac3a6250 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9e40035f0 con 0x55b5ac3a6250 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa9f002f610 con 0x55b5ac3a6250 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 10 client.17517 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 10 client.17517 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 20 client.17517 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac3bbf60 0x55b5ac3be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 4 client.17517 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b5ac317a70 con 0x55b5ac3bbf60 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.099+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac3bbf60 0x55b5ac3be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.100+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac3bbf60 0x55b5ac3be370 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3efbbb01057696d7 server_cookie=77c68a4c382ae2b8 in_seq=0 out_seq=0 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.100+0000 7fa9d7fff640 10 client.17517 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7fa9d7fff640 1 -- 172.21.15.69:0/3420285814 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b5ac317a70 con 0x55b5ac3bbf60 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7fa9d7fff640 10 client.17517 handle_command_reply: tid=0 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 client.17517 shutdown 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 msgr2=0x55b5ac3b0490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0009430 tx=0x7fa9f003f040 comp rx=0 tx=0).stop 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 shutdown_connections 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b79a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac3bbf60 0x55b5ac3be370 crc :-1 s=READY pgs=226 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b09d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7fa9d7fff640 0 client.17517 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 --2- 172.21.15.69:0/3420285814 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b0490 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 >> 172.21.15.69:0/3420285814 conn(0x55b5ac3a18c0 msgr2=0x55b5ac3a0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 shutdown_connections 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 1 -- 172.21.15.69:0/3420285814 wait complete. 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.102+0000 7faa019a6740 20 client.17517 trim_cache size 0 max 16384 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 Processor -- start 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 -- start start 2024-05-09T09:37:38.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3c3c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81aee40 con 0x55b5ac3abdb0 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81ac710 con 0x55b5ac3a6c40 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7faa019a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81aa560 con 0x55b5ac3a6250 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3c3c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:35890/0 (socket says 172.21.15.69:35890) 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 -- 172.21.15.69:0/48693246 learned_addr learned my addr 172.21.15.69:0/48693246 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 -- 172.21.15.69:0/48693246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 msgr2=0x55b5ac3b7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 -- 172.21.15.69:0/48693246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3abdb0 msgr2=0x55b5ac3c3c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3c3c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.120+0000 7fa9fd237640 1 -- 172.21.15.69:0/48693246 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b5ac39ff00 con 0x55b5ac3a6250 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.121+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.121+0000 7fa9fd237640 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0031870 tx=0x7fa9f0000f40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93a6b97a7f1dafe8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.121+0000 7fa9e27fc640 1 -- 172.21.15.69:0/48693246 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f002f610 con 0x55b5ac3a6250 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.121+0000 7fa9e27fc640 1 -- 172.21.15.69:0/48693246 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9f0036030 con 0x55b5ac3a6250 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.121+0000 7fa9e27fc640 1 -- 172.21.15.69:0/48693246 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0004280 con 0x55b5ac3a6250 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- 172.21.15.69:0/48693246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 msgr2=0x55b5ac3afab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0031870 tx=0x7fa9f0000f40 comp rx=0 tx=0).stop 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- 172.21.15.69:0/48693246 shutdown_connections 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3c3c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 --2- 172.21.15.69:0/48693246 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3a6250 0x55b5ac3afab0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- 172.21.15.69:0/48693246 >> 172.21.15.69:0/48693246 conn(0x55b5ac3a18c0 msgr2=0x55b5ac3ab490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- 172.21.15.69:0/48693246 shutdown_connections 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- 172.21.15.69:0/48693246 wait complete. 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 -- start start 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 Processor -- start 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.122+0000 7faa019a6740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b5a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b6a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81aabd0 con 0x55b5ac3a6250 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81ac040 con 0x55b5ac3a6c40 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9f81ab2a0 con 0x55b5ac3abdb0 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 10 client.0 _ll_register_callbacks cb 0x7fa9f0009b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44428/0 (socket says 172.21.15.69:44428) 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 -- 172.21.15.69:0/641925997 learned_addr learned my addr 172.21.15.69:0/641925997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b5a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b6a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 -- 172.21.15.69:0/641925997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 msgr2=0x55b5ac3b6a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b6a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 -- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 msgr2=0x55b5ac3b5a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b5a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 -- 172.21.15.69:0/641925997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b5ac3b7040 con 0x55b5ac3a6250 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9fd237640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0030040 tx=0x7fa9f002a8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d57e972c7df6e3d8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9d7fff640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9f0004d50 con 0x55b5ac3a6250 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9f0003970 con 0x55b5ac3a6250 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa9f0036030 con 0x55b5ac3a6250 2024-05-09T09:37:38.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.123+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa9f006a050 con 0x55b5ac3a6250 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 10 client.17523 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b5ac422980 con 0x55b5ac3a6250 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa9f0036030 con 0x55b5ac3a6250 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa9f004c4e0 con 0x55b5ac3a6250 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 10 client.17523 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 10 client.17523 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 20 client.17523 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac423240 0x55b5ac425660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 4 client.17523 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b5ac311dc0 con 0x55b5ac423240 2024-05-09T09:37:38.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.124+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac423240 0x55b5ac425660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.125+0000 7fa9f7fff640 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac423240 0x55b5ac425660 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7c3ab505f39bebd server_cookie=8532184622e92a67 in_seq=0 out_seq=0 2024-05-09T09:37:38.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.125+0000 7fa9d7fff640 10 client.17523 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7fa9d7fff640 1 -- 172.21.15.69:0/641925997 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b5ac311dc0 con 0x55b5ac423240 2024-05-09T09:37:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7fa9d7fff640 10 client.17523 handle_command_reply: tid=0 2024-05-09T09:37:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 client.17523 shutdown 2024-05-09T09:37:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 msgr2=0x55b5ac3b5500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fa9f0030040 tx=0x7fa9f002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 shutdown_connections 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b5ac423240 0x55b5ac425660 crc :-1 s=READY pgs=227 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b5ac3abdb0 0x55b5ac3b6a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7fa9d7fff640 0 client.17523 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b5ac3a6c40 0x55b5ac3b5a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 --2- 172.21.15.69:0/641925997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b5ac3a6250 0x55b5ac3b5500 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 >> 172.21.15.69:0/641925997 conn(0x55b5ac3a18c0 msgr2=0x55b5ac3b00a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 shutdown_connections 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 1 -- 172.21.15.69:0/641925997 wait complete. 2024-05-09T09:37:38.586 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.585+0000 7faa019a6740 20 client.17523 trim_cache size 0 max 16384 2024-05-09T09:37:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:37:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:37:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:37:38.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:37:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 msgr2=0x7fa9e80a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 0x7fa9e80a5260 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec0a42a0 tx=0x7fa9ec003e10 comp rx=0 tx=0).stop 2024-05-09T09:37:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 msgr2=0x7fa9f8104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7fa9ec00e9c0 tx=0x7fa9ec00ee80 comp rx=0 tx=0).stop 2024-05-09T09:37:38.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 shutdown_connections 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9e80a2da0 0x7fa9e80a5260 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9f8102340 0x7fa9f81032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9f8101980 0x7fa9f8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 --2- 172.21.15.69:0/1210793750 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9f8100790 0x7fa9f8104720 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 >> 172.21.15.69:0/1210793750 conn(0x7fa9f80fbce0 msgr2=0x7fa9f80fd670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 shutdown_connections 2024-05-09T09:37:38.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:38.586+0000 7fa9fec9a640 1 -- 172.21.15.69:0/1210793750 wait complete. 2024-05-09T09:37:38.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 182540 28.72 MB/sec execute 216 sec latency 389.597 ms 2024-05-09T09:37:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:39 smithi103 ceph-mon[34421]: pgmap v932: 129 pgs: 129 active+clean; 720 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 16 MiB/s wr, 806 op/s 2024-05-09T09:37:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:39 smithi069 ceph-mon[30123]: pgmap v932: 129 pgs: 129 active+clean; 720 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 16 MiB/s wr, 806 op/s 2024-05-09T09:37:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:39 smithi163 ceph-mon[34354]: pgmap v932: 129 pgs: 129 active+clean; 720 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 16 MiB/s wr, 806 op/s 2024-05-09T09:37:39.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 183142 28.71 MB/sec execute 217 sec latency 60.089 ms 2024-05-09T09:37:40.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 183897 28.77 MB/sec execute 218 sec latency 58.504 ms 2024-05-09T09:37:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:41 smithi103 ceph-mon[34421]: pgmap v933: 129 pgs: 129 active+clean; 704 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 706 op/s 2024-05-09T09:37:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:41 smithi069 ceph-mon[30123]: pgmap v933: 129 pgs: 129 active+clean; 704 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 706 op/s 2024-05-09T09:37:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:41 smithi163 ceph-mon[34354]: pgmap v933: 129 pgs: 129 active+clean; 704 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 15 MiB/s wr, 706 op/s 2024-05-09T09:37:41.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 184794 28.91 MB/sec execute 219 sec latency 38.723 ms 2024-05-09T09:37:42.831 INFO:tasks.workunit.client.0.smithi069.stdout: 10 185317 28.93 MB/sec execute 220 sec latency 254.864 ms 2024-05-09T09:37:43.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:43 smithi103 ceph-mon[34421]: pgmap v934: 129 pgs: 129 active+clean; 675 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 839 op/s 2024-05-09T09:37:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:43 smithi069 ceph-mon[30123]: pgmap v934: 129 pgs: 129 active+clean; 675 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 839 op/s 2024-05-09T09:37:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:43 smithi163 ceph-mon[34354]: pgmap v934: 129 pgs: 129 active+clean; 675 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 16 MiB/s wr, 839 op/s 2024-05-09T09:37:43.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 185644 28.85 MB/sec execute 221 sec latency 110.648 ms 2024-05-09T09:37:44.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 186115 28.83 MB/sec execute 222 sec latency 101.086 ms 2024-05-09T09:37:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:45 smithi103 ceph-mon[34421]: pgmap v935: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 15 MiB/s wr, 824 op/s 2024-05-09T09:37:45.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:45 smithi069 ceph-mon[30123]: pgmap v935: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 15 MiB/s wr, 824 op/s 2024-05-09T09:37:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:45 smithi163 ceph-mon[34354]: pgmap v935: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 15 MiB/s wr, 824 op/s 2024-05-09T09:37:45.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 186840 28.82 MB/sec execute 223 sec latency 126.940 ms 2024-05-09T09:37:46.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 187665 28.89 MB/sec execute 224 sec latency 106.629 ms 2024-05-09T09:37:47.357 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:47 smithi163 ceph-mon[34354]: pgmap v936: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-05-09T09:37:47.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:47 smithi103 ceph-mon[34421]: pgmap v936: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-05-09T09:37:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:47 smithi069 ceph-mon[30123]: pgmap v936: 129 pgs: 129 active+clean; 656 MiB data, 8.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 11 MiB/s wr, 781 op/s 2024-05-09T09:37:47.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 188462 29.05 MB/sec execute 225 sec latency 51.145 ms 2024-05-09T09:37:48.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 188918 29.02 MB/sec execute 226 sec latency 94.005 ms 2024-05-09T09:37:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:49 smithi103 ceph-mon[34421]: pgmap v937: 129 pgs: 129 active+clean; 658 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.5 MiB/s wr, 705 op/s 2024-05-09T09:37:49.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:49 smithi069 ceph-mon[30123]: pgmap v937: 129 pgs: 129 active+clean; 658 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.5 MiB/s wr, 705 op/s 2024-05-09T09:37:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:49 smithi163 ceph-mon[34354]: pgmap v937: 129 pgs: 129 active+clean; 658 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.5 MiB/s wr, 705 op/s 2024-05-09T09:37:49.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 189292 28.95 MB/sec execute 227 sec latency 63.707 ms 2024-05-09T09:37:50.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 189771 28.91 MB/sec execute 228 sec latency 90.050 ms 2024-05-09T09:37:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:51 smithi103 ceph-mon[34421]: pgmap v938: 129 pgs: 129 active+clean; 661 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.1 MiB/s wr, 520 op/s 2024-05-09T09:37:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:51 smithi069 ceph-mon[30123]: pgmap v938: 129 pgs: 129 active+clean; 661 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.1 MiB/s wr, 520 op/s 2024-05-09T09:37:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:51 smithi163 ceph-mon[34354]: pgmap v938: 129 pgs: 129 active+clean; 661 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.1 MiB/s wr, 520 op/s 2024-05-09T09:37:51.685 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 -- 172.21.15.69:0/3164518944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c100790 msgr2=0x7fc41c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3164518944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c100790 0x7fc41c100bb0 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009900 tx=0x7fc40c02a410 comp rx=0 tx=0).stop 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 -- 172.21.15.69:0/3164518944 shutdown_connections 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3164518944 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c102340 0x7fc41c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3164518944 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c101980 0x7fc41c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3164518944 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c100790 0x7fc41c100bb0 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 -- 172.21.15.69:0/3164518944 >> 172.21.15.69:0/3164518944 conn(0x7fc41c0fbd00 msgr2=0x7fc41c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 -- 172.21.15.69:0/3164518944 shutdown_connections 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 -- 172.21.15.69:0/3164518944 wait complete. 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.684+0000 7fc421cdf640 1 Processor -- start 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 -- start start 2024-05-09T09:37:51.686 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 0x7fc41c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c102340 0x7fc41c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c10d530 con 0x7fc41c101980 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c10d3b0 con 0x7fc41c102340 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc421cdf640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c10d6b0 con 0x7fc41c100790 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:52992/0 (socket says 172.21.15.69:52992) 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 -- 172.21.15.69:0/3784286390 learned_addr learned my addr 172.21.15.69:0/3784286390 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 0x7fc41c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41bfff640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c102340 0x7fc41c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 -- 172.21.15.69:0/3784286390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 msgr2=0x7fc41c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 0x7fc41c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 -- 172.21.15.69:0/3784286390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c102340 msgr2=0x7fc41c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.687 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c102340 0x7fc41c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 -- 172.21.15.69:0/3784286390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fc41c19f750 con 0x7fc41c101980 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 0x7fc41c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.685+0000 7fc41affd640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e9f0 tx=0x7fc41000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.686+0000 7fc418ff9640 1 -- 172.21.15.69:0/3784286390 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41000cd80 con 0x7fc41c101980 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.686+0000 7fc418ff9640 1 -- 172.21.15.69:0/3784286390 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc410010910 con 0x7fc41c101980 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.686+0000 7fc418ff9640 1 -- 172.21.15.69:0/3784286390 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc410018990 con 0x7fc41c101980 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.686+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fc41c19fa40 con 0x7fc41c101980 2024-05-09T09:37:51.688 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.686+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fc41c19ff90 con 0x7fc41c101980 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.688+0000 7fc418ff9640 1 -- 172.21.15.69:0/3784286390 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fc410010ac0 con 0x7fc41c101980 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 Processor -- start 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 -- start start 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b261ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b2620b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc418ff9640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 0x7fc4040a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2623650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d1b257b340 con 0x55d1b26211c0 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1b2523290 con 0x55d1b261b660 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc4249e0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d1b2577300 con 0x55d1b261c050 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b261ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b261ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40186/0 (socket says 172.21.15.69:40186) 2024-05-09T09:37:51.690 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41bfff640 1 -- 172.21.15.69:0/3518333066 learned_addr learned my addr 172.21.15.69:0/3518333066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41bfff640 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2623650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc418ff9640 1 -- 172.21.15.69:0/3784286390 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc4100cf6d0 con 0x7fc41c101980 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b2620b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.689+0000 7fc41affd640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 0x7fc4040a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3518333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 msgr2=0x55d1b261ba80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b261ba80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3518333066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 msgr2=0x55d1b2623650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2623650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3518333066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1b25478e0 con 0x55d1b261c050 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b2620b50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009230 tx=0x7fc40c005de0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6a643546883c2f41 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc41affd640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 0x7fc4040a50f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc41009b960 tx=0x7fc410009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc401ffb640 1 -- 172.21.15.69:0/3518333066 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c040070 con 0x55d1b261c050 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc401ffb640 1 -- 172.21.15.69:0/3518333066 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c0319b0 con 0x55d1b261c050 2024-05-09T09:37:51.691 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.690+0000 7fc401ffb640 1 -- 172.21.15.69:0/3518333066 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c035040 con 0x55d1b261c050 2024-05-09T09:37:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.691+0000 7fc4249e0740 1 -- 172.21.15.69:0/3518333066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 msgr2=0x55d1b2620b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.693 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.691+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b2620b50 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc40c009230 tx=0x7fc40c005de0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.691+0000 7fc4249e0740 1 -- 172.21.15.69:0/3518333066 shutdown_connections 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2623650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b2620b50 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3518333066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b261ba80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 -- 172.21.15.69:0/3518333066 >> 172.21.15.69:0/3518333066 conn(0x55d1b26128c0 msgr2=0x55d1b2612cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 -- 172.21.15.69:0/3518333066 shutdown_connections 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 -- 172.21.15.69:0/3518333066 wait complete. 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 -- start start 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.692+0000 7fc4249e0740 1 Processor -- start 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b261eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b26211c0 0x55d1b261fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.694 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d1b26170a0 con 0x55d1b261b660 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d1b2536790 con 0x55d1b26211c0 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d1b2617220 con 0x55d1b261c050 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc4249e0740 10 client.0 _ll_register_callbacks cb 0x55d1b2571a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:53002/0 (socket says 172.21.15.69:53002) 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 -- 172.21.15.69:0/1792470141 learned_addr learned my addr 172.21.15.69:0/1792470141 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b26211c0 0x55d1b261fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 -- 172.21.15.69:0/1792470141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 msgr2=0x55d1b261eba0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b261eba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 -- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b26211c0 msgr2=0x55d1b261fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.695 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b26211c0 0x55d1b261fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.693+0000 7fc41bfff640 1 -- 172.21.15.69:0/1792470141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1b228f210 con 0x55d1b261b660 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.694+0000 7fc41bfff640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc414011fd0 tx=0x7fc414004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=590f50a794b468de server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.694+0000 7fc3faffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.694+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc41400fe20 con 0x55d1b261b660 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.694+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc414011260 con 0x55d1b261b660 2024-05-09T09:37:51.696 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.694+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c0319b0 con 0x55d1b261b660 2024-05-09T09:37:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.695+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc414004b20 con 0x55d1b261b660 2024-05-09T09:37:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.695+0000 7fc4249e0740 10 client.17535 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.695+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d1b235b620 con 0x55d1b261b660 2024-05-09T09:37:51.697 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.695+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc414004cc0 con 0x55d1b261b660 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc414022420 con 0x55d1b261b660 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 10 client.17535 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 10 client.17535 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 20 client.17535 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b26354f0 0x55d1b26379b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 4 client.17535 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7fc40c0319b0 con 0x55d1b26354f0 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.696+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b26354f0 0x55d1b26379b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.698 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.697+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b26354f0 0x55d1b26379b0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5109c6f36df4c2 server_cookie=25d4b8dbca166c0d in_seq=0 out_seq=0 2024-05-09T09:37:51.699 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.697+0000 7fc3faffd640 10 client.17535 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.700+0000 7fc3faffd640 1 -- 172.21.15.69:0/1792470141 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7fc40c0319b0 con 0x55d1b26354f0 2024-05-09T09:37:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.700+0000 7fc3faffd640 10 client.17535 handle_command_reply: tid=0 2024-05-09T09:37:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.700+0000 7fc4249e0740 1 client.17535 shutdown 2024-05-09T09:37:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 msgr2=0x55d1b261e660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7fc414011fd0 tx=0x7fc414004a40 comp rx=0 tx=0).stop 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 shutdown_connections 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b26211c0 0x55d1b261fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b26354f0 0x55d1b26379b0 crc :-1 s=READY pgs=228 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261c050 0x55d1b261eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 --2- 172.21.15.69:0/1792470141 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261b660 0x55d1b261e660 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc3faffd640 0 client.17535 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 >> 172.21.15.69:0/1792470141 conn(0x55d1b26128c0 msgr2=0x55d1b2611a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 shutdown_connections 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 1 -- 172.21.15.69:0/1792470141 wait complete. 2024-05-09T09:37:51.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.701+0000 7fc4249e0740 20 client.17535 trim_cache size 0 max 16384 2024-05-09T09:37:51.718 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 Processor -- start 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 -- start start 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 0x55d1b261dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2611880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1aaad0 con 0x55d1b26211c0 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1a83a0 con 0x55d1b261c050 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc4249e0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1a61f0 con 0x55d1b261b660 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2611880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 0x55d1b261dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40192/0 (socket says 172.21.15.69:40192) 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 0x55d1b261dcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:49078/0 (socket says 172.21.15.69:49078) 2024-05-09T09:37:51.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3944281212 learned_addr learned my addr 172.21.15.69:0/3944281212 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3944281212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 msgr2=0x55d1b261dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 0x55d1b261dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3944281212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 msgr2=0x55d1b2611880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2611880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41b7fe640 1 -- 172.21.15.69:0/3944281212 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d1b260dbc0 con 0x55d1b261c050 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.718+0000 7fc41bfff640 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2611880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.719+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0314e0 tx=0x7fc40c031b60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=21361fcbcbeaf8b0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.719+0000 7fc4017fa640 1 -- 172.21.15.69:0/3944281212 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c054070 con 0x55d1b261c050 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.719+0000 7fc4017fa640 1 -- 172.21.15.69:0/3944281212 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c031dd0 con 0x55d1b261c050 2024-05-09T09:37:51.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.719+0000 7fc4017fa640 1 -- 172.21.15.69:0/3944281212 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc40c035040 con 0x55d1b261c050 2024-05-09T09:37:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- 172.21.15.69:0/3944281212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 msgr2=0x55d1b2616100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fc40c0314e0 tx=0x7fc40c031b60 comp rx=0 tx=0).stop 2024-05-09T09:37:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- 172.21.15.69:0/3944281212 shutdown_connections 2024-05-09T09:37:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b26211c0 0x55d1b2611880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.723 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261c050 0x55d1b2616100 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 --2- 172.21.15.69:0/3944281212 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b261b660 0x55d1b261dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- 172.21.15.69:0/3944281212 >> 172.21.15.69:0/3944281212 conn(0x55d1b26128c0 msgr2=0x55d1b2611170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- 172.21.15.69:0/3944281212 shutdown_connections 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- 172.21.15.69:0/3944281212 wait complete. 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 -- start start 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 Processor -- start 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.720+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261c050 0x55d1b2630220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b26211c0 0x55d1b26311c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1a6860 con 0x55d1b261c050 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1a7cd0 con 0x55d1b261b660 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fc41c1a6f30 con 0x55d1b26211c0 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc4249e0740 10 client.0 _ll_register_callbacks cb 0x7fc414005bd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:37:51.724 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40204/0 (socket says 172.21.15.69:40204) 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 -- 172.21.15.69:0/4243929308 learned_addr learned my addr 172.21.15.69:0/4243929308 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261c050 0x55d1b2630220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b26211c0 0x55d1b26311c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 -- 172.21.15.69:0/4243929308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b26211c0 msgr2=0x55d1b26311c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b26211c0 0x55d1b26311c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 -- 172.21.15.69:0/4243929308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261c050 msgr2=0x55d1b2630220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:37:51.725 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261c050 0x55d1b2630220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 -- 172.21.15.69:0/4243929308 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d1b26317b0 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc41bfff640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc41400fd30 tx=0x7fc414009f50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8637bbc25132e14b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc3faffd640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.721+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fc414011380 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fc40c031dd0 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fc41401a040 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fc414022a60 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 10 client.26816 fetch_fsmap learned FSMap version 47 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55d1b269bee0 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fc40c031dd0 con 0x55d1b261b660 2024-05-09T09:37:51.726 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fc414004f20 con 0x55d1b261b660 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 10 client.26816 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 10 client.26816 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 20 client.26816 populate_metadata read hostname 'smithi069' 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b269c8b0 0x55d1b269ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 4 client.26816 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.722+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d1b2588a70 con 0x55d1b269c8b0 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.723+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b269c8b0 0x55d1b269ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.723+0000 7fc3faffd640 10 client.26816 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:51.727 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:51.723+0000 7fc41b7fe640 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b269c8b0 0x55d1b269ecd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d6d9b0538031057 server_cookie=afba2cbf10b96da7 in_seq=0 out_seq=0 2024-05-09T09:37:51.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 190467 28.95 MB/sec execute 229 sec latency 59.837 ms 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc3faffd640 1 -- 172.21.15.69:0/4243929308 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d1b2588a70 con 0x55d1b269c8b0 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc3faffd640 10 client.26816 handle_command_reply: tid=0 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 client.26816 shutdown 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 msgr2=0x55d1b262fce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7fc41400fd30 tx=0x7fc414009f50 comp rx=0 tx=0).stop 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 shutdown_connections 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d1b269c8b0 0x55d1b269ecd0 crc :-1 s=READY pgs=229 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d1b26211c0 0x55d1b26311c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d1b261c050 0x55d1b2630220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc3faffd640 0 client.26816 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 --2- 172.21.15.69:0/4243929308 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d1b261b660 0x55d1b262fce0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.376+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 >> 172.21.15.69:0/4243929308 conn(0x55d1b26128c0 msgr2=0x55d1b2615f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 shutdown_connections 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc4249e0740 1 -- 172.21.15.69:0/4243929308 wait complete. 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc4249e0740 20 client.26816 trim_cache size 0 max 16384 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:37:52.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:37:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 msgr2=0x7fc4040a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 0x7fc4040a50f0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fc41009b960 tx=0x7fc410009310 comp rx=0 tx=0).stop 2024-05-09T09:37:52.379 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 msgr2=0x7fc41c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7fc41000e9f0 tx=0x7fc41000eeb0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 shutdown_connections 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fc4040a2c30 0x7fc4040a50f0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fc41c102340 0x7fc41c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fc41c101980 0x7fc41c198070 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 --2- 172.21.15.69:0/3784286390 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fc41c100790 0x7fc41c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 >> 172.21.15.69:0/3784286390 conn(0x7fc41c0fbd00 msgr2=0x7fc41c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 shutdown_connections 2024-05-09T09:37:52.380 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:37:52.377+0000 7fc421cdf640 1 -- 172.21.15.69:0/3784286390 wait complete. 2024-05-09T09:37:52.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 190842 28.90 MB/sec execute 230 sec latency 515.833 ms 2024-05-09T09:37:53.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:53 smithi103 ceph-mon[34421]: pgmap v939: 129 pgs: 129 active+clean; 666 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.4 MiB/s wr, 504 op/s 2024-05-09T09:37:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:53 smithi069 ceph-mon[30123]: pgmap v939: 129 pgs: 129 active+clean; 666 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.4 MiB/s wr, 504 op/s 2024-05-09T09:37:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:53 smithi163 ceph-mon[34354]: pgmap v939: 129 pgs: 129 active+clean; 666 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 3.4 MiB/s wr, 504 op/s 2024-05-09T09:37:53.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 191468 28.92 MB/sec execute 231 sec latency 96.226 ms 2024-05-09T09:37:54.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 192117 29.00 MB/sec execute 232 sec latency 58.271 ms 2024-05-09T09:37:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:55 smithi103 ceph-mon[34421]: pgmap v940: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 430 op/s 2024-05-09T09:37:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:55 smithi069 ceph-mon[30123]: pgmap v940: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 430 op/s 2024-05-09T09:37:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:55 smithi163 ceph-mon[34354]: pgmap v940: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 3.3 MiB/s wr, 430 op/s 2024-05-09T09:37:55.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 192569 28.97 MB/sec execute 233 sec latency 69.450 ms 2024-05-09T09:37:56.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 193078 28.95 MB/sec execute 234 sec latency 136.141 ms 2024-05-09T09:37:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:57 smithi103 ceph-mon[34421]: pgmap v941: 129 pgs: 129 active+clean; 667 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2024-05-09T09:37:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:57 smithi069 ceph-mon[30123]: pgmap v941: 129 pgs: 129 active+clean; 667 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2024-05-09T09:37:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:57 smithi163 ceph-mon[34354]: pgmap v941: 129 pgs: 129 active+clean; 667 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 3.5 MiB/s wr, 478 op/s 2024-05-09T09:37:57.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 193611 28.94 MB/sec execute 235 sec latency 75.472 ms 2024-05-09T09:37:58.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 194226 28.96 MB/sec execute 236 sec latency 67.188 ms 2024-05-09T09:37:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:37:59 smithi069 ceph-mon[30123]: pgmap v942: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 532 op/s 2024-05-09T09:37:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:37:59 smithi163 ceph-mon[34354]: pgmap v942: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 532 op/s 2024-05-09T09:37:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:37:59 smithi103 ceph-mon[34421]: pgmap v942: 129 pgs: 129 active+clean; 672 MiB data, 8.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.5 MiB/s wr, 532 op/s 2024-05-09T09:37:59.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 194894 29.01 MB/sec execute 237 sec latency 95.674 ms 2024-05-09T09:38:00.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 195556 29.00 MB/sec execute 238 sec latency 47.272 ms 2024-05-09T09:38:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:01 smithi069 ceph-mon[30123]: pgmap v943: 129 pgs: 129 active+clean; 673 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.9 MiB/s wr, 463 op/s 2024-05-09T09:38:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:01 smithi163 ceph-mon[34354]: pgmap v943: 129 pgs: 129 active+clean; 673 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.9 MiB/s wr, 463 op/s 2024-05-09T09:38:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:01 smithi103 ceph-mon[34421]: pgmap v943: 129 pgs: 129 active+clean; 673 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 2.9 MiB/s wr, 463 op/s 2024-05-09T09:38:01.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 195608 28.89 MB/sec execute 239 sec latency 958.576 ms 2024-05-09T09:38:02.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 195643 28.77 MB/sec execute 240 sec latency 1578.345 ms 2024-05-09T09:38:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:03 smithi069 ceph-mon[30123]: pgmap v944: 129 pgs: 129 active+clean; 676 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 460 op/s 2024-05-09T09:38:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:03 smithi163 ceph-mon[34354]: pgmap v944: 129 pgs: 129 active+clean; 676 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 460 op/s 2024-05-09T09:38:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:03 smithi103 ceph-mon[34421]: pgmap v944: 129 pgs: 129 active+clean; 676 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 2.9 MiB/s wr, 460 op/s 2024-05-09T09:38:03.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 195767 28.68 MB/sec execute 241 sec latency 724.264 ms 2024-05-09T09:38:04.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 196029 28.64 MB/sec execute 242 sec latency 135.881 ms 2024-05-09T09:38:05.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:05 smithi069 ceph-mon[30123]: pgmap v945: 129 pgs: 129 active+clean; 733 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 -- 172.21.15.69:0/366916077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020102a90 msgr2=0x7f3020104f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 --2- 172.21.15.69:0/366916077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020102a90 0x7f3020104f00 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3010009900 tx=0x7f301002a410 comp rx=0 tx=0).stop 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 -- 172.21.15.69:0/366916077 shutdown_connections 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 --2- 172.21.15.69:0/366916077 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020105440 0x7f30201058c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 --2- 172.21.15.69:0/366916077 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020102a90 0x7f3020104f00 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 --2- 172.21.15.69:0/366916077 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020100120 0x7f3020102550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.515 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.515+0000 7f3028142640 1 -- 172.21.15.69:0/366916077 >> 172.21.15.69:0/366916077 conn(0x7f30200fbd20 msgr2=0x7f30200fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.516+0000 7f3028142640 1 -- 172.21.15.69:0/366916077 shutdown_connections 2024-05-09T09:38:05.516 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.516+0000 7f3028142640 1 -- 172.21.15.69:0/366916077 wait complete. 2024-05-09T09:38:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 Processor -- start 2024-05-09T09:38:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 -- start start 2024-05-09T09:38:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020100120 0x7f3020070090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020105440 0x7f302019d260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3020113b60 con 0x7f3020102a90 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f30201139e0 con 0x7f3020105440 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.517+0000 7f3028142640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3020113ce0 con 0x7f3020100120 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57672/0 (socket says 172.21.15.69:57672) 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 -- 172.21.15.69:0/370727677 learned_addr learned my addr 172.21.15.69:0/370727677 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30266df640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020105440 0x7f302019d260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 -- 172.21.15.69:0/370727677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020100120 msgr2=0x7f3020070090 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020100120 0x7f3020070090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 -- 172.21.15.69:0/370727677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020105440 msgr2=0x7f302019d260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020105440 0x7f302019d260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.518+0000 7f30256dd640 1 -- 172.21.15.69:0/370727677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3020073770 con 0x7f3020102a90 2024-05-09T09:38:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.519+0000 7f30256dd640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3010031510 tx=0x7f3010032c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.522+0000 7f3016ffd640 1 -- 172.21.15.69:0/370727677 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f301003f030 con 0x7f3020102a90 2024-05-09T09:38:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.522+0000 7f3016ffd640 1 -- 172.21.15.69:0/370727677 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3010003ce0 con 0x7f3020102a90 2024-05-09T09:38:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.523+0000 7f3016ffd640 1 -- 172.21.15.69:0/370727677 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010035070 con 0x7f3020102a90 2024-05-09T09:38:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.523+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f302019d7a0 con 0x7f3020102a90 2024-05-09T09:38:05.525 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.523+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f302019dc00 con 0x7f3020102a90 2024-05-09T09:38:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.530+0000 7f3016ffd640 1 -- 172.21.15.69:0/370727677 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3010042030 con 0x7f3020102a90 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.536+0000 7f302ae55740 1 Processor -- start 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 -- start start 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0c40 0x558b42bb5740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bb8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558b42b14340 con 0x558b42bb0250 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558b42abc290 con 0x558b42bb5db0 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.537+0000 7f302ae55740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558b42b10300 con 0x558b42bb0c40 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f30266df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f30266df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57688/0 (socket says 172.21.15.69:57688) 2024-05-09T09:38:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f30266df640 1 -- 172.21.15.69:0/2705311965 learned_addr learned my addr 172.21.15.69:0/2705311965 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f3016ffd640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 0x7f300c0a5100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f3016ffd640 1 -- 172.21.15.69:0/370727677 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f30100ef220 con 0x7f3020102a90 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.538+0000 7f30256dd640 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bb8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.539+0000 7f30266df640 1 -- 172.21.15.69:0/2705311965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0c40 msgr2=0x558b42bb5740 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.539+0000 7f30266df640 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0c40 0x558b42bb5740 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.539+0000 7f30266df640 1 -- 172.21.15.69:0/2705311965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 msgr2=0x558b42bb8240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.539+0000 7f30266df640 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bb8240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.539+0000 7f30266df640 1 -- 172.21.15.69:0/2705311965 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b42ae08e0 con 0x558b42bb0250 2024-05-09T09:38:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.540+0000 7f30266df640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 0x7f300c0a5100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.541+0000 7f30266df640 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3018009d50 tx=0x7f3018010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b28fecdec3d359d0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.541+0000 7f30266df640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 0x7f300c0a5100 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3018018000 tx=0x7f301801a000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.541+0000 7f300bfff640 1 -- 172.21.15.69:0/2705311965 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3018013230 con 0x558b42bb0250 2024-05-09T09:38:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.542+0000 7f300bfff640 1 -- 172.21.15.69:0/2705311965 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3018012920 con 0x558b42bb0250 2024-05-09T09:38:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.543+0000 7f302ae55740 1 -- 172.21.15.69:0/2705311965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 msgr2=0x558b42bb0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.543+0000 7f302ae55740 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3018009d50 tx=0x7f3018010740 comp rx=0 tx=0).stop 2024-05-09T09:38:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.547+0000 7f302ae55740 1 -- 172.21.15.69:0/2705311965 shutdown_connections 2024-05-09T09:38:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.547+0000 7f302ae55740 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bb8240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.547+0000 7f302ae55740 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0c40 0x558b42bb5740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.547+0000 7f302ae55740 1 --2- 172.21.15.69:0/2705311965 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb0250 0x558b42bb0670 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.547 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.547+0000 7f302ae55740 1 -- 172.21.15.69:0/2705311965 >> 172.21.15.69:0/2705311965 conn(0x558b42bab8c0 msgr2=0x558b42babcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:05 smithi163 ceph-mon[34354]: pgmap v945: 129 pgs: 129 active+clean; 733 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-05-09T09:38:05.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.556+0000 7f302ae55740 1 -- 172.21.15.69:0/2705311965 shutdown_connections 2024-05-09T09:38:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.556+0000 7f302ae55740 1 -- 172.21.15.69:0/2705311965 wait complete. 2024-05-09T09:38:05.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.556+0000 7f302ae55740 1 -- start start 2024-05-09T09:38:05.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.557+0000 7f302ae55740 1 Processor -- start 2024-05-09T09:38:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bcb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 0x558b42bcb9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bd2a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x558b42bc9dc0 con 0x558b42bb5db0 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x558b42acf790 con 0x558b42bb0c40 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x558b42bc9f40 con 0x558b42bb0250 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f302ae55740 10 client.0 _ll_register_callbacks cb 0x558b42b0aa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.559+0000 7f30266df640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bd2a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.560+0000 7f3025ede640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bcb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.560+0000 7f3025ede640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bcb480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:40798/0 (socket says 172.21.15.69:40798) 2024-05-09T09:38:05.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.560+0000 7f3025ede640 1 -- 172.21.15.69:0/3522399126 learned_addr learned my addr 172.21.15.69:0/3522399126 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:05.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.560+0000 7f30256dd640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 0x558b42bcb9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 -- 172.21.15.69:0/3522399126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 msgr2=0x558b42bcb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bcb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 -- 172.21.15.69:0/3522399126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 msgr2=0x558b42bd2a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bd2a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 -- 172.21.15.69:0/3522399126 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b42828210 con 0x558b42bb0c40 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30266df640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bd2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30256dd640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 0x558b42bcb9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3010005ab0 tx=0x7f301002a910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8df6198c7838a546 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.561+0000 7f30097fa640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:38:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.565+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3010039350 con 0x558b42bb0c40 2024-05-09T09:38:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.565+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3018012920 con 0x558b42bb0c40 2024-05-09T09:38:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.565+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f30100b99a0 con 0x558b42bb0c40 2024-05-09T09:38:05.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.566+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f30100bbda0 con 0x558b42bb0c40 2024-05-09T09:38:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.567+0000 7f302ae55740 10 client.26828 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:05.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.567+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558b428f4620 con 0x558b42bb0c40 2024-05-09T09:38:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.567+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3018012920 con 0x558b42bb0c40 2024-05-09T09:38:05.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.568+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f30100b7d10 con 0x558b42bb0c40 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 10 client.26828 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 10 client.26828 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 20 client.26828 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42bd6e30 0x558b42bd9270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 4 client.26828 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:38:05.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.569+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x558b42b21a70 con 0x558b42bd6e30 2024-05-09T09:38:05.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.570+0000 7f3025ede640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42bd6e30 0x558b42bd9270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.575+0000 7f3025ede640 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42bd6e30 0x558b42bd9270 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9556bf6d8f0ac2e1 server_cookie=6f0fc05329f9cb28 in_seq=0 out_seq=0 2024-05-09T09:38:05.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.576+0000 7f30097fa640 10 client.26828 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.582+0000 7f30097fa640 1 -- 172.21.15.69:0/3522399126 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x558b42b21a70 con 0x558b42bd6e30 2024-05-09T09:38:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.582+0000 7f30097fa640 10 client.26828 handle_command_reply: tid=0 2024-05-09T09:38:05.583 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.583+0000 7f302ae55740 1 client.26828 shutdown 2024-05-09T09:38:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.587+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 msgr2=0x558b42bcb9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:05.587 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.587+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 0x558b42bcb9c0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3010005ab0 tx=0x7f301002a910 comp rx=0 tx=0).stop 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 shutdown_connections 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bd2a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb0c40 0x558b42bcb9c0 secure :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f3010005ab0 tx=0x7f301002a910 comp rx=0 tx=0).stop 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42bd6e30 0x558b42bd9270 crc :-1 s=READY pgs=230 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 --2- 172.21.15.69:0/3522399126 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bcb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 >> 172.21.15.69:0/3522399126 conn(0x558b42bab8c0 msgr2=0x558b42baaa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:05.588 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.588+0000 7f30097fa640 0 client.26828 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:05.589 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.589+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 shutdown_connections 2024-05-09T09:38:05.590 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.590+0000 7f302ae55740 1 -- 172.21.15.69:0/3522399126 wait complete. 2024-05-09T09:38:05.591 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.591+0000 7f302ae55740 20 client.26828 trim_cache size 0 max 16384 2024-05-09T09:38:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:05 smithi103 ceph-mon[34421]: pgmap v945: 129 pgs: 129 active+clean; 733 MiB data, 8.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 8.0 MiB/s wr, 587 op/s 2024-05-09T09:38:05.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.627+0000 7f302ae55740 1 Processor -- start 2024-05-09T09:38:05.627 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.627+0000 7f302ae55740 1 -- start start 2024-05-09T09:38:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.627+0000 7f302ae55740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bb1e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.627+0000 7f302ae55740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.628 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.627+0000 7f302ae55740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bd29c0 0x558b42bdeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f302ae55740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f30201a2190 con 0x558b42bb5db0 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f302ae55740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f302019fa60 con 0x558b42bd29c0 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f302ae55740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f302010e8e0 con 0x558b42bb0250 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57706/0 (socket says 172.21.15.69:57706) 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 -- 172.21.15.69:0/2601668262 learned_addr learned my addr 172.21.15.69:0/2601668262 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:05.629 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30266df640 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bd29c0 0x558b42bdeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 -- 172.21.15.69:0/2601668262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 msgr2=0x558b42bb1e20 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bb1e20 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 -- 172.21.15.69:0/2601668262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bd29c0 msgr2=0x558b42bdeb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bd29c0 0x558b42bdeb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.628+0000 7f30256dd640 1 -- 172.21.15.69:0/2601668262 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x558b42bafe90 con 0x558b42bb5db0 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.629+0000 7f30256dd640 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f30100bb630 tx=0x7f3010000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df42cb6537673157 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.629+0000 7f300b7fe640 1 -- 172.21.15.69:0/2601668262 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100b7940 con 0x558b42bb5db0 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.630+0000 7f300b7fe640 1 -- 172.21.15.69:0/2601668262 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3010039350 con 0x558b42bb5db0 2024-05-09T09:38:05.630 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.630+0000 7f300b7fe640 1 -- 172.21.15.69:0/2601668262 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100bbac0 con 0x558b42bb5db0 2024-05-09T09:38:05.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 -- 172.21.15.69:0/2601668262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 msgr2=0x558b42bb2360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:05.631 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7f30100bb630 tx=0x7f3010000bc0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 -- 172.21.15.69:0/2601668262 shutdown_connections 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bd29c0 0x558b42bdeb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bb5db0 0x558b42bb2360 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 --2- 172.21.15.69:0/2601668262 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bb0250 0x558b42bb1e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.631+0000 7f302ae55740 1 -- 172.21.15.69:0/2601668262 >> 172.21.15.69:0/2601668262 conn(0x558b42bab8c0 msgr2=0x558b42bcaaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:05.632 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.632+0000 7f302ae55740 1 -- 172.21.15.69:0/2601668262 shutdown_connections 2024-05-09T09:38:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.632+0000 7f302ae55740 1 -- 172.21.15.69:0/2601668262 wait complete. 2024-05-09T09:38:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.632+0000 7f302ae55740 1 -- start start 2024-05-09T09:38:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.632+0000 7f302ae55740 1 Processor -- start 2024-05-09T09:38:05.633 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bcfb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 0x558b42bd00d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bd10e0 0x558b42bd1560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f302019df20 con 0x558b42bd10e0 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f302019f390 con 0x558b42bb5db0 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f302019e5f0 con 0x558b42bd29c0 2024-05-09T09:38:05.634 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.633+0000 7f302ae55740 10 client.0 _ll_register_callbacks cb 0x7f3010032ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f3025ede640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bcfb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f3025ede640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bcfb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37862/0 (socket says 172.21.15.69:37862) 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f3025ede640 1 -- 172.21.15.69:0/40850066 learned_addr learned my addr 172.21.15.69:0/40850066 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 0x558b42bd00d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 -- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 msgr2=0x558b42bcfb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bcfb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 -- 172.21.15.69:0/40850066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bd10e0 msgr2=0x558b42bd1560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bd10e0 0x558b42bd1560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:05.635 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.634+0000 7f30256dd640 1 -- 172.21.15.69:0/40850066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x558b42bd1aa0 con 0x558b42bd29c0 2024-05-09T09:38:05.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.637+0000 7f30256dd640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 0x558b42bd00d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f30100b7b00 tx=0x7f3010031540 comp rx=0 tx=0).ready entity=mon.1 client_cookie=68c92d6cbc675a97 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:05.638 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.637+0000 7f30097fa640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:38:05.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.639+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f30100bbac0 con 0x558b42bd29c0 2024-05-09T09:38:05.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.639+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f3010039350 con 0x558b42bd29c0 2024-05-09T09:38:05.639 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.639+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f30100b7d10 con 0x558b42bd29c0 2024-05-09T09:38:05.643 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.643+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f30100b7940 con 0x558b42bd29c0 2024-05-09T09:38:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.646+0000 7f302ae55740 10 client.26746 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:05.646 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.646+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x558b42bd1aa0 con 0x558b42bd29c0 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3010039350 con 0x558b42bd29c0 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f30100baaf0 con 0x558b42bd29c0 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 10 client.26746 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 10 client.26746 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 20 client.26746 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42c3ddd0 0x558b42c401f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:05.653 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 4 client.26746 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:38:05.654 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.653+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x558b42b1bdc0 con 0x558b42c3ddd0 2024-05-09T09:38:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.654+0000 7f3025ede640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42c3ddd0 0x558b42c401f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:05.655 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.655+0000 7f30097fa640 10 client.26746 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:05.656 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:05.656+0000 7f3025ede640 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42c3ddd0 0x558b42c401f0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4424c37882a876d4 server_cookie=5a9d29359d85ef28 in_seq=0 out_seq=0 2024-05-09T09:38:05.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 196350 28.60 MB/sec execute 243 sec latency 147.443 ms 2024-05-09T09:38:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.092+0000 7f30097fa640 1 -- 172.21.15.69:0/40850066 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x558b42b1bdc0 con 0x558b42c3ddd0 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.092+0000 7f30097fa640 10 client.26746 handle_command_reply: tid=0 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.092+0000 7f302ae55740 1 client.26746 shutdown 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 msgr2=0x558b42bd00d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 0x558b42bd00d0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f30100b7b00 tx=0x7f3010031540 comp rx=0 tx=0).stop 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 shutdown_connections 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x558b42bd10e0 0x558b42bd1560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x558b42c3ddd0 0x558b42c401f0 crc :-1 s=READY pgs=231 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x558b42bd29c0 0x558b42bd00d0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 --2- 172.21.15.69:0/40850066 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x558b42bb5db0 0x558b42bcfb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 >> 172.21.15.69:0/40850066 conn(0x558b42bab8c0 msgr2=0x558b42bae840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f30097fa640 0 client.26746 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 shutdown_connections 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 1 -- 172.21.15.69:0/40850066 wait complete. 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.093+0000 7f302ae55740 20 client.26746 trim_cache size 0 max 16384 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:38:06.095 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 msgr2=0x7f300c0a5100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 0x7f300c0a5100 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f3018018000 tx=0x7f301801a000 comp rx=0 tx=0).stop 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 msgr2=0x7f30200705d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3010031510 tx=0x7f3010032c90 comp rx=0 tx=0).stop 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 shutdown_connections 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f300c0a2c40 0x7f300c0a5100 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3020105440 0x7f302019d260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.096 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3020102a90 0x7f30200705d0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 --2- 172.21.15.69:0/370727677 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3020100120 0x7f3020070090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 >> 172.21.15.69:0/370727677 conn(0x7f30200fbd20 msgr2=0x7f3020100a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 shutdown_connections 2024-05-09T09:38:06.097 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:06.094+0000 7f3028142640 1 -- 172.21.15.69:0/370727677 wait complete. 2024-05-09T09:38:06.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 196610 28.56 MB/sec execute 244 sec latency 278.284 ms 2024-05-09T09:38:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:07 smithi069 ceph-mon[30123]: pgmap v946: 129 pgs: 129 active+clean; 756 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 635 op/s 2024-05-09T09:38:07.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:07 smithi163 ceph-mon[34354]: pgmap v946: 129 pgs: 129 active+clean; 756 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 635 op/s 2024-05-09T09:38:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:07 smithi103 ceph-mon[34421]: pgmap v946: 129 pgs: 129 active+clean; 756 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 11 MiB/s wr, 635 op/s 2024-05-09T09:38:07.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 197183 28.56 MB/sec execute 245 sec latency 124.884 ms 2024-05-09T09:38:08.832 INFO:tasks.workunit.client.0.smithi069.stdout: 10 197702 28.55 MB/sec execute 246 sec latency 54.328 ms 2024-05-09T09:38:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:09 smithi069 ceph-mon[30123]: pgmap v947: 129 pgs: 129 active+clean; 739 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 618 op/s 2024-05-09T09:38:09.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:09 smithi163 ceph-mon[34354]: pgmap v947: 129 pgs: 129 active+clean; 739 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 618 op/s 2024-05-09T09:38:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:09 smithi103 ceph-mon[34421]: pgmap v947: 129 pgs: 129 active+clean; 739 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 12 MiB/s wr, 618 op/s 2024-05-09T09:38:09.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 198145 28.54 MB/sec execute 247 sec latency 82.692 ms 2024-05-09T09:38:10.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 198723 28.54 MB/sec execute 248 sec latency 59.937 ms 2024-05-09T09:38:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:11 smithi069 ceph-mon[30123]: pgmap v948: 129 pgs: 129 active+clean; 681 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 645 op/s 2024-05-09T09:38:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:11 smithi163 ceph-mon[34354]: pgmap v948: 129 pgs: 129 active+clean; 681 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 645 op/s 2024-05-09T09:38:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:11 smithi103 ceph-mon[34421]: pgmap v948: 129 pgs: 129 active+clean; 681 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 13 MiB/s wr, 645 op/s 2024-05-09T09:38:11.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 199476 28.55 MB/sec execute 249 sec latency 43.632 ms 2024-05-09T09:38:12.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 200273 28.64 MB/sec execute 250 sec latency 62.112 ms 2024-05-09T09:38:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:13 smithi069 ceph-mon[30123]: pgmap v949: 129 pgs: 129 active+clean; 667 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 717 op/s 2024-05-09T09:38:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:13 smithi163 ceph-mon[34354]: pgmap v949: 129 pgs: 129 active+clean; 667 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 717 op/s 2024-05-09T09:38:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:13 smithi103 ceph-mon[34421]: pgmap v949: 129 pgs: 129 active+clean; 667 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 13 MiB/s wr, 717 op/s 2024-05-09T09:38:13.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 200835 28.68 MB/sec execute 251 sec latency 87.402 ms 2024-05-09T09:38:14.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 201366 28.68 MB/sec execute 252 sec latency 241.725 ms 2024-05-09T09:38:15.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:15 smithi069 ceph-mon[30123]: pgmap v950: 129 pgs: 129 active+clean; 651 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-05-09T09:38:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:15 smithi163 ceph-mon[34354]: pgmap v950: 129 pgs: 129 active+clean; 651 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-05-09T09:38:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:15 smithi103 ceph-mon[34421]: pgmap v950: 129 pgs: 129 active+clean; 651 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 774 op/s 2024-05-09T09:38:15.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 201755 28.64 MB/sec execute 253 sec latency 76.322 ms 2024-05-09T09:38:16.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 202315 28.62 MB/sec execute 254 sec latency 81.815 ms 2024-05-09T09:38:17.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:17 smithi103 ceph-mon[34421]: pgmap v951: 129 pgs: 129 active+clean; 657 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 675 op/s 2024-05-09T09:38:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:17 smithi069 ceph-mon[30123]: pgmap v951: 129 pgs: 129 active+clean; 657 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 675 op/s 2024-05-09T09:38:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:17 smithi163 ceph-mon[34354]: pgmap v951: 129 pgs: 129 active+clean; 657 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 8.3 MiB/s wr, 675 op/s 2024-05-09T09:38:17.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 203120 28.67 MB/sec execute 255 sec latency 57.220 ms 2024-05-09T09:38:18.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 203848 28.71 MB/sec execute 256 sec latency 56.239 ms 2024-05-09T09:38:19.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.209+0000 7f70dbfff640 1 -- 172.21.15.69:0/205533629 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d00319b0 con 0x7f70dc1008d0 2024-05-09T09:38:19.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 -- 172.21.15.69:0/205533629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc1008d0 msgr2=0x7f70dc100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 --2- 172.21.15.69:0/205533629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc1008d0 0x7f70dc100d50 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009900 tx=0x7f70d002a410 comp rx=0 tx=0).stop 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 -- 172.21.15.69:0/205533629 shutdown_connections 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 --2- 172.21.15.69:0/205533629 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc101290 0x7f70dc105b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 --2- 172.21.15.69:0/205533629 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc1008d0 0x7f70dc100d50 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 --2- 172.21.15.69:0/205533629 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc0ffb70 0x7f70dc0fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 -- 172.21.15.69:0/205533629 >> 172.21.15.69:0/205533629 conn(0x7f70dc0fb4a0 msgr2=0x7f70dc0fd8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:19.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 -- 172.21.15.69:0/205533629 shutdown_connections 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.210+0000 7f70e3363640 1 -- 172.21.15.69:0/205533629 wait complete. 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.211+0000 7f70e3363640 1 Processor -- start 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.211+0000 7f70e3363640 1 -- start start 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.211+0000 7f70e3363640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.211+0000 7f70e3363640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc1008d0 0x7f70dc10c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.211+0000 7f70e3363640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e3363640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc113bb0 con 0x7f70dc101290 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e3363640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc113a30 con 0x7f70dc1008d0 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e3363640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc113d30 con 0x7f70dc0ffb70 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e1900640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:42696/0 (socket says 172.21.15.69:42696) 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e1900640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55932/0 (socket says 172.21.15.69:55932) 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 -- 172.21.15.69:0/1349434912 learned_addr learned my addr 172.21.15.69:0/1349434912 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:19.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc1008d0 0x7f70dc10c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 -- 172.21.15.69:0/1349434912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc1008d0 msgr2=0x7f70dc10c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc1008d0 0x7f70dc10c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 -- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 msgr2=0x7f70dc1ae280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e10ff640 1 -- 172.21.15.69:0/1349434912 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f70dc10f4a0 con 0x7f70dc0ffb70 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.212+0000 7f70e1900640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00c890 tx=0x7f70cc00cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70da7fc640 1 -- 172.21.15.69:0/1349434912 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc004220 con 0x7f70dc0ffb70 2024-05-09T09:38:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f70dc1ae820 con 0x7f70dc0ffb70 2024-05-09T09:38:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70da7fc640 1 -- 172.21.15.69:0/1349434912 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc0043c0 con 0x7f70dc0ffb70 2024-05-09T09:38:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70da7fc640 1 -- 172.21.15.69:0/1349434912 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc004a00 con 0x7f70dc0ffb70 2024-05-09T09:38:19.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.213+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f70dc1aed90 con 0x7f70dc0ffb70 2024-05-09T09:38:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 Processor -- start 2024-05-09T09:38:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 -- start start 2024-05-09T09:38:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbde8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbded740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.220 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5581fbd4c340 con 0x5581fbde8c40 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5581fbcf4290 con 0x5581fbdeddb0 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e6073740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5581fbd48300 con 0x5581fbde8250 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.217+0000 7f70e08fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36584/0 (socket says 172.21.15.69:36584) 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 -- 172.21.15.69:0/2305819704 learned_addr learned my addr 172.21.15.69:0/2305819704 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e10ff640 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbded740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 -- 172.21.15.69:0/2305819704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 msgr2=0x5581fbde8670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbde8670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 -- 172.21.15.69:0/2305819704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 msgr2=0x5581fbded740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbded740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.218+0000 7f70e08fe640 1 -- 172.21.15.69:0/2305819704 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581fbd188e0 con 0x5581fbdeddb0 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.219+0000 7f70e08fe640 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70d0031610 tx=0x7f70d0004260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a2bf312a616f6844 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.221 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.221+0000 7f70da7fc640 1 -- 172.21.15.69:0/1349434912 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f70cc0079b0 con 0x7f70dc0ffb70 2024-05-09T09:38:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.222+0000 7f70da7fc640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 0x7f70c40a5240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.222+0000 7f70da7fc640 1 -- 172.21.15.69:0/1349434912 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f70cc0cf390 con 0x7f70dc0ffb70 2024-05-09T09:38:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.222+0000 7f70c37fe640 1 -- 172.21.15.69:0/2305819704 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0035040 con 0x5581fbdeddb0 2024-05-09T09:38:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.222+0000 7f70c37fe640 1 -- 172.21.15.69:0/2305819704 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0005520 con 0x5581fbdeddb0 2024-05-09T09:38:19.222 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.222+0000 7f70c37fe640 1 -- 172.21.15.69:0/2305819704 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0031910 con 0x5581fbdeddb0 2024-05-09T09:38:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.223+0000 7f70e6073740 1 -- 172.21.15.69:0/2305819704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 msgr2=0x5581fbdf0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.223+0000 7f70e6073740 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f70d0031610 tx=0x7f70d0004260 comp rx=0 tx=0).stop 2024-05-09T09:38:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.223+0000 7f70e1900640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 0x7f70c40a5240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.224+0000 7f70e1900640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 0x7f70c40a5240 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fd0 tx=0x7f70d4009600 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.225 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.225+0000 7f70e6073740 1 -- 172.21.15.69:0/2305819704 shutdown_connections 2024-05-09T09:38:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.225+0000 7f70e6073740 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbdf0240 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.225+0000 7f70e6073740 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbded740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.225+0000 7f70e6073740 1 --2- 172.21.15.69:0/2305819704 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbde8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.225+0000 7f70e6073740 1 -- 172.21.15.69:0/2305819704 >> 172.21.15.69:0/2305819704 conn(0x5581fbde38c0 msgr2=0x5581fbde3cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:19.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.229+0000 7f70e6073740 1 -- 172.21.15.69:0/2305819704 shutdown_connections 2024-05-09T09:38:19.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.230+0000 7f70e6073740 1 -- 172.21.15.69:0/2305819704 wait complete. 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.230+0000 7f70e6073740 1 -- start start 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.230+0000 7f70e6073740 1 Processor -- start 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe03a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbdeddb0 0x5581fbe0aa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5581fbdf0d00 con 0x5581fbdeddb0 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5581fbd07790 con 0x5581fbde8c40 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5581fbdf0e80 con 0x5581fbde8250 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e6073740 10 client.0 _ll_register_callbacks cb 0x5581fbd42a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:42716/0 (socket says 172.21.15.69:42716) 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.232+0000 7f70e10ff640 1 -- 172.21.15.69:0/1736289773 learned_addr learned my addr 172.21.15.69:0/1736289773 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.233+0000 7f70e10ff640 1 -- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 msgr2=0x5581fbe03a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.233+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe03a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.233+0000 7f70e10ff640 1 -- 172.21.15.69:0/1736289773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbdeddb0 msgr2=0x5581fbe0aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.233+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbdeddb0 0x5581fbe0aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.233+0000 7f70e10ff640 1 -- 172.21.15.69:0/1736289773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581fba60210 con 0x5581fbde8250 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.234+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f70cc097d20 tx=0x7f70cc0053a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=def83328f5f03a0d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.234+0000 7f70c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:38:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.238+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc01d040 con 0x5581fbde8250 2024-05-09T09:38:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.238+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70d0005520 con 0x5581fbde8250 2024-05-09T09:38:19.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.238+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f70cc0860c0 con 0x5581fbde8250 2024-05-09T09:38:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.240+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f70cc09b690 con 0x5581fbde8250 2024-05-09T09:38:19.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.240+0000 7f70e6073740 10 client.26755 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:19.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.240+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5581fbb2c620 con 0x5581fbde8250 2024-05-09T09:38:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.242+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70d0005520 con 0x5581fbde8250 2024-05-09T09:38:19.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.242+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f70cc0a3ff0 con 0x5581fbde8250 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 10 client.26755 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 10 client.26755 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 20 client.26755 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe0f070 0x5581fbe11500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 4 client.26755 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:38:19.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.243+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5581fbd59a70 con 0x5581fbe0f070 2024-05-09T09:38:19.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.245+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe0f070 0x5581fbe11500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.246+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe0f070 0x5581fbe11500 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4bf685fc1bd4035c server_cookie=e26976696a17afba in_seq=0 out_seq=0 2024-05-09T09:38:19.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.246+0000 7f70c0ff9640 10 client.26755 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:19.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.249+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1736289773 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5581fbd59a70 con 0x5581fbe0f070 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70c0ff9640 10 client.26755 handle_command_reply: tid=0 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 client.26755 shutdown 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 msgr2=0x5581fbe034d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f70cc097d20 tx=0x7f70cc0053a0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 shutdown_connections 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbdeddb0 0x5581fbe0aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe03a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe0f070 0x5581fbe11500 crc :-1 s=READY pgs=232 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 --2- 172.21.15.69:0/1736289773 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe034d0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 >> 172.21.15.69:0/1736289773 conn(0x5581fbde38c0 msgr2=0x5581fbde2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70c0ff9640 0 client.26755 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 shutdown_connections 2024-05-09T09:38:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 1 -- 172.21.15.69:0/1736289773 wait complete. 2024-05-09T09:38:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.250+0000 7f70e6073740 20 client.26755 trim_cache size 0 max 16384 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.273+0000 7f70e6073740 1 Processor -- start 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.273+0000 7f70e6073740 1 -- start start 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.273+0000 7f70e6073740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.273+0000 7f70e6073740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe173a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e6073740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbdeddb0 0x5581fbde6870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.274 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e6073740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc19d4c0 con 0x5581fbde8250 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e6073740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc19ad90 con 0x5581fbde8c40 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e6073740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc073320 con 0x5581fbdeddb0 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55948/0 (socket says 172.21.15.69:55948) 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 -- 172.21.15.69:0/37637389 learned_addr learned my addr 172.21.15.69:0/37637389 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e08fe640 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe173a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e1900640 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbdeddb0 0x5581fbde6870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 -- 172.21.15.69:0/37637389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbdeddb0 msgr2=0x5581fbde6870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbdeddb0 0x5581fbde6870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 -- 172.21.15.69:0/37637389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 msgr2=0x5581fbe173a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe173a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 -- 172.21.15.69:0/37637389 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5581fbde9020 con 0x5581fbde8250 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70e10ff640 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0058c0 tx=0x7f70cc005300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b96363da72d67803 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.275 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.274+0000 7f70c2ffd640 1 -- 172.21.15.69:0/37637389 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc01d040 con 0x5581fbde8250 2024-05-09T09:38:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70c2ffd640 1 -- 172.21.15.69:0/37637389 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc084e50 con 0x5581fbde8250 2024-05-09T09:38:19.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70c2ffd640 1 -- 172.21.15.69:0/37637389 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70cc09b220 con 0x5581fbde8250 2024-05-09T09:38:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 -- 172.21.15.69:0/37637389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 msgr2=0x5581fbe16e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7f70cc0058c0 tx=0x7f70cc005300 comp rx=0 tx=0).stop 2024-05-09T09:38:19.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 -- 172.21.15.69:0/37637389 shutdown_connections 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbdeddb0 0x5581fbde6870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbde8c40 0x5581fbe173a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 --2- 172.21.15.69:0/37637389 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8250 0x5581fbe16e60 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 -- 172.21.15.69:0/37637389 >> 172.21.15.69:0/37637389 conn(0x5581fbde38c0 msgr2=0x5581fbe10e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 -- 172.21.15.69:0/37637389 shutdown_connections 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.275+0000 7f70e6073740 1 -- 172.21.15.69:0/37637389 wait complete. 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 -- start start 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 Processor -- start 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe077d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbe08d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc199250 con 0x5581fbde8c40 2024-05-09T09:38:19.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc19a6c0 con 0x5581fbdeddb0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f70dc199920 con 0x5581fbde8250 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e6073740 10 client.0 _ll_register_callbacks cb 0x7f70cc0057c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55950/0 (socket says 172.21.15.69:55950) 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 -- 172.21.15.69:0/1377561158 learned_addr learned my addr 172.21.15.69:0/1377561158 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe077d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e1900640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbe08d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 -- 172.21.15.69:0/1377561158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 msgr2=0x5581fbe077d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe077d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 -- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 msgr2=0x5581fbe08d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.276+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbe08d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e08fe640 1 -- 172.21.15.69:0/1377561158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5581fbe09310 con 0x5581fbde8c40 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e08fe640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009850 tx=0x7f70d0005e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dbd2a788d6a2fa93 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:38:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f70d0035040 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f70d0031b80 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f70cc084e50 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f70d0080050 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 10 client.17571 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5581fbe75810 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f70cc084e50 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f70d00710e0 con 0x5581fbde8c40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 10 client.17571 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 10 client.17571 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 20 client.17571 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe75e40 0x5581fbe78230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 4 client.17571 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.277+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5581fbd53dc0 con 0x5581fbe75e40 2024-05-09T09:38:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.278+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe75e40 0x5581fbe78230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.278+0000 7f70e10ff640 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe75e40 0x5581fbe78230 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e802a33c6a51f8f server_cookie=4b091f00db81ccdb in_seq=0 out_seq=0 2024-05-09T09:38:19.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:19.278+0000 7f70c0ff9640 10 client.17571 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:19 smithi069 ceph-mon[30123]: pgmap v952: 129 pgs: 129 active+clean; 658 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 557 op/s 2024-05-09T09:38:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:19 smithi163 ceph-mon[34354]: pgmap v952: 129 pgs: 129 active+clean; 658 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 557 op/s 2024-05-09T09:38:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:19 smithi103 ceph-mon[34421]: pgmap v952: 129 pgs: 129 active+clean; 658 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 4.8 MiB/s wr, 557 op/s 2024-05-09T09:38:19.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 204232 28.70 MB/sec execute 257 sec latency 491.065 ms 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.012+0000 7f70c0ff9640 1 -- 172.21.15.69:0/1377561158 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5581fbd53dc0 con 0x5581fbe75e40 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.012+0000 7f70c0ff9640 10 client.17571 handle_command_reply: tid=0 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.012+0000 7f70e6073740 1 client.17571 shutdown 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.012+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 msgr2=0x5581fbe07d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.012+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7f70d0009850 tx=0x7f70d0005e70 comp rx=0 tx=0).stop 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 shutdown_connections 2024-05-09T09:38:20.013 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5581fbdeddb0 0x5581fbe08d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5581fbde8c40 0x5581fbe07d10 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5581fbe75e40 0x5581fbe78230 crc :-1 s=READY pgs=233 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70c0ff9640 0 client.17571 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 --2- 172.21.15.69:0/1377561158 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5581fbde8250 0x5581fbe077d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 >> 172.21.15.69:0/1377561158 conn(0x5581fbde38c0 msgr2=0x5581fbde6870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 shutdown_connections 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 1 -- 172.21.15.69:0/1377561158 wait complete. 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.013+0000 7f70e6073740 20 client.17571 trim_cache size 0 max 16384 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:38:20.014 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:38:20.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 msgr2=0x7f70c40a5240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 0x7f70c40a5240 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f70d4009fd0 tx=0x7f70d4009600 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 msgr2=0x7f70dc10be30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f70cc00c890 tx=0x7f70cc00cd50 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 shutdown_connections 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f70c40a2d80 0x7f70c40a5240 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f70dc101290 0x7f70dc1ae280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f70dc1008d0 0x7f70dc10c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 --2- 172.21.15.69:0/1349434912 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f70dc0ffb70 0x7f70dc10be30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 >> 172.21.15.69:0/1349434912 conn(0x7f70dc0fb4a0 msgr2=0x7f70dc101b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.016+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 shutdown_connections 2024-05-09T09:38:20.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:20.017+0000 7f70e3363640 1 -- 172.21.15.69:0/1349434912 wait complete. 2024-05-09T09:38:20.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 204771 28.73 MB/sec execute 258 sec latency 583.564 ms 2024-05-09T09:38:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:21 smithi069 ceph-mon[30123]: pgmap v953: 129 pgs: 129 active+clean; 666 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 555 op/s 2024-05-09T09:38:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:21 smithi163 ceph-mon[34354]: pgmap v953: 129 pgs: 129 active+clean; 666 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 555 op/s 2024-05-09T09:38:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:21 smithi103 ceph-mon[34421]: pgmap v953: 129 pgs: 129 active+clean; 666 MiB data, 8.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 4.1 MiB/s wr, 555 op/s 2024-05-09T09:38:21.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 205288 28.71 MB/sec execute 259 sec latency 81.422 ms 2024-05-09T09:38:22.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 205841 28.72 MB/sec execute 260 sec latency 53.197 ms 2024-05-09T09:38:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:23 smithi069 ceph-mon[30123]: pgmap v954: 129 pgs: 129 active+clean; 670 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 470 op/s 2024-05-09T09:38:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:23 smithi163 ceph-mon[34354]: pgmap v954: 129 pgs: 129 active+clean; 670 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 470 op/s 2024-05-09T09:38:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:23 smithi103 ceph-mon[34421]: pgmap v954: 129 pgs: 129 active+clean; 670 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.2 MiB/s wr, 470 op/s 2024-05-09T09:38:23.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 206381 28.68 MB/sec execute 261 sec latency 68.573 ms 2024-05-09T09:38:24.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:24 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:38:24.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:24 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:38:24.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:24 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:38:24.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 207112 28.72 MB/sec execute 262 sec latency 53.703 ms 2024-05-09T09:38:25.412 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:25 smithi069 ceph-mon[30123]: pgmap v955: 129 pgs: 129 active+clean; 674 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 573 op/s 2024-05-09T09:38:25.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:38:25.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:38:25.413 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:38:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:25 smithi163 ceph-mon[34354]: pgmap v955: 129 pgs: 129 active+clean; 674 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 573 op/s 2024-05-09T09:38:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:38:25.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:38:25.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:38:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:25 smithi103 ceph-mon[34421]: pgmap v955: 129 pgs: 129 active+clean; 674 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 573 op/s 2024-05-09T09:38:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:38:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:38:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:38:25.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 207715 28.74 MB/sec execute 263 sec latency 86.889 ms 2024-05-09T09:38:26.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 208344 28.80 MB/sec execute 264 sec latency 137.653 ms 2024-05-09T09:38:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:27 smithi069 ceph-mon[30123]: pgmap v956: 129 pgs: 129 active+clean; 675 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 558 op/s 2024-05-09T09:38:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:27 smithi163 ceph-mon[34354]: pgmap v956: 129 pgs: 129 active+clean; 675 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 558 op/s 2024-05-09T09:38:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:27 smithi103 ceph-mon[34421]: pgmap v956: 129 pgs: 129 active+clean; 675 MiB data, 8.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.4 MiB/s wr, 558 op/s 2024-05-09T09:38:27.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 208947 28.83 MB/sec execute 265 sec latency 76.302 ms 2024-05-09T09:38:28.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 209282 28.77 MB/sec execute 266 sec latency 123.923 ms 2024-05-09T09:38:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:29 smithi069 ceph-mon[30123]: pgmap v957: 129 pgs: 129 active+clean; 680 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 543 op/s 2024-05-09T09:38:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:29 smithi163 ceph-mon[34354]: pgmap v957: 129 pgs: 129 active+clean; 680 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 543 op/s 2024-05-09T09:38:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:29 smithi103 ceph-mon[34421]: pgmap v957: 129 pgs: 129 active+clean; 680 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.3 MiB/s wr, 543 op/s 2024-05-09T09:38:29.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 209689 28.74 MB/sec execute 267 sec latency 70.356 ms 2024-05-09T09:38:30.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 210145 28.70 MB/sec execute 268 sec latency 110.292 ms 2024-05-09T09:38:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:31 smithi069 ceph-mon[30123]: pgmap v958: 129 pgs: 129 active+clean; 685 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 585 op/s 2024-05-09T09:38:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:31 smithi163 ceph-mon[34354]: pgmap v958: 129 pgs: 129 active+clean; 685 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 585 op/s 2024-05-09T09:38:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:31 smithi103 ceph-mon[34421]: pgmap v958: 129 pgs: 129 active+clean; 685 MiB data, 8.7 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.7 MiB/s wr, 585 op/s 2024-05-09T09:38:31.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 210583 28.64 MB/sec execute 269 sec latency 255.208 ms 2024-05-09T09:38:32.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 210630 28.56 MB/sec execute 270 sec latency 1255.253 ms 2024-05-09T09:38:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.187+0000 7f62813c8640 1 -- 172.21.15.69:0/3505509829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c10db50 msgr2=0x7f627c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.188 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.187+0000 7f62813c8640 1 --2- 172.21.15.69:0/3505509829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c10db50 0x7f627c10dfd0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7f627001a910 tx=0x7f627003b470 comp rx=0 tx=0).stop 2024-05-09T09:38:33.190 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.191+0000 7f62813c8640 1 -- 172.21.15.69:0/3505509829 shutdown_connections 2024-05-09T09:38:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.191+0000 7f62813c8640 1 --2- 172.21.15.69:0/3505509829 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c10db50 0x7f627c10dfd0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.191+0000 7f62813c8640 1 --2- 172.21.15.69:0/3505509829 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10b1a0 0x7f627c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.191+0000 7f62813c8640 1 --2- 172.21.15.69:0/3505509829 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c108830 0x7f627c10ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.191 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.191+0000 7f62813c8640 1 -- 172.21.15.69:0/3505509829 >> 172.21.15.69:0/3505509829 conn(0x7f627c06e8a0 msgr2=0x7f627c070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:33.201 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.199+0000 7f62813c8640 1 -- 172.21.15.69:0/3505509829 shutdown_connections 2024-05-09T09:38:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.204+0000 7f62813c8640 1 -- 172.21.15.69:0/3505509829 wait complete. 2024-05-09T09:38:33.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.205+0000 7f62813c8640 1 Processor -- start 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 -- start start 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c108830 0x7f627c19e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c10b1a0 0x7f627c19e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111be0 con 0x7f627c108830 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111a60 con 0x7f627c10db50 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.207+0000 7f62813c8640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f627c111d60 con 0x7f627c10b1a0 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.209+0000 7f627a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c10b1a0 0x7f627c19e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.209+0000 7f627b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.209+0000 7f627b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58902/0 (socket says 172.21.15.69:58902) 2024-05-09T09:38:33.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.209+0000 7f627b7fe640 1 -- 172.21.15.69:0/4132464812 learned_addr learned my addr 172.21.15.69:0/4132464812 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:33.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.211+0000 7f627affd640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c108830 0x7f627c19e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.216+0000 7f627b7fe640 1 -- 172.21.15.69:0/4132464812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c10b1a0 msgr2=0x7f627c19e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.216+0000 7f627b7fe640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c10b1a0 0x7f627c19e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.216+0000 7f627b7fe640 1 -- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c108830 msgr2=0x7f627c19e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.216+0000 7f627b7fe640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c108830 0x7f627c19e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.218 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.216+0000 7f627b7fe640 1 -- 172.21.15.69:0/4132464812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f627c1a5e30 con 0x7f627c10db50 2024-05-09T09:38:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.221+0000 7f627b7fe640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6270004cd0 tx=0x7f6270004d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.223+0000 7f6263fff640 1 -- 172.21.15.69:0/4132464812 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f627004f070 con 0x7f627c10db50 2024-05-09T09:38:33.223 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.223+0000 7f6263fff640 1 -- 172.21.15.69:0/4132464812 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6270040cf0 con 0x7f627c10db50 2024-05-09T09:38:33.226 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.224+0000 7f6263fff640 1 -- 172.21.15.69:0/4132464812 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6270046030 con 0x7f627c10db50 2024-05-09T09:38:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.225+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f627c1a60c0 con 0x7f627c10db50 2024-05-09T09:38:33.227 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.226+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f627c1a65b0 con 0x7f627c10db50 2024-05-09T09:38:33.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.232+0000 7f62841b5740 1 Processor -- start 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 -- start start 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c8262670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8267740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c826a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e2c81c6340 con 0x55e2c8267db0 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e2c816e290 con 0x55e2c8262250 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f62841b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e2c81c2300 con 0x55e2c8262c40 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.233+0000 7f6263fff640 1 -- 172.21.15.69:0/4132464812 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6270052070 con 0x7f627c10db50 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.234+0000 7f6263fff640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 0x7f62640a5200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.235+0000 7f6263fff640 1 -- 172.21.15.69:0/4132464812 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f62701004f0 con 0x7f627c10db50 2024-05-09T09:38:33.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.236+0000 7f627a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c8262670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.236+0000 7f627a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c8262670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58914/0 (socket says 172.21.15.69:58914) 2024-05-09T09:38:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.236+0000 7f627a7fc640 1 -- 172.21.15.69:0/2718477013 learned_addr learned my addr 172.21.15.69:0/2718477013 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.237+0000 7f627b7fe640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 0x7f62640a5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.237+0000 7f627affd640 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8267740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.241+0000 7f627a7fc640 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c826a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.242+0000 7f627affd640 1 -- 172.21.15.69:0/2718477013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 msgr2=0x55e2c8262670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.242+0000 7f627affd640 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c8262670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.242+0000 7f627affd640 1 -- 172.21.15.69:0/2718477013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 msgr2=0x55e2c826a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.242+0000 7f627affd640 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c826a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.242+0000 7f627affd640 1 -- 172.21.15.69:0/2718477013 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c81928e0 con 0x55e2c8262c40 2024-05-09T09:38:33.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.243+0000 7f627affd640 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8267740 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f626c009980 tx=0x7f626c009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=66c3169088f228ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.246+0000 7f62617fa640 1 -- 172.21.15.69:0/2718477013 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c017070 con 0x55e2c8262c40 2024-05-09T09:38:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.246+0000 7f62617fa640 1 -- 172.21.15.69:0/2718477013 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626c013470 con 0x55e2c8262c40 2024-05-09T09:38:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.246+0000 7f62617fa640 1 -- 172.21.15.69:0/2718477013 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c012630 con 0x55e2c8262c40 2024-05-09T09:38:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.247+0000 7f62841b5740 1 -- 172.21.15.69:0/2718477013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 msgr2=0x55e2c8267740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.247+0000 7f62841b5740 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8267740 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f626c009980 tx=0x7f626c009e40 comp rx=0 tx=0).stop 2024-05-09T09:38:33.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.248+0000 7f627b7fe640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 0x7f62640a5200 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f62700c8d40 tx=0x7f6270040510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.252+0000 7f62841b5740 1 -- 172.21.15.69:0/2718477013 shutdown_connections 2024-05-09T09:38:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.252+0000 7f62841b5740 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c826a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.252+0000 7f62841b5740 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8267740 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.252+0000 7f62841b5740 1 --2- 172.21.15.69:0/2718477013 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c8262670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.252+0000 7f62841b5740 1 -- 172.21.15.69:0/2718477013 >> 172.21.15.69:0/2718477013 conn(0x55e2c825d8c0 msgr2=0x55e2c825dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:33.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.253+0000 7f62841b5740 1 -- 172.21.15.69:0/2718477013 shutdown_connections 2024-05-09T09:38:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.257+0000 7f62841b5740 1 -- 172.21.15.69:0/2718477013 wait complete. 2024-05-09T09:38:33.256 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.257+0000 7f62841b5740 1 -- start start 2024-05-09T09:38:33.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.257+0000 7f62841b5740 1 Processor -- start 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c826c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 0x55e2c826ca30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8267db0 0x55e2c8273a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55e2c826ad60 con 0x55e2c8262c40 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55e2c8181790 con 0x55e2c8262250 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55e2c826aee0 con 0x55e2c8267db0 2024-05-09T09:38:33.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.261+0000 7f62841b5740 10 client.0 _ll_register_callbacks cb 0x55e2c81bca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.264+0000 7f627affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 0x55e2c826ca30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.264+0000 7f627affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 0x55e2c826ca30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:42226/0 (socket says 172.21.15.69:42226) 2024-05-09T09:38:33.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.264+0000 7f627affd640 1 -- 172.21.15.69:0/3565294960 learned_addr learned my addr 172.21.15.69:0/3565294960 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.265+0000 7f627a7fc640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c826c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.265+0000 7f627a7fc640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8267db0 0x55e2c8273a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.272+0000 7f627a7fc640 1 -- 172.21.15.69:0/3565294960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8267db0 msgr2=0x55e2c8273a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.272+0000 7f627a7fc640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8267db0 0x55e2c8273a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.271 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.272+0000 7f627a7fc640 1 -- 172.21.15.69:0/3565294960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 msgr2=0x55e2c826ca30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.272+0000 7f627a7fc640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 0x55e2c826ca30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.272+0000 7f627a7fc640 1 -- 172.21.15.69:0/3565294960 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c7eda210 con 0x55e2c8262250 2024-05-09T09:38:33.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.280+0000 7f627a7fc640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c826c4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6274009fd0 tx=0x7f6274004a60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c7e5012a77ce7351 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.280+0000 7f625eff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:38:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.288+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740198e0 con 0x55e2c8262250 2024-05-09T09:38:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.288+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6274019a80 con 0x55e2c8262250 2024-05-09T09:38:33.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.289+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f626c013470 con 0x55e2c8262250 2024-05-09T09:38:33.296 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.296+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6274019c80 con 0x55e2c8262250 2024-05-09T09:38:33.298 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.297+0000 7f62841b5740 10 client.26855 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:33.298 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.298+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e2c7fa6620 con 0x55e2c8262250 2024-05-09T09:38:33.298 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626c013470 con 0x55e2c8262250 2024-05-09T09:38:33.298 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f6274017360 con 0x55e2c8262250 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 10 client.26855 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 10 client.26855 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 20 client.26855 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c8278370 0x55e2c827a810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 4 client.26855 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:38:33.299 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.299+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55e2c81d3a70 con 0x55e2c8278370 2024-05-09T09:38:33.306 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.304+0000 7f627affd640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c8278370 0x55e2c827a810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.314 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.314+0000 7f627affd640 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c8278370 0x55e2c827a810 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6a9d850deca055a7 server_cookie=bf68d125f755b883 in_seq=0 out_seq=0 2024-05-09T09:38:33.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.318+0000 7f625eff5640 10 client.26855 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.326+0000 7f625eff5640 1 -- 172.21.15.69:0/3565294960 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55e2c81d3a70 con 0x55e2c8278370 2024-05-09T09:38:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.326+0000 7f625eff5640 10 client.26855 handle_command_reply: tid=0 2024-05-09T09:38:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.326+0000 7f62841b5740 1 client.26855 shutdown 2024-05-09T09:38:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.328+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 msgr2=0x55e2c826c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.327 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.328+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c826c4f0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6274009fd0 tx=0x7f6274004a60 comp rx=0 tx=0).stop 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 shutdown_connections 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c8278370 0x55e2c827a810 crc :-1 s=READY pgs=234 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8267db0 0x55e2c8273a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8262c40 0x55e2c826ca30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 --2- 172.21.15.69:0/3565294960 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c826c4f0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.331+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 >> 172.21.15.69:0/3565294960 conn(0x55e2c825d8c0 msgr2=0x55e2c825ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.332+0000 7f625eff5640 0 client.26855 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.333+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 shutdown_connections 2024-05-09T09:38:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.333+0000 7f62841b5740 1 -- 172.21.15.69:0/3565294960 wait complete. 2024-05-09T09:38:33.333 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.334+0000 7f62841b5740 20 client.26855 trim_cache size 0 max 16384 2024-05-09T09:38:33.370 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.370+0000 7f62841b5740 1 Processor -- start 2024-05-09T09:38:33.372 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 -- start start 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262250 0x55e2c826c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c825c170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1b32b0 con 0x55e2c8267db0 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1b0b80 con 0x55e2c8262c40 2024-05-09T09:38:33.373 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.371+0000 7f62841b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1ae9d0 con 0x55e2c8262250 2024-05-09T09:38:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.374+0000 7f627affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.374+0000 7f627affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58944/0 (socket says 172.21.15.69:58944) 2024-05-09T09:38:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.374+0000 7f627affd640 1 -- 172.21.15.69:0/1892884324 learned_addr learned my addr 172.21.15.69:0/1892884324 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.374+0000 7f627a7fc640 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262250 0x55e2c826c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.375 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.374+0000 7f627a7fc640 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c825c170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.377 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.376+0000 7f627affd640 1 -- 172.21.15.69:0/1892884324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262250 msgr2=0x55e2c826c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.376+0000 7f627affd640 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262250 0x55e2c826c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.376+0000 7f627affd640 1 -- 172.21.15.69:0/1892884324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 msgr2=0x55e2c825c170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.376+0000 7f627affd640 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c825c170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.378 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.376+0000 7f627affd640 1 -- 172.21.15.69:0/1892884324 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55e2c8263a50 con 0x55e2c8262c40 2024-05-09T09:38:33.381 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.379+0000 7f627affd640 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f626c00e5a0 tx=0x7f626c008ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=721b2c9963af2e29 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.383+0000 7f6260ff9640 1 -- 172.21.15.69:0/1892884324 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f626c017040 con 0x55e2c8262c40 2024-05-09T09:38:33.382 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.383+0000 7f6260ff9640 1 -- 172.21.15.69:0/1892884324 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626c005e50 con 0x55e2c8262c40 2024-05-09T09:38:33.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.384+0000 7f62841b5740 1 -- 172.21.15.69:0/1892884324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 msgr2=0x55e2c8273a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:33.383 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.384+0000 7f62841b5740 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f626c00e5a0 tx=0x7f626c008ff0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.389+0000 7f62841b5740 1 -- 172.21.15.69:0/1892884324 shutdown_connections 2024-05-09T09:38:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.389+0000 7f62841b5740 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c825c170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.389+0000 7f62841b5740 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262c40 0x55e2c8273a30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.389+0000 7f62841b5740 1 --2- 172.21.15.69:0/1892884324 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262250 0x55e2c826c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.389+0000 7f62841b5740 1 -- 172.21.15.69:0/1892884324 >> 172.21.15.69:0/1892884324 conn(0x55e2c825d8c0 msgr2=0x55e2c826c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:33.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.395+0000 7f62841b5740 1 -- 172.21.15.69:0/1892884324 shutdown_connections 2024-05-09T09:38:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.398+0000 7f62841b5740 1 -- 172.21.15.69:0/1892884324 wait complete. 2024-05-09T09:38:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.398+0000 7f62841b5740 1 -- start start 2024-05-09T09:38:33.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.398+0000 7f62841b5740 1 Processor -- start 2024-05-09T09:38:33.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8270ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c82dc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1af040 con 0x55e2c8267db0 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1b04b0 con 0x55e2c8262250 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f627c1af710 con 0x55e2c8262c40 2024-05-09T09:38:33.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.406+0000 7f62841b5740 10 client.0 _ll_register_callbacks cb 0x7f627400a4e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:33.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.407+0000 7f627a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.407+0000 7f627a7fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58954/0 (socket says 172.21.15.69:58954) 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.407+0000 7f627a7fc640 1 -- 172.21.15.69:0/4204546314 learned_addr learned my addr 172.21.15.69:0/4204546314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.407+0000 7f627a7fc640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c82dc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627affd640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8270ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627a7fc640 1 -- 172.21.15.69:0/4204546314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 msgr2=0x55e2c8270ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627a7fc640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8270ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627a7fc640 1 -- 172.21.15.69:0/4204546314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 msgr2=0x55e2c82dc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627a7fc640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c82dc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:33.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.409+0000 7f627a7fc640 1 -- 172.21.15.69:0/4204546314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55e2c82dcef0 con 0x55e2c8262250 2024-05-09T09:38:33.410 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.411+0000 7f627a7fc640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f627401b6c0 tx=0x7f627400a590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e521d70d0cd78ded server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:33.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.412+0000 7f625eff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:38:33.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.412+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f62740198e0 con 0x55e2c8262250 2024-05-09T09:38:33.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.412+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f626c005e50 con 0x55e2c8262250 2024-05-09T09:38:33.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.412+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6274019a80 con 0x55e2c8262250 2024-05-09T09:38:33.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.417+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f6274017950 con 0x55e2c8262250 2024-05-09T09:38:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.418+0000 7f62841b5740 10 client.26861 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:33.418 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.418+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55e2c82de1e0 con 0x55e2c8262250 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.421+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f626c005e50 con 0x55e2c8262250 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.421+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f627403a030 con 0x55e2c8262250 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 10 client.26861 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 10 client.26861 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 20 client.26861 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c82deb30 0x55e2c82e0f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:33.421 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 4 client.26861 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:38:33.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55e2c81cddc0 con 0x55e2c82deb30 2024-05-09T09:38:33.422 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.422+0000 7f627affd640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c82deb30 0x55e2c82e0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:33.435 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.436+0000 7f627affd640 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c82deb30 0x55e2c82e0f50 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4bdbe99d765b3b2 server_cookie=208cbce96d5d8ee in_seq=0 out_seq=0 2024-05-09T09:38:33.437 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:33.438+0000 7f625eff5640 10 client.26861 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:33 smithi069 ceph-mon[30123]: pgmap v959: 129 pgs: 129 active+clean; 686 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 512 op/s 2024-05-09T09:38:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:33 smithi163 ceph-mon[34354]: pgmap v959: 129 pgs: 129 active+clean; 686 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 512 op/s 2024-05-09T09:38:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:33 smithi103 ceph-mon[34421]: pgmap v959: 129 pgs: 129 active+clean; 686 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.0 MiB/s wr, 512 op/s 2024-05-09T09:38:33.833 INFO:tasks.workunit.client.0.smithi069.stdout: 10 210647 28.46 MB/sec execute 271 sec latency 1889.965 ms 2024-05-09T09:38:34.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 210780 28.39 MB/sec execute 272 sec latency 947.788 ms 2024-05-09T09:38:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f625eff5640 1 -- 172.21.15.69:0/4204546314 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55e2c81cddc0 con 0x55e2c82deb30 2024-05-09T09:38:34.943 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f625eff5640 10 client.26861 handle_command_reply: tid=0 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 client.26861 shutdown 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 msgr2=0x55e2c82709b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f627401b6c0 tx=0x7f627400a590 comp rx=0 tx=0).stop 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 shutdown_connections 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55e2c8267db0 0x55e2c82dc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55e2c82deb30 0x55e2c82e0f50 crc :-1 s=READY pgs=235 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55e2c8262c40 0x55e2c8270ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.943+0000 7f62841b5740 1 --2- 172.21.15.69:0/4204546314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55e2c8262250 0x55e2c82709b0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.944+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 >> 172.21.15.69:0/4204546314 conn(0x55e2c825d8c0 msgr2=0x55e2c8260640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.944+0000 7f625eff5640 0 client.26861 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.944+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 shutdown_connections 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.944+0000 7f62841b5740 1 -- 172.21.15.69:0/4204546314 wait complete. 2024-05-09T09:38:34.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.944+0000 7f62841b5740 20 client.26861 trim_cache size 0 max 16384 2024-05-09T09:38:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:38:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:38:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:38:34.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:38:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 msgr2=0x7f62640a5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:34.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 0x7f62640a5200 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f62700c8d40 tx=0x7f6270040510 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 msgr2=0x7f627c1a57c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6270004cd0 tx=0x7f6270004d00 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 shutdown_connections 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f62640a2d40 0x7f62640a5200 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f627c10db50 0x7f627c1a57c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f627c10b1a0 0x7f627c19e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 --2- 172.21.15.69:0/4132464812 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f627c108830 0x7f627c19e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 >> 172.21.15.69:0/4132464812 conn(0x7f627c06e8a0 msgr2=0x7f627c10b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 shutdown_connections 2024-05-09T09:38:34.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:34.945+0000 7f62813c8640 1 -- 172.21.15.69:0/4132464812 wait complete. 2024-05-09T09:38:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:35 smithi069 ceph-mon[30123]: pgmap v960: 129 pgs: 129 active+clean; 714 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.7 MiB/s wr, 592 op/s 2024-05-09T09:38:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:35 smithi163 ceph-mon[34354]: pgmap v960: 129 pgs: 129 active+clean; 714 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.7 MiB/s wr, 592 op/s 2024-05-09T09:38:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:35 smithi103 ceph-mon[34421]: pgmap v960: 129 pgs: 129 active+clean; 714 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 6.7 MiB/s wr, 592 op/s 2024-05-09T09:38:35.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 211071 28.34 MB/sec execute 273 sec latency 1125.759 ms 2024-05-09T09:38:36.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 211376 28.30 MB/sec execute 274 sec latency 155.553 ms 2024-05-09T09:38:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:37 smithi069 ceph-mon[30123]: pgmap v961: 129 pgs: 129 active+clean; 747 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 538 op/s 2024-05-09T09:38:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:37 smithi163 ceph-mon[34354]: pgmap v961: 129 pgs: 129 active+clean; 747 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 538 op/s 2024-05-09T09:38:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:37 smithi103 ceph-mon[34421]: pgmap v961: 129 pgs: 129 active+clean; 747 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 10 MiB/s wr, 538 op/s 2024-05-09T09:38:37.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 211749 28.28 MB/sec execute 275 sec latency 119.990 ms 2024-05-09T09:38:38.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 212296 28.31 MB/sec execute 276 sec latency 166.016 ms 2024-05-09T09:38:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:39 smithi069 ceph-mon[30123]: pgmap v962: 129 pgs: 129 active+clean; 742 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-05-09T09:38:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:39 smithi163 ceph-mon[34354]: pgmap v962: 129 pgs: 129 active+clean; 742 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-05-09T09:38:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:39 smithi103 ceph-mon[34421]: pgmap v962: 129 pgs: 129 active+clean; 742 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 10 MiB/s wr, 493 op/s 2024-05-09T09:38:39.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 212867 28.33 MB/sec execute 277 sec latency 54.236 ms 2024-05-09T09:38:40.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 213226 28.30 MB/sec execute 278 sec latency 84.675 ms 2024-05-09T09:38:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:41 smithi069 ceph-mon[30123]: pgmap v963: 129 pgs: 129 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-05-09T09:38:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:41 smithi163 ceph-mon[34354]: pgmap v963: 129 pgs: 129 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-05-09T09:38:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:41 smithi103 ceph-mon[34421]: pgmap v963: 129 pgs: 129 active+clean; 695 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 13 MiB/s wr, 654 op/s 2024-05-09T09:38:41.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 213789 28.26 MB/sec execute 279 sec latency 69.654 ms 2024-05-09T09:38:42.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 214508 28.30 MB/sec execute 280 sec latency 99.266 ms 2024-05-09T09:38:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:43 smithi069 ceph-mon[30123]: pgmap v964: 129 pgs: 129 active+clean; 662 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-05-09T09:38:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:43 smithi163 ceph-mon[34354]: pgmap v964: 129 pgs: 129 active+clean; 662 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-05-09T09:38:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:43 smithi103 ceph-mon[34421]: pgmap v964: 129 pgs: 129 active+clean; 662 MiB data, 8.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 13 MiB/s wr, 737 op/s 2024-05-09T09:38:43.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 215199 28.34 MB/sec execute 281 sec latency 65.381 ms 2024-05-09T09:38:44.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 215930 28.38 MB/sec execute 282 sec latency 55.657 ms 2024-05-09T09:38:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:45 smithi069 ceph-mon[30123]: pgmap v965: 129 pgs: 129 active+clean; 669 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 810 op/s 2024-05-09T09:38:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:45 smithi163 ceph-mon[34354]: pgmap v965: 129 pgs: 129 active+clean; 669 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 810 op/s 2024-05-09T09:38:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:45 smithi103 ceph-mon[34421]: pgmap v965: 129 pgs: 129 active+clean; 669 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 13 MiB/s wr, 810 op/s 2024-05-09T09:38:45.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 216448 28.41 MB/sec execute 283 sec latency 64.707 ms 2024-05-09T09:38:46.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 217098 28.40 MB/sec execute 284 sec latency 185.915 ms 2024-05-09T09:38:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:47 smithi069 ceph-mon[30123]: pgmap v966: 129 pgs: 129 active+clean; 672 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.7 MiB/s wr, 672 op/s 2024-05-09T09:38:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:47 smithi163 ceph-mon[34354]: pgmap v966: 129 pgs: 129 active+clean; 672 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.7 MiB/s wr, 672 op/s 2024-05-09T09:38:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:47 smithi103 ceph-mon[34421]: pgmap v966: 129 pgs: 129 active+clean; 672 MiB data, 8.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.3 MiB/s rd, 9.7 MiB/s wr, 672 op/s 2024-05-09T09:38:47.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 217635 28.43 MB/sec execute 285 sec latency 78.016 ms 2024-05-09T09:38:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 -- 172.21.15.69:0/2217966401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c100100 msgr2=0x7f480c102530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 --2- 172.21.15.69:0/2217966401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c100100 0x7f480c102530 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f47fc009860 tx=0x7f47fc02a340 comp rx=0 tx=0).stop 2024-05-09T09:38:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 -- 172.21.15.69:0/2217966401 shutdown_connections 2024-05-09T09:38:48.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 --2- 172.21.15.69:0/2217966401 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c105420 0x7f480c1058a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 --2- 172.21.15.69:0/2217966401 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c102a70 0x7f480c104ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 --2- 172.21.15.69:0/2217966401 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c100100 0x7f480c102530 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 -- 172.21.15.69:0/2217966401 >> 172.21.15.69:0/2217966401 conn(0x7f480c0fbd00 msgr2=0x7f480c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 -- 172.21.15.69:0/2217966401 shutdown_connections 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.106+0000 7f48127ce640 1 -- 172.21.15.69:0/2217966401 wait complete. 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 Processor -- start 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 -- start start 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 0x7f480c19e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c105420 0x7f480c1a5650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f480c113ad0 con 0x7f480c100100 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f4803fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f4803fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:54684/0 (socket says 172.21.15.69:54684) 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f4803fff640 1 -- 172.21.15.69:0/1623768248 learned_addr learned my addr 172.21.15.69:0/1623768248 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f480bfff640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 0x7f480c19e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f480c113950 con 0x7f480c105420 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.107+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f480c113c50 con 0x7f480c102a70 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4810d6b640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c105420 0x7f480c1a5650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 -- 172.21.15.69:0/1623768248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c105420 msgr2=0x7f480c1a5650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c105420 0x7f480c1a5650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 -- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 msgr2=0x7f480c19e0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 0x7f480c19e0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 -- 172.21.15.69:0/1623768248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f480c1a5cc0 con 0x7f480c102a70 2024-05-09T09:38:48.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f480bfff640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 0x7f480c19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4803fff640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47f400e9f0 tx=0x7f47f400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4809ffb640 1 -- 172.21.15.69:0/1623768248 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f400cd80 con 0x7f480c102a70 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4809ffb640 1 -- 172.21.15.69:0/1623768248 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f4010910 con 0x7f480c102a70 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f480c1a5fb0 con 0x7f480c102a70 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.108+0000 7f4809ffb640 1 -- 172.21.15.69:0/1623768248 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4018990 con 0x7f480c102a70 2024-05-09T09:38:48.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.109+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f480c1a64c0 con 0x7f480c102a70 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.111+0000 7f4809ffb640 1 -- 172.21.15.69:0/1623768248 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f47f4010ac0 con 0x7f480c102a70 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.111+0000 7f48155b8740 1 Processor -- start 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.111+0000 7f48155b8740 1 -- start start 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 0x55fb20904670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20909740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 0x55fb2090c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fb20868340 con 0x55fb20904c40 2024-05-09T09:38:48.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fb20810290 con 0x55fb20904250 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f48155b8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fb20864300 con 0x55fb20909db0 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4810d6b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 0x55fb20904670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4810d6b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 0x55fb20904670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:52098/0 (socket says 172.21.15.69:52098) 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4810d6b640 1 -- 172.21.15.69:0/3537988779 learned_addr learned my addr 172.21.15.69:0/3537988779 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4809ffb640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 0x7f47f80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f480bfff640 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20909740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 0x55fb2090c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4810d6b640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 0x7f47f80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4809ffb640 1 -- 172.21.15.69:0/1623768248 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f47f40cf640 con 0x7f480c102a70 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 -- 172.21.15.69:0/3537988779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 msgr2=0x55fb20904670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 0x55fb20904670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 -- 172.21.15.69:0/3537988779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 msgr2=0x55fb20909740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20909740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.112+0000 7f4803fff640 1 -- 172.21.15.69:0/3537988779 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb208348e0 con 0x55fb20909db0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f4810d6b640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 0x7f47f80a5170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f480400ed30 tx=0x7f4804009cd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f4803fff640 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 0x55fb2090c240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47f40cce10 tx=0x7f47f4086750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4605f82ccbb9d0c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f48027fc640 1 -- 172.21.15.69:0/3537988779 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f409bbd0 con 0x55fb20909db0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f48027fc640 1 -- 172.21.15.69:0/3537988779 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f409fed0 con 0x55fb20909db0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f48027fc640 1 -- 172.21.15.69:0/3537988779 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f4098240 con 0x55fb20909db0 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f48155b8740 1 -- 172.21.15.69:0/3537988779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 msgr2=0x55fb2090c240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.113+0000 7f48155b8740 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 0x55fb2090c240 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f47f40cce10 tx=0x7f47f4086750 comp rx=0 tx=0).stop 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 -- 172.21.15.69:0/3537988779 shutdown_connections 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20909db0 0x55fb2090c240 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20909740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 --2- 172.21.15.69:0/3537988779 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20904250 0x55fb20904670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 -- 172.21.15.69:0/3537988779 >> 172.21.15.69:0/3537988779 conn(0x55fb208ff8c0 msgr2=0x55fb208ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 -- 172.21.15.69:0/3537988779 shutdown_connections 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 -- 172.21.15.69:0/3537988779 wait complete. 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 -- start start 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 Processor -- start 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb2090e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.114+0000 7f48155b8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.125 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fb2090cd00 con 0x55fb20904c40 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39314/0 (socket says 172.21.15.69:39314) 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f480bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb2090e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 -- 172.21.15.69:0/550369833 learned_addr learned my addr 172.21.15.69:0/550369833 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fb20823790 con 0x55fb20909db0 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fb2090ce80 con 0x55fb20904250 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4810d6b640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.126 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 10 client.0 _ll_register_callbacks cb 0x55fb2085ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 -- 172.21.15.69:0/550369833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 msgr2=0x55fb2090e3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb2090e3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 -- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 msgr2=0x55fb20915950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb2057c210 con 0x55fb20904c40 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f480bfff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb2090e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f4803fff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f47f400e9c0 tx=0x7f47f40a4850 comp rx=0 tx=0).ready entity=mon.0 client_cookie=406d8b932e17da2d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f47eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:38:48.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47f40863c0 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47f409fed0 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.115+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f47f40a26c0 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f47f4086560 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 10 client.17601 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fb20648620 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47f409fed0 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f47f409b020 con 0x55fb20904c40 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 10 client.17601 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 10 client.17601 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 20 client.17601 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20919a90 0x55fb2091be80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 4 client.17601 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:38:48.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fb20875a70 con 0x55fb20919a90 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.116+0000 7f480bfff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20919a90 0x55fb2091be80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.117+0000 7f480bfff640 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20919a90 0x55fb2091be80 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3334ffc6ec357c59 server_cookie=e6971e87fc5229fa in_seq=0 out_seq=0 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.117+0000 7f47eb7fe640 10 client.17601 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f47eb7fe640 1 -- 172.21.15.69:0/550369833 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fb20875a70 con 0x55fb20919a90 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f47eb7fe640 10 client.17601 handle_command_reply: tid=0 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 client.17601 shutdown 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 msgr2=0x55fb2090e910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7f47f400e9c0 tx=0x7f47f40a4850 comp rx=0 tx=0).stop 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 shutdown_connections 2024-05-09T09:38:48.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb2090e910 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20919a90 0x55fb2091be80 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 --2- 172.21.15.69:0/550369833 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb2090e3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f47eb7fe640 0 client.17601 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.119+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 >> 172.21.15.69:0/550369833 conn(0x55fb208ff8c0 msgr2=0x55fb208fea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.120+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 shutdown_connections 2024-05-09T09:38:48.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.120+0000 7f48155b8740 1 -- 172.21.15.69:0/550369833 wait complete. 2024-05-09T09:38:48.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.120+0000 7f48155b8740 20 client.17601 trim_cache size 0 max 16384 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 Processor -- start 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 -- start start 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb209216a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20921be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1b3260 con 0x55fb20904c40 2024-05-09T09:38:48.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1b0b30 con 0x55fb20909db0 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f48155b8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1ae980 con 0x55fb20904250 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f4810d6b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f4803fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20921be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f4803fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20921be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:39316/0 (socket says 172.21.15.69:39316) 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f4803fff640 1 -- 172.21.15.69:0/50821500 learned_addr learned my addr 172.21.15.69:0/50821500 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f480bfff640 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb209216a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.139+0000 7f4810d6b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:52114/0 (socket says 172.21.15.69:52114) 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 -- 172.21.15.69:0/50821500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 msgr2=0x55fb20915910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 -- 172.21.15.69:0/50821500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 msgr2=0x55fb20921be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20921be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 -- 172.21.15.69:0/50821500 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fb20905020 con 0x55fb20904250 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f480bfff640 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb209216a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47fc030410 tx=0x7f47fc030570 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4d1e7a3363590317 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f4801ffb640 1 -- 172.21.15.69:0/50821500 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc04d070 con 0x55fb20904250 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f4801ffb640 1 -- 172.21.15.69:0/50821500 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47fc005520 con 0x55fb20904250 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.140+0000 7f4801ffb640 1 -- 172.21.15.69:0/50821500 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc037040 con 0x55fb20904250 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 -- 172.21.15.69:0/50821500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 msgr2=0x55fb209216a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb209216a0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47fc030410 tx=0x7f47fc030570 comp rx=0 tx=0).stop 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 -- 172.21.15.69:0/50821500 shutdown_connections 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb20909db0 0x55fb20915910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20904c40 0x55fb20921be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 --2- 172.21.15.69:0/50821500 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904250 0x55fb209216a0 secure :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f47fc030410 tx=0x7f47fc030570 comp rx=0 tx=0).stop 2024-05-09T09:38:48.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.141+0000 7f48155b8740 1 -- 172.21.15.69:0/50821500 >> 172.21.15.69:0/50821500 conn(0x55fb208ff8c0 msgr2=0x55fb20909460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.142+0000 7f48155b8740 1 -- 172.21.15.69:0/50821500 shutdown_connections 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.142+0000 7f48155b8740 1 -- 172.21.15.69:0/50821500 wait complete. 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.142+0000 7f48155b8740 1 -- start start 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 Processor -- start 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 0x55fb20913420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb209143e0 0x55fb20914860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1aeff0 con 0x55fb20909db0 2024-05-09T09:38:48.149 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1b0460 con 0x55fb209143e0 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f480c1af6c0 con 0x55fb20904c40 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f48155b8740 10 client.0 _ll_register_callbacks cb 0x7f4804009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f4803fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 0x55fb20913420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:54716/0 (socket says 172.21.15.69:54716) 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 -- 172.21.15.69:0/747974899 learned_addr learned my addr 172.21.15.69:0/747974899 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f4810d6b640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb209143e0 0x55fb20914860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 -- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb209143e0 msgr2=0x55fb20914860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb209143e0 0x55fb20914860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 -- 172.21.15.69:0/747974899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 msgr2=0x55fb20913420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:38:48.150 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 0x55fb20913420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.143+0000 7f480bfff640 1 -- 172.21.15.69:0/747974899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fb20914da0 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f4803fff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 0x55fb20913420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f480bfff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47fc031000 tx=0x7f47fc030b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=95bf1b38942e2cfa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f47eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f47fc055060 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f47fc005520 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f47fc0041e0 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f47fc05c050 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f48155b8740 10 client.26794 fetch_fsmap learned FSMap version 47 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.144+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55fb20980190 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f47fc005520 con 0x55fb20904c40 2024-05-09T09:38:48.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f47fc032440 con 0x55fb20904c40 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 10 client.26794 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 10 client.26794 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 20 client.26794 populate_metadata read hostname 'smithi069' 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20980990 0x55fb20982db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 4 client.26794 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fb2086fdc0 con 0x55fb20980990 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.145+0000 7f4803fff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20980990 0x55fb20982db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.146+0000 7f4803fff640 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20980990 0x55fb20982db0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=286ea788cfb7f3ba server_cookie=5402b24a2a04c429 in_seq=0 out_seq=0 2024-05-09T09:38:48.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.146+0000 7f47eb7fe640 10 client.26794 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.754+0000 7f47eb7fe640 1 -- 172.21.15.69:0/747974899 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fb2086fdc0 con 0x55fb20980990 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.754+0000 7f47eb7fe640 10 client.26794 handle_command_reply: tid=0 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.754+0000 7f48155b8740 1 client.26794 shutdown 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.754+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 msgr2=0x55fb20912ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.754+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f47fc031000 tx=0x7f47fc030b00 comp rx=0 tx=0).stop 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 shutdown_connections 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fb209143e0 0x55fb20914860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fb20909db0 0x55fb20913420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.756 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fb20980990 0x55fb20982db0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 --2- 172.21.15.69:0/747974899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fb20904c40 0x55fb20912ee0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f47eb7fe640 0 client.26794 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 >> 172.21.15.69:0/747974899 conn(0x55fb208ff8c0 msgr2=0x55fb2091baa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 shutdown_connections 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 1 -- 172.21.15.69:0/747974899 wait complete. 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.755+0000 7f48155b8740 20 client.26794 trim_cache size 0 max 16384 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:38:48.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:38:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:38:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 msgr2=0x7f47f80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 0x7f47f80a5170 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f480400ed30 tx=0x7f4804009cd0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 msgr2=0x7f480c19e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:38:48.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f47f400e9f0 tx=0x7f47f400eeb0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 shutdown_connections 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f47f80a2cb0 0x7f47f80a5170 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f480c105420 0x7f480c1a5650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f480c102a70 0x7f480c19e5e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 --2- 172.21.15.69:0/1623768248 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f480c100100 0x7f480c19e0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.756+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 >> 172.21.15.69:0/1623768248 conn(0x7f480c0fbd00 msgr2=0x7f480c1033b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.757+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 shutdown_connections 2024-05-09T09:38:48.759 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:38:48.757+0000 7f48127ce640 1 -- 172.21.15.69:0/1623768248 wait complete. 2024-05-09T09:38:48.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 217930 28.37 MB/sec execute 286 sec latency 451.235 ms 2024-05-09T09:38:49.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:49 smithi069 ceph-mon[30123]: pgmap v967: 129 pgs: 129 active+clean; 675 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-05-09T09:38:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:49 smithi163 ceph-mon[34354]: pgmap v967: 129 pgs: 129 active+clean; 675 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-05-09T09:38:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:49 smithi103 ceph-mon[34421]: pgmap v967: 129 pgs: 129 active+clean; 675 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 583 op/s 2024-05-09T09:38:49.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 218436 28.37 MB/sec execute 287 sec latency 166.963 ms 2024-05-09T09:38:50.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 218988 28.37 MB/sec execute 288 sec latency 132.715 ms 2024-05-09T09:38:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:51 smithi069 ceph-mon[30123]: pgmap v968: 129 pgs: 129 active+clean; 683 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.1 MiB/s wr, 651 op/s 2024-05-09T09:38:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:51 smithi163 ceph-mon[34354]: pgmap v968: 129 pgs: 129 active+clean; 683 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.1 MiB/s wr, 651 op/s 2024-05-09T09:38:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:51 smithi103 ceph-mon[34421]: pgmap v968: 129 pgs: 129 active+clean; 683 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 6.1 MiB/s wr, 651 op/s 2024-05-09T09:38:51.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 219602 28.38 MB/sec execute 289 sec latency 85.874 ms 2024-05-09T09:38:52.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 220262 28.41 MB/sec execute 290 sec latency 88.411 ms 2024-05-09T09:38:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:53 smithi069 ceph-mon[30123]: pgmap v969: 129 pgs: 129 active+clean; 674 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 517 op/s 2024-05-09T09:38:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:53 smithi163 ceph-mon[34354]: pgmap v969: 129 pgs: 129 active+clean; 674 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 517 op/s 2024-05-09T09:38:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:53 smithi103 ceph-mon[34421]: pgmap v969: 129 pgs: 129 active+clean; 674 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 517 op/s 2024-05-09T09:38:53.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 220810 28.41 MB/sec execute 291 sec latency 73.534 ms 2024-05-09T09:38:54.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 221435 28.42 MB/sec execute 292 sec latency 148.712 ms 2024-05-09T09:38:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:55 smithi069 ceph-mon[30123]: pgmap v970: 129 pgs: 129 active+clean; 664 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 493 op/s 2024-05-09T09:38:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:55 smithi163 ceph-mon[34354]: pgmap v970: 129 pgs: 129 active+clean; 664 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 493 op/s 2024-05-09T09:38:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:55 smithi103 ceph-mon[34421]: pgmap v970: 129 pgs: 129 active+clean; 664 MiB data, 9.0 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 493 op/s 2024-05-09T09:38:55.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 222033 28.47 MB/sec execute 293 sec latency 58.221 ms 2024-05-09T09:38:56.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 222591 28.46 MB/sec execute 294 sec latency 64.434 ms 2024-05-09T09:38:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:57 smithi069 ceph-mon[30123]: pgmap v971: 129 pgs: 129 active+clean; 672 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:38:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:57 smithi163 ceph-mon[34354]: pgmap v971: 129 pgs: 129 active+clean; 672 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:38:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:57 smithi103 ceph-mon[34421]: pgmap v971: 129 pgs: 129 active+clean; 672 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:38:57.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 223056 28.43 MB/sec execute 295 sec latency 127.345 ms 2024-05-09T09:38:58.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 223549 28.44 MB/sec execute 296 sec latency 107.756 ms 2024-05-09T09:38:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:38:59 smithi069 ceph-mon[30123]: pgmap v972: 129 pgs: 129 active+clean; 673 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 479 op/s 2024-05-09T09:38:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:38:59 smithi163 ceph-mon[34354]: pgmap v972: 129 pgs: 129 active+clean; 673 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 479 op/s 2024-05-09T09:38:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:38:59 smithi103 ceph-mon[34421]: pgmap v972: 129 pgs: 129 active+clean; 673 MiB data, 9.1 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.1 MiB/s wr, 479 op/s 2024-05-09T09:38:59.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 224305 28.45 MB/sec execute 297 sec latency 69.169 ms 2024-05-09T09:39:00.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 224889 28.47 MB/sec execute 298 sec latency 78.201 ms 2024-05-09T09:39:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:01 smithi069 ceph-mon[30123]: pgmap v973: 129 pgs: 129 active+clean; 680 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 514 op/s 2024-05-09T09:39:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:01 smithi163 ceph-mon[34354]: pgmap v973: 129 pgs: 129 active+clean; 680 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 514 op/s 2024-05-09T09:39:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:01 smithi103 ceph-mon[34421]: pgmap v973: 129 pgs: 129 active+clean; 680 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.5 MiB/s wr, 514 op/s 2024-05-09T09:39:01.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 225553 28.53 MB/sec execute 299 sec latency 52.654 ms 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 -- 172.21.15.69:0/3583766477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c100790 msgr2=0x7fb33c100bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 --2- 172.21.15.69:0/3583766477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c100790 0x7fb33c100bb0 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7fb338009900 tx=0x7fb33802a410 comp rx=0 tx=0).stop 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 -- 172.21.15.69:0/3583766477 shutdown_connections 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 --2- 172.21.15.69:0/3583766477 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c102340 0x7fb33c108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 --2- 172.21.15.69:0/3583766477 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c101980 0x7fb33c101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 --2- 172.21.15.69:0/3583766477 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c100790 0x7fb33c100bb0 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.858 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.856+0000 7fb344763640 1 -- 172.21.15.69:0/3583766477 >> 172.21.15.69:0/3583766477 conn(0x7fb33c0fbd00 msgr2=0x7fb33c0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- 172.21.15.69:0/3583766477 shutdown_connections 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- 172.21.15.69:0/3583766477 wait complete. 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 Processor -- start 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- start start 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c100790 0x7fb33c197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 0x7fb33c19f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c10d530 con 0x7fb33c101980 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c10d3b0 con 0x7fb33c100790 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb344763640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c10d6b0 con 0x7fb33c102340 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55946/0 (socket says 172.21.15.69:55946) 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb341cfe640 1 -- 172.21.15.69:0/935570541 learned_addr learned my addr 172.21.15.69:0/935570541 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:01.859 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb3424ff640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c100790 0x7fb33c197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.857+0000 7fb342d00640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 0x7fb33c19f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 -- 172.21.15.69:0/935570541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 msgr2=0x7fb33c19f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 0x7fb33c19f0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 -- 172.21.15.69:0/935570541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c100790 msgr2=0x7fb33c197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c100790 0x7fb33c197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 -- 172.21.15.69:0/935570541 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb33c19f750 con 0x7fb33c101980 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb342d00640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 0x7fb33c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb341cfe640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00e9f0 tx=0x7fb32c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb3337fe640 1 -- 172.21.15.69:0/935570541 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c00cd80 con 0x7fb33c101980 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb3337fe640 1 -- 172.21.15.69:0/935570541 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c005500 con 0x7fb33c101980 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb3337fe640 1 -- 172.21.15.69:0/935570541 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c010690 con 0x7fb33c101980 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb33c19fa40 con 0x7fb33c101980 2024-05-09T09:39:01.860 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.858+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb33c19ff40 con 0x7fb33c101980 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.860+0000 7fb3337fe640 1 -- 172.21.15.69:0/935570541 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb32c0046c0 con 0x7fb33c101980 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.860+0000 7fb347473740 1 Processor -- start 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 -- start start 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe889a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe88eb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563bbe7e9340 con 0x563bbe88f1c0 2024-05-09T09:39:01.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563bbe791290 con 0x563bbe889660 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb347473740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563bbe7e5300 con 0x563bbe88a050 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55956/0 (socket says 172.21.15.69:55956) 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 -- 172.21.15.69:0/1441072991 learned_addr learned my addr 172.21.15.69:0/1441072991 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb3424ff640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe88eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb342d00640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe889a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb3337fe640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 0x7fb3280a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb342d00640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 0x7fb3280a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb3337fe640 1 -- 172.21.15.69:0/935570541 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb32c0cf890 con 0x7fb33c101980 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 -- 172.21.15.69:0/1441072991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 msgr2=0x563bbe88eb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe88eb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 -- 172.21.15.69:0/1441072991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 msgr2=0x563bbe889a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe889a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb341cfe640 1 -- 172.21.15.69:0/1441072991 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bbe7b58e0 con 0x563bbe88f1c0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb342d00640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe889a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.861+0000 7fb3424ff640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe88eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb341cfe640 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb32c09ba70 tx=0x7fb32c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e81ca3d640219e15 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb330ff9640 1 -- 172.21.15.69:0/1441072991 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c09b6d0 con 0x563bbe88f1c0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb342d00640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 0x7fb3280a50f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x563bbe88a8b0 tx=0x7fb334004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb330ff9640 1 -- 172.21.15.69:0/1441072991 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c09c470 con 0x563bbe88f1c0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb330ff9640 1 -- 172.21.15.69:0/1441072991 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0a4550 con 0x563bbe88f1c0 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb347473740 1 -- 172.21.15.69:0/1441072991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 msgr2=0x563bbe891650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.862+0000 7fb347473740 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7fb32c09ba70 tx=0x7fb32c004a40 comp rx=0 tx=0).stop 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- 172.21.15.69:0/1441072991 shutdown_connections 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe891650 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe88eb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- 172.21.15.69:0/1441072991 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe889a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- 172.21.15.69:0/1441072991 >> 172.21.15.69:0/1441072991 conn(0x563bbe8808c0 msgr2=0x563bbe880cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- 172.21.15.69:0/1441072991 shutdown_connections 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- 172.21.15.69:0/1441072991 wait complete. 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- start start 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 Processor -- start 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe893760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe893ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x563bbe892080 con 0x563bbe88f1c0 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x563bbe7a4790 con 0x563bbe889660 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x563bbe892200 con 0x563bbe88a050 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb347473740 10 client.0 _ll_register_callbacks cb 0x563bbe7dfa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb342d00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb342d00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55968/0 (socket says 172.21.15.69:55968) 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb342d00640 1 -- 172.21.15.69:0/295243119 learned_addr learned my addr 172.21.15.69:0/295243119 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.863+0000 7fb341cfe640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe893ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb3424ff640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe893760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 -- 172.21.15.69:0/295243119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 msgr2=0x563bbe893ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe893ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 -- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 msgr2=0x563bbe893760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe893760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 -- 172.21.15.69:0/295243119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bbe4fd210 con 0x563bbe88f1c0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb342d00640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb33400ec50 tx=0x7fb334004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e77eaa4ebbe0caa server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb325ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334024030 con 0x563bbe88f1c0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb334004b20 con 0x563bbe88f1c0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb32c09c470 con 0x563bbe88f1c0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb334004d20 con 0x563bbe88f1c0 2024-05-09T09:39:01.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb347473740 10 client.17619 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563bbe5c9620 con 0x563bbe88f1c0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.864+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c09c470 con 0x563bbe88f1c0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb33401e040 con 0x563bbe88f1c0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 10 client.17619 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 10 client.17619 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 20 client.17619 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe89f250 0x563bbe8a16a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 4 client.17619 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x563bbe7f6a70 con 0x563bbe89f250 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb3424ff640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe89f250 0x563bbe8a16a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb325ffb640 10 client.17619 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.865+0000 7fb3424ff640 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe89f250 0x563bbe8a16a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=23f39b760cf8fa49 server_cookie=39e2c570c696a457 in_seq=0 out_seq=0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.866+0000 7fb325ffb640 1 -- 172.21.15.69:0/295243119 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x563bbe7f6a70 con 0x563bbe89f250 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.866+0000 7fb325ffb640 10 client.17619 handle_command_reply: tid=0 2024-05-09T09:39:01.872 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 client.17619 shutdown 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 msgr2=0x563bbe89ace0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7fb33400ec50 tx=0x7fb334004a40 comp rx=0 tx=0).stop 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 shutdown_connections 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88f1c0 0x563bbe89ace0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe89f250 0x563bbe8a16a0 crc :-1 s=READY pgs=238 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88a050 0x563bbe893ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 --2- 172.21.15.69:0/295243119 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe893760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb325ffb640 0 client.17619 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 >> 172.21.15.69:0/295243119 conn(0x563bbe8808c0 msgr2=0x563bbe87fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 shutdown_connections 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 1 -- 172.21.15.69:0/295243119 wait complete. 2024-05-09T09:39:01.873 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.867+0000 7fb347473740 20 client.17619 trim_cache size 0 max 16384 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 Processor -- start 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 -- start start 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe889660 0x563bbe8a6fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe88f1c0 0x563bbe89aca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1aaaa0 con 0x563bbe88a050 2024-05-09T09:39:01.886 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a8370 con 0x563bbe88f1c0 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb347473740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a61c0 con 0x563bbe889660 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55976/0 (socket says 172.21.15.69:55976) 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb341cfe640 1 -- 172.21.15.69:0/504642235 learned_addr learned my addr 172.21.15.69:0/504642235 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.886+0000 7fb3424ff640 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe889660 0x563bbe8a6fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 -- 172.21.15.69:0/504642235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe889660 msgr2=0x563bbe8a6fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe889660 0x563bbe8a6fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 -- 172.21.15.69:0/504642235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe88f1c0 msgr2=0x563bbe89aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe88f1c0 0x563bbe89aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 -- 172.21.15.69:0/504642235 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x563bbe88a430 con 0x563bbe88a050 2024-05-09T09:39:01.887 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb341cfe640 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0cd380 tx=0x7fb32c0cfb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ac8eac5ce59453 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb327fff640 1 -- 172.21.15.69:0/504642235 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c0b8070 con 0x563bbe88a050 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb327fff640 1 -- 172.21.15.69:0/504642235 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c09c470 con 0x563bbe88a050 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.887+0000 7fb327fff640 1 -- 172.21.15.69:0/504642235 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb32c098570 con 0x563bbe88a050 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- 172.21.15.69:0/504642235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 msgr2=0x563bbe8a74f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0cd380 tx=0x7fb32c0cfb90 comp rx=0 tx=0).stop 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- 172.21.15.69:0/504642235 shutdown_connections 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe88f1c0 0x563bbe89aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe88a050 0x563bbe8a74f0 secure :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7fb32c0cd380 tx=0x7fb32c0cfb90 comp rx=0 tx=0).stop 2024-05-09T09:39:01.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 --2- 172.21.15.69:0/504642235 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe889660 0x563bbe8a6fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- 172.21.15.69:0/504642235 >> 172.21.15.69:0/504642235 conn(0x563bbe8808c0 msgr2=0x563bbe88e800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- 172.21.15.69:0/504642235 shutdown_connections 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- 172.21.15.69:0/504642235 wait complete. 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.888+0000 7fb347473740 1 -- start start 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 Processor -- start 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe898b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 0x563bbe899060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a6830 con 0x563bbe89a070 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a7ca0 con 0x563bbe889660 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb33c1a6f00 con 0x563bbe88f1c0 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb347473740 10 client.0 _ll_register_callbacks cb 0x7fb334009c30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 0x563bbe899060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55990/0 (socket says 172.21.15.69:55990) 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 -- 172.21.15.69:0/2703119564 learned_addr learned my addr 172.21.15.69:0/2703119564 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb341cfe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 0x563bbe899060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:44066/0 (socket says 172.21.15.69:44066) 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb3424ff640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe898b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 -- 172.21.15.69:0/2703119564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 msgr2=0x563bbe899060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 0x563bbe899060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 -- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 msgr2=0x563bbe898b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe898b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 -- 172.21.15.69:0/2703119564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x563bbe89aa30 con 0x563bbe89a070 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb3424ff640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe898b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.889+0000 7fb342d00640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb334012fd0 tx=0x7fb3340209c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=991ff6f49e9f2a6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb334024040 con 0x563bbe89a070 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb32c09c470 con 0x563bbe89a070 2024-05-09T09:39:01.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb334020a40 con 0x563bbe89a070 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb334020c60 con 0x563bbe89a070 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 10 client.17625 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x563bbe906290 con 0x563bbe89a070 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb32c09c470 con 0x563bbe89a070 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb334029ba0 con 0x563bbe89a070 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 10 client.17625 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 10 client.17625 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 20 client.17625 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe906d10 0x563bbe909130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 4 client.17625 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.890+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x563bbe7f0dc0 con 0x563bbe906d10 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.891+0000 7fb3424ff640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe906d10 0x563bbe909130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.891+0000 7fb3424ff640 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe906d10 0x563bbe909130 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=200670278d414611 server_cookie=7e1d1a1558c45452 in_seq=0 out_seq=0 2024-05-09T09:39:01.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:01.891+0000 7fb325ffb640 10 client.17625 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.761+0000 7fb325ffb640 1 -- 172.21.15.69:0/2703119564 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x563bbe7f0dc0 con 0x563bbe906d10 2024-05-09T09:39:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.761+0000 7fb325ffb640 10 client.17625 handle_command_reply: tid=0 2024-05-09T09:39:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.761+0000 7fb347473740 1 client.17625 shutdown 2024-05-09T09:39:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 msgr2=0x563bbe89a4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:02.762 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7fb334012fd0 tx=0x7fb3340209c0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 shutdown_connections 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x563bbe89a070 0x563bbe89a4f0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x563bbe906d10 0x563bbe909130 crc :-1 s=READY pgs=239 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb325ffb640 0 client.17625 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x563bbe88f1c0 0x563bbe899060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 --2- 172.21.15.69:0/2703119564 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x563bbe889660 0x563bbe898b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 >> 172.21.15.69:0/2703119564 conn(0x563bbe8808c0 msgr2=0x563bbe87f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 shutdown_connections 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 1 -- 172.21.15.69:0/2703119564 wait complete. 2024-05-09T09:39:02.763 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.762+0000 7fb347473740 20 client.17625 trim_cache size 0 max 16384 2024-05-09T09:39:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:39:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:39:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:39:02.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:39:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 msgr2=0x7fb3280a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 0x7fb3280a50f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x563bbe88a8b0 tx=0x7fb334004ab0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 msgr2=0x7fb33c198070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7fb32c00e9f0 tx=0x7fb32c00edf0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 shutdown_connections 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb3280a2c30 0x7fb3280a50f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb33c102340 0x7fb33c19f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb33c101980 0x7fb33c198070 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 --2- 172.21.15.69:0/935570541 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb33c100790 0x7fb33c197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 >> 172.21.15.69:0/935570541 conn(0x7fb33c0fbd00 msgr2=0x7fb33c104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 shutdown_connections 2024-05-09T09:39:02.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:02.763+0000 7fb344763640 1 -- 172.21.15.69:0/935570541 wait complete. 2024-05-09T09:39:02.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 225725 28.46 MB/sec execute 300 sec latency 696.068 ms 2024-05-09T09:39:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:03 smithi069 ceph-mon[30123]: pgmap v974: 129 pgs: 129 active+clean; 685 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 463 op/s 2024-05-09T09:39:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:03 smithi163 ceph-mon[34354]: pgmap v974: 129 pgs: 129 active+clean; 685 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 463 op/s 2024-05-09T09:39:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:03 smithi103 ceph-mon[34421]: pgmap v974: 129 pgs: 129 active+clean; 685 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 463 op/s 2024-05-09T09:39:03.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 226173 28.44 MB/sec execute 301 sec latency 116.608 ms 2024-05-09T09:39:04.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 226701 28.43 MB/sec execute 302 sec latency 73.466 ms 2024-05-09T09:39:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:05 smithi069 ceph-mon[30123]: pgmap v975: 129 pgs: 129 active+clean; 693 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 521 op/s 2024-05-09T09:39:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:05 smithi163 ceph-mon[34354]: pgmap v975: 129 pgs: 129 active+clean; 693 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 521 op/s 2024-05-09T09:39:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:05 smithi103 ceph-mon[34421]: pgmap v975: 129 pgs: 129 active+clean; 693 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 3.7 MiB/s wr, 521 op/s 2024-05-09T09:39:05.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 227337 28.43 MB/sec execute 303 sec latency 76.975 ms 2024-05-09T09:39:06.834 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228074 28.48 MB/sec execute 304 sec latency 77.502 ms 2024-05-09T09:39:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:07 smithi069 ceph-mon[30123]: pgmap v976: 129 pgs: 129 active+clean; 687 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.5 MiB/s wr, 528 op/s 2024-05-09T09:39:07.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:07 smithi163 ceph-mon[34354]: pgmap v976: 129 pgs: 129 active+clean; 687 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.5 MiB/s wr, 528 op/s 2024-05-09T09:39:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:07 smithi103 ceph-mon[34421]: pgmap v976: 129 pgs: 129 active+clean; 687 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.5 MiB/s wr, 528 op/s 2024-05-09T09:39:07.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228498 28.47 MB/sec execute 305 sec latency 414.425 ms 2024-05-09T09:39:08.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228509 28.38 MB/sec execute 306 sec latency 1398.532 ms 2024-05-09T09:39:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:09 smithi069 ceph-mon[30123]: pgmap v977: 129 pgs: 129 active+clean; 690 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.9 MiB/s wr, 474 op/s 2024-05-09T09:39:09.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:09 smithi163 ceph-mon[34354]: pgmap v977: 129 pgs: 129 active+clean; 690 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.9 MiB/s wr, 474 op/s 2024-05-09T09:39:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:09 smithi103 ceph-mon[34421]: pgmap v977: 129 pgs: 129 active+clean; 690 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 2.9 MiB/s wr, 474 op/s 2024-05-09T09:39:09.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228520 28.29 MB/sec execute 307 sec latency 2218.476 ms 2024-05-09T09:39:10.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228663 28.24 MB/sec execute 308 sec latency 1415.834 ms 2024-05-09T09:39:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:11 smithi069 ceph-mon[30123]: pgmap v978: 129 pgs: 129 active+clean; 715 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.3 MiB/s wr, 578 op/s 2024-05-09T09:39:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:11 smithi163 ceph-mon[34354]: pgmap v978: 129 pgs: 129 active+clean; 715 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.3 MiB/s wr, 578 op/s 2024-05-09T09:39:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:11 smithi103 ceph-mon[34421]: pgmap v978: 129 pgs: 129 active+clean; 715 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 5.3 MiB/s wr, 578 op/s 2024-05-09T09:39:11.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 228905 28.18 MB/sec execute 309 sec latency 293.251 ms 2024-05-09T09:39:12.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 229154 28.13 MB/sec execute 310 sec latency 127.902 ms 2024-05-09T09:39:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:13 smithi069 ceph-mon[30123]: pgmap v979: 129 pgs: 129 active+clean; 778 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 627 op/s 2024-05-09T09:39:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:13 smithi163 ceph-mon[34354]: pgmap v979: 129 pgs: 129 active+clean; 778 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 627 op/s 2024-05-09T09:39:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:13 smithi103 ceph-mon[34421]: pgmap v979: 129 pgs: 129 active+clean; 778 MiB data, 9.2 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 11 MiB/s wr, 627 op/s 2024-05-09T09:39:13.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 229397 28.08 MB/sec execute 311 sec latency 109.217 ms 2024-05-09T09:39:14.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 229698 28.03 MB/sec execute 312 sec latency 85.182 ms 2024-05-09T09:39:15.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:15 smithi069 ceph-mon[30123]: pgmap v980: 129 pgs: 129 active+clean; 787 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-05-09T09:39:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:15 smithi163 ceph-mon[34354]: pgmap v980: 129 pgs: 129 active+clean; 787 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-05-09T09:39:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:15 smithi103 ceph-mon[34421]: pgmap v980: 129 pgs: 129 active+clean; 787 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 12 MiB/s wr, 656 op/s 2024-05-09T09:39:15.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 230021 28.01 MB/sec execute 313 sec latency 80.881 ms 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 -- 172.21.15.69:0/3239356007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984103450 msgr2=0x7fa984109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 --2- 172.21.15.69:0/3239356007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984103450 0x7fa984109ce0 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009ca0 tx=0x7fa97c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 -- 172.21.15.69:0/3239356007 shutdown_connections 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 --2- 172.21.15.69:0/3239356007 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984103450 0x7fa984109ce0 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 --2- 172.21.15.69:0/3239356007 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984102a90 0x7fa984102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 --2- 172.21.15.69:0/3239356007 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 0x7fa984101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 -- 172.21.15.69:0/3239356007 >> 172.21.15.69:0/3239356007 conn(0x7fa9840fcdf0 msgr2=0x7fa9840ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.387 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 -- 172.21.15.69:0/3239356007 shutdown_connections 2024-05-09T09:39:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.387+0000 7fa989bf9640 1 -- 172.21.15.69:0/3239356007 wait complete. 2024-05-09T09:39:16.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 Processor -- start 2024-05-09T09:39:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 -- start start 2024-05-09T09:39:16.389 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 0x7fa984197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984103450 0x7fa98419f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410d520 con 0x7fa984102a90 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410d3a0 con 0x7fa9841018a0 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa989bf9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa98410d6a0 con 0x7fa984103450 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48288/0 (socket says 172.21.15.69:48288) 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa982ffd640 1 -- 172.21.15.69:0/1671539058 learned_addr learned my addr 172.21.15.69:0/1671539058 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa9837fe640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 0x7fa984197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.390 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa983fff640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984103450 0x7fa98419f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.388+0000 7fa982ffd640 1 -- 172.21.15.69:0/1671539058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984103450 msgr2=0x7fa98419f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa982ffd640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984103450 0x7fa98419f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa982ffd640 1 -- 172.21.15.69:0/1671539058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 msgr2=0x7fa984197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa982ffd640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 0x7fa984197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa982ffd640 1 -- 172.21.15.69:0/1671539058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa98419f6c0 con 0x7fa984102a90 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa982ffd640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e9f0 tx=0x7fa97800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa980ff9640 1 -- 172.21.15.69:0/1671539058 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97800cd80 con 0x7fa984102a90 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa98419f9b0 con 0x7fa984102a90 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa980ff9640 1 -- 172.21.15.69:0/1671539058 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa978005500 con 0x7fa984102a90 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa980ff9640 1 -- 172.21.15.69:0/1671539058 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa978010690 con 0x7fa984102a90 2024-05-09T09:39:16.391 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.389+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa98419feb0 con 0x7fa984102a90 2024-05-09T09:39:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 Processor -- start 2024-05-09T09:39:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 -- start start 2024-05-09T09:39:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa980ff9640 1 -- 172.21.15.69:0/1671539058 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa9780046c0 con 0x7fa984102a90 2024-05-09T09:39:16.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0947740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a094a240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b2a08a6340 con 0x55b2a0942250 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b2a084e290 con 0x55b2a0942c40 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b2a08a2300 con 0x55b2a0947db0 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa983fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa983fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48302/0 (socket says 172.21.15.69:48302) 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa983fff640 1 -- 172.21.15.69:0/3290005279 learned_addr learned my addr 172.21.15.69:0/3290005279 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.392+0000 7fa9837fe640 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0947740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 -- 172.21.15.69:0/3290005279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 msgr2=0x55b2a094a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a094a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 -- 172.21.15.69:0/3290005279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 msgr2=0x55b2a0947740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0947740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 -- 172.21.15.69:0/3290005279 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2a08728e0 con 0x55b2a0942250 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa9837fe640 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0947740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa980ff9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 0x7fa96c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa983fff640 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa97c035000 tx=0x7fa97c009360 comp rx=0 tx=0).ready entity=mon.0 client_cookie=395298fad62967a7 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa9837fe640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 0x7fa96c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa980ff9640 1 -- 172.21.15.69:0/1671539058 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9780cf850 con 0x7fa984102a90 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa969ffb640 1 -- 172.21.15.69:0/3290005279 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c031cb0 con 0x55b2a0942250 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa969ffb640 1 -- 172.21.15.69:0/3290005279 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97c036050 con 0x55b2a0942250 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.393+0000 7fa969ffb640 1 -- 172.21.15.69:0/3290005279 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c004430 con 0x55b2a0942250 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3290005279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 msgr2=0x55b2a0942670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa9837fe640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 0x7fa96c0a50f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55b2a0943040 tx=0x7fa974005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7fa97c035000 tx=0x7fa97c009360 comp rx=0 tx=0).stop 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3290005279 shutdown_connections 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a094a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0947740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3290005279 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0942670 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3290005279 >> 172.21.15.69:0/3290005279 conn(0x55b2a093d8c0 msgr2=0x55b2a093dcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3290005279 shutdown_connections 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3290005279 wait complete. 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.394+0000 7fa98c9e3740 1 -- start start 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 Processor -- start 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 0x55b2a0957cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b2a094ef90 con 0x55b2a0947db0 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b2a0861790 con 0x55b2a0942c40 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b2a094f110 con 0x55b2a0942250 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa98c9e3740 10 client.0 _ll_register_callbacks cb 0x55b2a089ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:16.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa983fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 0x55b2a0957cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54588/0 (socket says 172.21.15.69:54588) 2024-05-09T09:39:16.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.395+0000 7fa982ffd640 1 -- 172.21.15.69:0/1755165469 learned_addr learned my addr 172.21.15.69:0/1755165469 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 -- 172.21.15.69:0/1755165469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942250 msgr2=0x55b2a0950750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 -- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 msgr2=0x55b2a0950c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 -- 172.21.15.69:0/1755165469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2a05ba210 con 0x55b2a0947db0 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa982ffd640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa983fff640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 0x55b2a0957cd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009ca0 tx=0x7fa97c0318e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6faab70da3068d30 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:16.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97c0041d0 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa97c004370 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa97c036050 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa97c059050 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa98c9e3740 10 client.17637 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b2a0686620 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.396+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97c036050 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa97c054030 con 0x55b2a0947db0 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 10 client.17637 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 10 client.17637 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 20 client.17637 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a095bf70 0x55b2a095e360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 4 client.17637 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b2a08b3a70 con 0x55b2a095bf70 2024-05-09T09:39:16.403 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.397+0000 7fa9837fe640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a095bf70 0x55b2a095e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.398+0000 7fa9837fe640 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a095bf70 0x55b2a095e360 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f3a62b067634764 server_cookie=faa609cb6b05ca90 in_seq=0 out_seq=0 2024-05-09T09:39:16.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.398+0000 7fa962ffd640 10 client.17637 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.400+0000 7fa962ffd640 1 -- 172.21.15.69:0/1755165469 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b2a08b3a70 con 0x55b2a095bf70 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.400+0000 7fa962ffd640 10 client.17637 handle_command_reply: tid=0 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.400+0000 7fa98c9e3740 1 client.17637 shutdown 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 msgr2=0x55b2a0957cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 0x55b2a0957cd0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7fa97c009ca0 tx=0x7fa97c0318e0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 shutdown_connections 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0947db0 0x55b2a0957cd0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a095bf70 0x55b2a095e360 crc :-1 s=READY pgs=240 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/1755165469 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.405 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa962ffd640 0 client.17637 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 >> 172.21.15.69:0/1755165469 conn(0x55b2a093d8c0 msgr2=0x55b2a093ca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 shutdown_connections 2024-05-09T09:39:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 1 -- 172.21.15.69:0/1755165469 wait complete. 2024-05-09T09:39:16.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.401+0000 7fa98c9e3740 20 client.17637 trim_cache size 0 max 16384 2024-05-09T09:39:16.420 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 Processor -- start 2024-05-09T09:39:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 -- start start 2024-05-09T09:39:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0947db0 0x55b2a0940870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841aa950 con 0x55b2a0942250 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841a8220 con 0x55b2a0947db0 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841a6070 con 0x55b2a0942c40 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48326/0 (socket says 172.21.15.69:48326) 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.421+0000 7fa9837fe640 1 -- 172.21.15.69:0/329184294 learned_addr learned my addr 172.21.15.69:0/329184294 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 -- 172.21.15.69:0/329184294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942c40 msgr2=0x55b2a0950c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 -- 172.21.15.69:0/329184294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0947db0 msgr2=0x55b2a0940870 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0947db0 0x55b2a0940870 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 -- 172.21.15.69:0/329184294 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b2a0941e90 con 0x55b2a0942250 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9837fe640 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x55b2a09436b0 tx=0x7fa97401b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97699c414f7f164a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.424 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9697fa640 1 -- 172.21.15.69:0/329184294 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97402f070 con 0x55b2a0942250 2024-05-09T09:39:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9697fa640 1 -- 172.21.15.69:0/329184294 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97402b530 con 0x55b2a0942250 2024-05-09T09:39:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.422+0000 7fa9697fa640 1 -- 172.21.15.69:0/329184294 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97402aa00 con 0x55b2a0942250 2024-05-09T09:39:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.423+0000 7fa98c9e3740 1 -- 172.21.15.69:0/329184294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 msgr2=0x55b2a0950750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.423+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x55b2a09436b0 tx=0x7fa97401b000 comp rx=0 tx=0).stop 2024-05-09T09:39:16.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 -- 172.21.15.69:0/329184294 shutdown_connections 2024-05-09T09:39:16.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0947db0 0x55b2a0940870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.428 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0942c40 0x55b2a0950c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/329184294 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942250 0x55b2a0950750 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 -- 172.21.15.69:0/329184294 >> 172.21.15.69:0/329184294 conn(0x55b2a093d8c0 msgr2=0x55b2a093dca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 -- 172.21.15.69:0/329184294 shutdown_connections 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 -- 172.21.15.69:0/329184294 wait complete. 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.424+0000 7fa98c9e3740 1 -- start start 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.425+0000 7fa98c9e3740 1 Processor -- start 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.425+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 0x55b2a0954bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.425+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa98c9e3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a0956060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841a66e0 con 0x55b2a0942c40 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841a7b50 con 0x55b2a0942250 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa98c9e3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9841a6db0 con 0x55b2a0947db0 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa98c9e3740 10 client.0 _ll_register_callbacks cb 0x7fa9740097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:16.429 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:48334/0 (socket says 172.21.15.69:48334) 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 -- 172.21.15.69:0/3809226299 learned_addr learned my addr 172.21.15.69:0/3809226299 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa9837fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 0x55b2a0954bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa983fff640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a0956060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 -- 172.21.15.69:0/3809226299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 msgr2=0x55b2a0956060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a0956060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 -- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 msgr2=0x55b2a0954bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 0x55b2a0954bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 -- 172.21.15.69:0/3809226299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b2a0956650 con 0x55b2a0942c40 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa9837fe640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 0x55b2a0954bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa982ffd640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa9780986b0 tx=0x7fa97809b930 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc16af60ba50c749 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa962ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.426+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa97809b380 con 0x55b2a0942c40 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9780ba570 con 0x55b2a0942c40 2024-05-09T09:39:16.430 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa97402b530 con 0x55b2a0942c40 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa97809bd30 con 0x55b2a0942c40 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 10 client.17643 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b2a09c26d0 con 0x55b2a0942c40 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa97402b530 con 0x55b2a0942c40 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa9780b2030 con 0x55b2a0942c40 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 10 client.17643 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 10 client.17643 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 20 client.17643 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a09c30c0 0x55b2a09c54e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 4 client.17643 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:39:16.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b2a08addc0 con 0x55b2a09c30c0 2024-05-09T09:39:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.427+0000 7fa9837fe640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a09c30c0 0x55b2a09c54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.428+0000 7fa962ffd640 10 client.17643 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:16.432 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.428+0000 7fa9837fe640 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a09c30c0 0x55b2a09c54e0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fca5d381e32ec09 server_cookie=a3dff33c7836fa46 in_seq=0 out_seq=0 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa962ffd640 1 -- 172.21.15.69:0/3809226299 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b2a08addc0 con 0x55b2a09c30c0 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa962ffd640 10 client.17643 handle_command_reply: tid=0 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 client.17643 shutdown 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 msgr2=0x55b2a09550f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7fa9780986b0 tx=0x7fa97809b930 comp rx=0 tx=0).stop 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 shutdown_connections 2024-05-09T09:39:16.764 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b2a09c30c0 0x55b2a09c54e0 crc :-1 s=READY pgs=241 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b2a0947db0 0x55b2a0956060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa962ffd640 0 client.17643 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b2a0942c40 0x55b2a09550f0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 --2- 172.21.15.69:0/3809226299 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b2a0942250 0x55b2a0954bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 >> 172.21.15.69:0/3809226299 conn(0x55b2a093d8c0 msgr2=0x55b2a0943f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 shutdown_connections 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 1 -- 172.21.15.69:0/3809226299 wait complete. 2024-05-09T09:39:16.765 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.764+0000 7fa98c9e3740 20 client.17643 trim_cache size 0 max 16384 2024-05-09T09:39:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:39:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:39:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:39:16.766 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 msgr2=0x7fa96c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 0x7fa96c0a50f0 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x55b2a0943040 tx=0x7fa974005f90 comp rx=0 tx=0).stop 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 msgr2=0x7fa984197fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fa97800e9f0 tx=0x7fa97800edf0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 shutdown_connections 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa96c0a2c30 0x7fa96c0a50f0 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa984103450 0x7fa98419f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa984102a90 0x7fa984197fe0 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.767 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 --2- 172.21.15.69:0/1671539058 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9841018a0 0x7fa984197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 >> 172.21.15.69:0/1671539058 conn(0x7fa9840fcdf0 msgr2=0x7fa984105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 shutdown_connections 2024-05-09T09:39:16.768 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:16.765+0000 7fa989bf9640 1 -- 172.21.15.69:0/1671539058 wait complete. 2024-05-09T09:39:16.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 230554 27.99 MB/sec execute 314 sec latency 247.458 ms 2024-05-09T09:39:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:17 smithi069 ceph-mon[30123]: pgmap v981: 129 pgs: 129 active+clean; 722 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 795 op/s 2024-05-09T09:39:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:17 smithi163 ceph-mon[34354]: pgmap v981: 129 pgs: 129 active+clean; 722 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 795 op/s 2024-05-09T09:39:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:17 smithi103 ceph-mon[34421]: pgmap v981: 129 pgs: 129 active+clean; 722 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 15 MiB/s wr, 795 op/s 2024-05-09T09:39:17.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 231256 28.03 MB/sec execute 315 sec latency 69.636 ms 2024-05-09T09:39:18.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 231807 28.06 MB/sec execute 316 sec latency 157.666 ms 2024-05-09T09:39:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:19 smithi069 ceph-mon[30123]: pgmap v982: 129 pgs: 129 active+clean; 713 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 15 MiB/s wr, 721 op/s 2024-05-09T09:39:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:19 smithi163 ceph-mon[34354]: pgmap v982: 129 pgs: 129 active+clean; 713 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 15 MiB/s wr, 721 op/s 2024-05-09T09:39:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:19 smithi103 ceph-mon[34421]: pgmap v982: 129 pgs: 129 active+clean; 713 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 4.9 MiB/s rd, 15 MiB/s wr, 721 op/s 2024-05-09T09:39:19.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 232354 28.05 MB/sec execute 317 sec latency 107.433 ms 2024-05-09T09:39:20.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 232775 28.03 MB/sec execute 318 sec latency 75.395 ms 2024-05-09T09:39:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:21 smithi069 ceph-mon[30123]: pgmap v983: 129 pgs: 129 active+clean; 684 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 16 MiB/s wr, 798 op/s 2024-05-09T09:39:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:21 smithi163 ceph-mon[34354]: pgmap v983: 129 pgs: 129 active+clean; 684 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 16 MiB/s wr, 798 op/s 2024-05-09T09:39:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:21 smithi103 ceph-mon[34421]: pgmap v983: 129 pgs: 129 active+clean; 684 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 MiB/s rd, 16 MiB/s wr, 798 op/s 2024-05-09T09:39:21.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 233422 28.06 MB/sec execute 319 sec latency 79.514 ms 2024-05-09T09:39:22.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 234179 28.07 MB/sec execute 320 sec latency 52.820 ms 2024-05-09T09:39:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:23 smithi069 ceph-mon[30123]: pgmap v984: 129 pgs: 129 active+clean; 655 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 757 op/s 2024-05-09T09:39:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:23 smithi163 ceph-mon[34354]: pgmap v984: 129 pgs: 129 active+clean; 655 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 757 op/s 2024-05-09T09:39:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:23 smithi103 ceph-mon[34421]: pgmap v984: 129 pgs: 129 active+clean; 655 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 13 MiB/s wr, 757 op/s 2024-05-09T09:39:23.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 234827 28.10 MB/sec execute 321 sec latency 51.511 ms 2024-05-09T09:39:24.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 235187 28.11 MB/sec execute 322 sec latency 561.471 ms 2024-05-09T09:39:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:25 smithi163 ceph-mon[34354]: pgmap v985: 129 pgs: 129 active+clean; 662 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 711 op/s 2024-05-09T09:39:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:25 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:39:25.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:25 smithi069 ceph-mon[30123]: pgmap v985: 129 pgs: 129 active+clean; 662 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 711 op/s 2024-05-09T09:39:25.596 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:25 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:39:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:25 smithi103 ceph-mon[34421]: pgmap v985: 129 pgs: 129 active+clean; 662 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 6.7 MiB/s rd, 7.9 MiB/s wr, 711 op/s 2024-05-09T09:39:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:25 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:39:25.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 235729 28.11 MB/sec execute 323 sec latency 67.144 ms 2024-05-09T09:39:26.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 236160 28.08 MB/sec execute 324 sec latency 84.184 ms 2024-05-09T09:39:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: pgmap v986: 129 pgs: 129 active+clean; 669 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.4 MiB/s wr, 707 op/s 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:27 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: pgmap v986: 129 pgs: 129 active+clean; 669 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.4 MiB/s wr, 707 op/s 2024-05-09T09:39:27.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.619 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:27 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: pgmap v986: 129 pgs: 129 active+clean; 669 MiB data, 9.3 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 6.4 MiB/s wr, 707 op/s 2024-05-09T09:39:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:27 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:27.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 236626 28.07 MB/sec execute 325 sec latency 68.771 ms 2024-05-09T09:39:28.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:39:28.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:39:28.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:28.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:39:28.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:39:28.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:39:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:39:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:39:28.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 237287 28.08 MB/sec execute 326 sec latency 91.035 ms 2024-05-09T09:39:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:29 smithi163 ceph-mon[34354]: pgmap v987: 129 pgs: 129 active+clean; 674 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 484 op/s 2024-05-09T09:39:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:29 smithi103 ceph-mon[34421]: pgmap v987: 129 pgs: 129 active+clean; 674 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 484 op/s 2024-05-09T09:39:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:29 smithi069 ceph-mon[30123]: pgmap v987: 129 pgs: 129 active+clean; 674 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 3.0 MiB/s wr, 484 op/s 2024-05-09T09:39:29.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 238052 28.13 MB/sec execute 327 sec latency 69.661 ms 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 -- 172.21.15.69:0/3695727120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64102340 msgr2=0x7f7f64108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 --2- 172.21.15.69:0/3695727120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64102340 0x7f7f64108bd0 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7f7f5c009ca0 tx=0x7f7f5c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 -- 172.21.15.69:0/3695727120 shutdown_connections 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 --2- 172.21.15.69:0/3695727120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64102340 0x7f7f64108bd0 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 --2- 172.21.15.69:0/3695727120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64101980 0x7f7f64101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 --2- 172.21.15.69:0/3695727120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64100790 0x7f7f64100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 -- 172.21.15.69:0/3695727120 >> 172.21.15.69:0/3695727120 conn(0x7f7f640fbd20 msgr2=0x7f7f640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 -- 172.21.15.69:0/3695727120 shutdown_connections 2024-05-09T09:39:29.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.916+0000 7f7f6a429640 1 -- 172.21.15.69:0/3695727120 wait complete. 2024-05-09T09:39:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 Processor -- start 2024-05-09T09:39:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 -- start start 2024-05-09T09:39:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64101980 0x7f7f64197fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64102340 0x7f7f6419f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410c3f0 con 0x7f7f64100790 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410c270 con 0x7f7f64102340 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f6a429640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f6410c570 con 0x7f7f64101980 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60424/0 (socket says 172.21.15.69:60424) 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f63fff640 1 -- 172.21.15.69:0/195835046 learned_addr learned my addr 172.21.15.69:0/195835046 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64101980 0x7f7f64197fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.917+0000 7f7f689c6640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64102340 0x7f7f6419f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 -- 172.21.15.69:0/195835046 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64101980 msgr2=0x7f7f64197fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64101980 0x7f7f64197fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 -- 172.21.15.69:0/195835046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64102340 msgr2=0x7f7f6419f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64102340 0x7f7f6419f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 -- 172.21.15.69:0/195835046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f7f6419f680 con 0x7f7f64100790 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f63fff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400e9c0 tx=0x7f7f5400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f61ffb640 1 -- 172.21.15.69:0/195835046 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5400cd80 con 0x7f7f64100790 2024-05-09T09:39:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f61ffb640 1 -- 172.21.15.69:0/195835046 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f540035f0 con 0x7f7f64100790 2024-05-09T09:39:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f61ffb640 1 -- 172.21.15.69:0/195835046 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f54010690 con 0x7f7f64100790 2024-05-09T09:39:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f7f6419f970 con 0x7f7f64100790 2024-05-09T09:39:29.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.918+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f7f6419fe70 con 0x7f7f64100790 2024-05-09T09:39:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.920+0000 7f7f61ffb640 1 -- 172.21.15.69:0/195835046 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f7f540037a0 con 0x7f7f64100790 2024-05-09T09:39:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.921+0000 7f7f61ffb640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 0x7f7f500a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.921+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 0x7f7f500a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.921+0000 7f7f61ffb640 1 -- 172.21.15.69:0/195835046 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f540cf7a0 con 0x7f7f64100790 2024-05-09T09:39:29.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.922+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 0x7f7f500a50f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7f64198f70 tx=0x7f7f4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 Processor -- start 2024-05-09T09:39:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 -- start start 2024-05-09T09:39:29.928 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea19740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea1c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a59e978340 con 0x55a59ea14250 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a59e920290 con 0x55a59ea14c40 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a59e974300 con 0x55a59ea19db0 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f689c6640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea1c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60430/0 (socket says 172.21.15.69:60430) 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 -- 172.21.15.69:0/287098873 learned_addr learned my addr 172.21.15.69:0/287098873 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea19740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 -- 172.21.15.69:0/287098873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 msgr2=0x55a59ea1c240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea1c240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 -- 172.21.15.69:0/287098873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 msgr2=0x55a59ea19740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea19740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f63fff640 1 -- 172.21.15.69:0/287098873 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59e9448e0 con 0x55a59ea14250 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.927+0000 7f7f689c6640 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea1c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.928+0000 7f7f63fff640 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7f540980e0 tx=0x7f7f5409c380 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56b96189ee812f72 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.928+0000 7f7f5a7fc640 1 -- 172.21.15.69:0/287098873 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a48f0 con 0x55a59ea14250 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.928+0000 7f7f5a7fc640 1 -- 172.21.15.69:0/287098873 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f54099f80 con 0x55a59ea14250 2024-05-09T09:39:29.930 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.928+0000 7f7f5a7fc640 1 -- 172.21.15.69:0/287098873 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5409b3c0 con 0x55a59ea14250 2024-05-09T09:39:29.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- 172.21.15.69:0/287098873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 msgr2=0x55a59ea14670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f7f540980e0 tx=0x7f7f5409c380 comp rx=0 tx=0).stop 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- 172.21.15.69:0/287098873 shutdown_connections 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea1c240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea19740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/287098873 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea14670 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- 172.21.15.69:0/287098873 >> 172.21.15.69:0/287098873 conn(0x55a59ea0f8c0 msgr2=0x55a59ea0fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:29.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- 172.21.15.69:0/287098873 shutdown_connections 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- 172.21.15.69:0/287098873 wait complete. 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 -- start start 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.929+0000 7f7f6d20b740 1 Processor -- start 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea1e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea259e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a59ea1cd00 con 0x55a59ea14250 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a59e933790 con 0x55a59ea14c40 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a59ea1ce80 con 0x55a59ea19db0 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 10 client.0 _ll_register_callbacks cb 0x55a59e96ea20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60436/0 (socket says 172.21.15.69:60436) 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 -- 172.21.15.69:0/3606463865 learned_addr learned my addr 172.21.15.69:0/3606463865 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:29.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f689c6640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea259e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea1e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 -- 172.21.15.69:0/3606463865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 msgr2=0x55a59ea259e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea259e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 -- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 msgr2=0x55a59ea1e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea1e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 -- 172.21.15.69:0/3606463865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59e68c210 con 0x55a59ea14250 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7f540a43c0 tx=0x7f7f54003e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e080038c66ea287 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5407f060 con 0x55a59ea14250 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f5407acf0 con 0x55a59ea14250 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.930+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f54099f80 con 0x55a59ea14250 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7f540a48f0 con 0x55a59ea14250 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 10 client.17655 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:29.936 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a59e758620 con 0x55a59ea14250 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f54099f80 con 0x55a59ea14250 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f7f540b4030 con 0x55a59ea14250 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 10 client.17655 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 10 client.17655 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 20 client.17655 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea29b40 0x55a59ea2bf30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 4 client.17655 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a59e985a70 con 0x55a59ea29b40 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.931+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea29b40 0x55a59ea2bf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.933+0000 7f7f437fe640 10 client.17655 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:29.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.933+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea29b40 0x55a59ea2bf30 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41b78592e867ddda server_cookie=1743c226e5f06dca in_seq=0 out_seq=0 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f437fe640 1 -- 172.21.15.69:0/3606463865 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a59e985a70 con 0x55a59ea29b40 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f437fe640 10 client.17655 handle_command_reply: tid=0 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 client.17655 shutdown 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 msgr2=0x55a59ea1e460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f7f540a43c0 tx=0x7f7f54003e30 comp rx=0 tx=0).stop 2024-05-09T09:39:29.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 shutdown_connections 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea29b40 0x55a59ea2bf30 crc :-1 s=READY pgs=242 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea19db0 0x55a59ea259e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea14c40 0x55a59ea1e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f437fe640 0 client.17655 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3606463865 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1e460 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.943+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 >> 172.21.15.69:0/3606463865 conn(0x55a59ea0f8c0 msgr2=0x55a59ea0ea00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.944+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 shutdown_connections 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.944+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3606463865 wait complete. 2024-05-09T09:39:29.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.944+0000 7f7f6d20b740 20 client.17655 trim_cache size 0 max 16384 2024-05-09T09:39:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 Processor -- start 2024-05-09T09:39:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 -- start start 2024-05-09T09:39:29.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14c40 0x55a59ea25970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea1e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641acc20 con 0x55a59ea14250 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641aa4f0 con 0x55a59ea19db0 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641a8340 con 0x55a59ea14c40 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60438/0 (socket says 172.21.15.69:60438) 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 -- 172.21.15.69:0/3025444893 learned_addr learned my addr 172.21.15.69:0/3025444893 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14c40 0x55a59ea25970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f689c6640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea1e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 -- 172.21.15.69:0/3025444893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14c40 msgr2=0x55a59ea25970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14c40 0x55a59ea25970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 -- 172.21.15.69:0/3025444893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 msgr2=0x55a59ea1e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea1e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.962+0000 7f7f63fff640 1 -- 172.21.15.69:0/3025444893 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a59ea0df00 con 0x55a59ea14250 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.963+0000 7f7f689c6640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea1e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.963+0000 7f7f63fff640 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c6a0 tx=0x7f7f54003ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=906b35391b3bed28 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.963+0000 7f7f59ffb640 1 -- 172.21.15.69:0/3025444893 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f540a9070 con 0x55a59ea14250 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.963+0000 7f7f59ffb640 1 -- 172.21.15.69:0/3025444893 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f54099f80 con 0x55a59ea14250 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.963+0000 7f7f59ffb640 1 -- 172.21.15.69:0/3025444893 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f5407f070 con 0x55a59ea14250 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3025444893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 msgr2=0x55a59ea1c1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400c6a0 tx=0x7f7f54003ea0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3025444893 shutdown_connections 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea1e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14c40 0x55a59ea25970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/3025444893 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14250 0x55a59ea1c1a0 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3025444893 >> 172.21.15.69:0/3025444893 conn(0x55a59ea0f8c0 msgr2=0x55a59ea0a620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3025444893 shutdown_connections 2024-05-09T09:39:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- 172.21.15.69:0/3025444893 wait complete. 2024-05-09T09:39:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 -- start start 2024-05-09T09:39:29.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.964+0000 7f7f6d20b740 1 Processor -- start 2024-05-09T09:39:29.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14250 0x55a59ea22820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea8df90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60450/0 (socket says 172.21.15.69:60450) 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f63fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14250 0x55a59ea22820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 -- 172.21.15.69:0/4014389060 learned_addr learned my addr 172.21.15.69:0/4014389060 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641a89b0 con 0x55a59ea14c40 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641a9e20 con 0x55a59ea19db0 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f7f641a9080 con 0x55a59ea14250 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 10 client.0 _ll_register_callbacks cb 0x7f7f4c0099e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 -- 172.21.15.69:0/4014389060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14250 msgr2=0x55a59ea22820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14250 0x55a59ea22820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 -- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 msgr2=0x55a59ea8df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:29.970 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea8df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a59ea8e4d0 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f5bfff640 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009a90 tx=0x7f7f4c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc2c4a87cef495a5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f437fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7f4c02c8e0 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7f4c0035f0 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.965+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f7f54099f80 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f7f4c03d050 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 10 client.17661 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a59ea8f7e0 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f7f4c003cd0 con 0x55a59ea14c40 2024-05-09T09:39:29.971 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f7f4c033030 con 0x55a59ea14c40 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 10 client.17661 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 10 client.17661 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 20 client.17661 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea90170 0x55a59ea92590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 4 client.17661 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f7f54099f80 con 0x55a59ea90170 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.966+0000 7f7f63fff640 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea90170 0x55a59ea92590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.967+0000 7f7f437fe640 10 client.17661 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:29.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:29.967+0000 7f7f63fff640 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea90170 0x55a59ea92590 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c214bc72b675cad server_cookie=2ab2d161d8d65b6d in_seq=0 out_seq=0 2024-05-09T09:39:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.739+0000 7f7f437fe640 1 -- 172.21.15.69:0/4014389060 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f7f54099f80 con 0x55a59ea90170 2024-05-09T09:39:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.739+0000 7f7f437fe640 10 client.17661 handle_command_reply: tid=0 2024-05-09T09:39:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 client.17661 shutdown 2024-05-09T09:39:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 msgr2=0x55a59ea22d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:30.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7f7f4c009a90 tx=0x7f7f4c0046c0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 shutdown_connections 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a59ea19db0 0x55a59ea8df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a59ea14c40 0x55a59ea22d60 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a59ea90170 0x55a59ea92590 crc :-1 s=READY pgs=243 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 --2- 172.21.15.69:0/4014389060 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a59ea14250 0x55a59ea22820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 >> 172.21.15.69:0/4014389060 conn(0x55a59ea0f8c0 msgr2=0x55a59ea1dfe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f437fe640 0 client.17661 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 shutdown_connections 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 1 -- 172.21.15.69:0/4014389060 wait complete. 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.740+0000 7f7f6d20b740 20 client.17661 trim_cache size 0 max 16384 2024-05-09T09:39:30.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:39:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:39:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:39:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:39:30.743 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 msgr2=0x7f7f500a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 0x7f7f500a50f0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f7f64198f70 tx=0x7f7f4c005f90 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 msgr2=0x7f7f64197a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7f7f5400e9c0 tx=0x7f7f5400ee80 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 shutdown_connections 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f7f500a2c30 0x7f7f500a50f0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f7f64102340 0x7f7f6419f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f7f64101980 0x7f7f64197fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 --2- 172.21.15.69:0/195835046 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f7f64100790 0x7f7f64197a60 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 >> 172.21.15.69:0/195835046 conn(0x7f7f640fbd20 msgr2=0x7f7f640fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 shutdown_connections 2024-05-09T09:39:30.744 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:30.741+0000 7f7f6a429640 1 -- 172.21.15.69:0/195835046 wait complete. 2024-05-09T09:39:30.835 INFO:tasks.workunit.client.0.smithi069.stdout: 10 238323 28.07 MB/sec execute 328 sec latency 598.695 ms 2024-05-09T09:39:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:31 smithi163 ceph-mon[34354]: pgmap v988: 129 pgs: 129 active+clean; 679 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 496 op/s 2024-05-09T09:39:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:31 smithi103 ceph-mon[34421]: pgmap v988: 129 pgs: 129 active+clean; 679 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 496 op/s 2024-05-09T09:39:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:31 smithi069 ceph-mon[30123]: pgmap v988: 129 pgs: 129 active+clean; 679 MiB data, 9.4 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 496 op/s 2024-05-09T09:39:31.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 238783 28.10 MB/sec execute 329 sec latency 80.066 ms 2024-05-09T09:39:32.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 239146 28.08 MB/sec execute 330 sec latency 261.270 ms 2024-05-09T09:39:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:33 smithi163 ceph-mon[34354]: pgmap v989: 129 pgs: 129 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 429 op/s 2024-05-09T09:39:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:33 smithi103 ceph-mon[34421]: pgmap v989: 129 pgs: 129 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 429 op/s 2024-05-09T09:39:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:33 smithi069 ceph-mon[30123]: pgmap v989: 129 pgs: 129 active+clean; 684 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.3 MiB/s wr, 429 op/s 2024-05-09T09:39:33.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 239554 28.05 MB/sec execute 331 sec latency 81.978 ms 2024-05-09T09:39:34.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 240022 28.01 MB/sec execute 332 sec latency 148.876 ms 2024-05-09T09:39:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:35 smithi163 ceph-mon[34354]: pgmap v990: 129 pgs: 129 active+clean; 682 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 403 op/s 2024-05-09T09:39:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:35 smithi103 ceph-mon[34421]: pgmap v990: 129 pgs: 129 active+clean; 682 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 403 op/s 2024-05-09T09:39:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:35 smithi069 ceph-mon[30123]: pgmap v990: 129 pgs: 129 active+clean; 682 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.2 MiB/s wr, 403 op/s 2024-05-09T09:39:35.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 240554 27.99 MB/sec execute 333 sec latency 79.238 ms 2024-05-09T09:39:36.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 241387 28.07 MB/sec execute 334 sec latency 57.497 ms 2024-05-09T09:39:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:37 smithi163 ceph-mon[34354]: pgmap v991: 129 pgs: 129 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:39:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:37 smithi103 ceph-mon[34421]: pgmap v991: 129 pgs: 129 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:39:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:37 smithi069 ceph-mon[30123]: pgmap v991: 129 pgs: 129 active+clean; 687 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.6 MiB/s wr, 570 op/s 2024-05-09T09:39:37.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 241958 28.09 MB/sec execute 335 sec latency 96.242 ms 2024-05-09T09:39:38.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 242430 28.10 MB/sec execute 336 sec latency 70.611 ms 2024-05-09T09:39:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:39 smithi163 ceph-mon[34354]: pgmap v992: 129 pgs: 129 active+clean; 689 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.0 MiB/s wr, 532 op/s 2024-05-09T09:39:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:39 smithi103 ceph-mon[34421]: pgmap v992: 129 pgs: 129 active+clean; 689 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.0 MiB/s wr, 532 op/s 2024-05-09T09:39:39.763 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:39 smithi069 ceph-mon[30123]: pgmap v992: 129 pgs: 129 active+clean; 689 MiB data, 9.5 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 3.0 MiB/s wr, 532 op/s 2024-05-09T09:39:39.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 242853 28.08 MB/sec execute 337 sec latency 66.661 ms 2024-05-09T09:39:40.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 243311 28.05 MB/sec execute 338 sec latency 83.049 ms 2024-05-09T09:39:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:41 smithi163 ceph-mon[34354]: pgmap v993: 129 pgs: 129 active+clean; 692 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 MiB/s wr, 490 op/s 2024-05-09T09:39:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:41 smithi103 ceph-mon[34421]: pgmap v993: 129 pgs: 129 active+clean; 692 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 MiB/s wr, 490 op/s 2024-05-09T09:39:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:41 smithi069 ceph-mon[30123]: pgmap v993: 129 pgs: 129 active+clean; 692 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.8 MiB/s wr, 490 op/s 2024-05-09T09:39:41.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 243863 28.03 MB/sec execute 339 sec latency 56.703 ms 2024-05-09T09:39:42.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 244512 28.06 MB/sec execute 340 sec latency 109.386 ms 2024-05-09T09:39:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:43 smithi163 ceph-mon[34354]: pgmap v994: 129 pgs: 129 active+clean; 697 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.9 MiB/s wr, 491 op/s 2024-05-09T09:39:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:43 smithi103 ceph-mon[34421]: pgmap v994: 129 pgs: 129 active+clean; 697 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.9 MiB/s wr, 491 op/s 2024-05-09T09:39:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:43 smithi069 ceph-mon[30123]: pgmap v994: 129 pgs: 129 active+clean; 697 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.9 MiB/s wr, 491 op/s 2024-05-09T09:39:43.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 245143 28.10 MB/sec execute 341 sec latency 198.085 ms 2024-05-09T09:39:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.865+0000 7f2785e10640 1 -- 172.21.15.69:0/3069746110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 msgr2=0x7f2780108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.866 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.865+0000 7f2785e10640 1 --2- 172.21.15.69:0/3069746110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f2780108bd0 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7f2774009860 tx=0x7f277402e790 comp rx=0 tx=0).stop 2024-05-09T09:39:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.866+0000 7f2785e10640 1 -- 172.21.15.69:0/3069746110 shutdown_connections 2024-05-09T09:39:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.866+0000 7f2785e10640 1 --2- 172.21.15.69:0/3069746110 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f2780108bd0 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.866+0000 7f2785e10640 1 --2- 172.21.15.69:0/3069746110 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780101980 0x7f2780101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.866+0000 7f2785e10640 1 --2- 172.21.15.69:0/3069746110 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780100790 0x7f2780100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.866+0000 7f2785e10640 1 -- 172.21.15.69:0/3069746110 >> 172.21.15.69:0/3069746110 conn(0x7f27800fbd00 msgr2=0x7f27800fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:43.874 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.874+0000 7f2785e10640 1 -- 172.21.15.69:0/3069746110 shutdown_connections 2024-05-09T09:39:43.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.875+0000 7f2785e10640 1 -- 172.21.15.69:0/3069746110 wait complete. 2024-05-09T09:39:43.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.876+0000 7f2785e10640 1 Processor -- start 2024-05-09T09:39:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 -- start start 2024-05-09T09:39:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780100790 0x7f2780070010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f27801ae330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f278010b410 con 0x7f2780102340 2024-05-09T09:39:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f278010b290 con 0x7f2780101980 2024-05-09T09:39:43.880 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.879+0000 7f2785e10640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f278010b590 con 0x7f2780100790 2024-05-09T09:39:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.880+0000 7f277effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.880+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780100790 0x7f2780070010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.880+0000 7f277effd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40864/0 (socket says 172.21.15.69:40864) 2024-05-09T09:39:43.881 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.880+0000 7f277effd640 1 -- 172.21.15.69:0/3180592134 learned_addr learned my addr 172.21.15.69:0/3180592134 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.882+0000 7f277ffff640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f27801ae330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.884+0000 7f277effd640 1 -- 172.21.15.69:0/3180592134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780100790 msgr2=0x7f2780070010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.884+0000 7f277effd640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780100790 0x7f2780070010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.884+0000 7f277effd640 1 -- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 msgr2=0x7f27801ae330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.884+0000 7f277effd640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f27801ae330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.884 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.884+0000 7f277effd640 1 -- 172.21.15.69:0/3180592134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f27800736d0 con 0x7f2780101980 2024-05-09T09:39:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.888+0000 7f277effd640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f277800efb0 tx=0x7f277800c800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.889+0000 7f277cff9640 1 -- 172.21.15.69:0/3180592134 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277801b070 con 0x7f2780101980 2024-05-09T09:39:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.889+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f27801a1b00 con 0x7f2780101980 2024-05-09T09:39:43.889 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.889+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f27801a2050 con 0x7f2780101980 2024-05-09T09:39:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.889+0000 7f277cff9640 1 -- 172.21.15.69:0/3180592134 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f277800ba50 con 0x7f2780101980 2024-05-09T09:39:43.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.889+0000 7f277cff9640 1 -- 172.21.15.69:0/3180592134 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778016c00 con 0x7f2780101980 2024-05-09T09:39:43.898 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.895+0000 7f2788bf3740 1 Processor -- start 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.896+0000 7f277cff9640 1 -- 172.21.15.69:0/3180592134 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f277801e070 con 0x7f2780101980 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.898+0000 7f277cff9640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 0x7f27680a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.898+0000 7f277cff9640 1 -- 172.21.15.69:0/3180592134 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f27780cf710 con 0x7f2780101980 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.898+0000 7f2788bf3740 1 -- start start 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107b4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x559410710340 con 0x5594107b1db0 2024-05-09T09:39:43.899 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5594106b8290 con 0x5594107ac250 2024-05-09T09:39:43.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.899+0000 7f2788bf3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55941070c300 con 0x5594107acc40 2024-05-09T09:39:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:47588/0 (socket says 172.21.15.69:47588) 2024-05-09T09:39:43.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 -- 172.21.15.69:0/6111597 learned_addr learned my addr 172.21.15.69:0/6111597 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 -- 172.21.15.69:0/6111597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 msgr2=0x5594107ac670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 -- 172.21.15.69:0/6111597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 msgr2=0x5594107b4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107b4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.901+0000 7f277effd640 1 -- 172.21.15.69:0/6111597 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594106dc8e0 con 0x5594107acc40 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.902+0000 7f277effd640 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f277809bbd0 tx=0x7f27780a0080 comp rx=0 tx=0).ready entity=mon.1 client_cookie=65fd0740f5dc0ca6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.902+0000 7f277ffff640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 0x7f27680a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.906+0000 7f2765ffb640 1 -- 172.21.15.69:0/6111597 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f27780cf3f0 con 0x5594107acc40 2024-05-09T09:39:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.906+0000 7f2765ffb640 1 -- 172.21.15.69:0/6111597 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778016410 con 0x5594107acc40 2024-05-09T09:39:43.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.906+0000 7f2765ffb640 1 -- 172.21.15.69:0/6111597 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277809b2f0 con 0x5594107acc40 2024-05-09T09:39:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.907+0000 7f2788bf3740 1 -- 172.21.15.69:0/6111597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 msgr2=0x5594107b1740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.907+0000 7f2788bf3740 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f277809bbd0 tx=0x7f27780a0080 comp rx=0 tx=0).stop 2024-05-09T09:39:43.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.908+0000 7f277ffff640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 0x7f27680a5330 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f277402ed80 tx=0x7f2774034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.909+0000 7f2788bf3740 1 -- 172.21.15.69:0/6111597 shutdown_connections 2024-05-09T09:39:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.909+0000 7f2788bf3740 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107b4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.909+0000 7f2788bf3740 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b1740 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.909+0000 7f2788bf3740 1 --2- 172.21.15.69:0/6111597 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107ac670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.909+0000 7f2788bf3740 1 -- 172.21.15.69:0/6111597 >> 172.21.15.69:0/6111597 conn(0x5594107a78c0 msgr2=0x5594107a7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:43.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.911+0000 7f2788bf3740 1 -- 172.21.15.69:0/6111597 shutdown_connections 2024-05-09T09:39:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.912+0000 7f2788bf3740 1 -- 172.21.15.69:0/6111597 wait complete. 2024-05-09T09:39:43.913 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.913+0000 7f2788bf3740 1 -- start start 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.913+0000 7f2788bf3740 1 Processor -- start 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b6920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5594107b4be0 con 0x5594107b1db0 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5594106cb790 con 0x5594107ac250 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5594107b4d60 con 0x5594107acc40 2024-05-09T09:39:43.914 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f2788bf3740 10 client.0 _ll_register_callbacks cb 0x559410706a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f277ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f277ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49256/0 (socket says 172.21.15.69:49256) 2024-05-09T09:39:43.915 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.914+0000 7f277ffff640 1 -- 172.21.15.69:0/3402327133 learned_addr learned my addr 172.21.15.69:0/3402327133 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.916+0000 7f277effd640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b6920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277ffff640 1 -- 172.21.15.69:0/3402327133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 msgr2=0x5594107b6920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277ffff640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b6920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277ffff640 1 -- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 msgr2=0x5594107b63e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277ffff640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b63e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277ffff640 1 -- 172.21.15.69:0/3402327133 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x559410424210 con 0x5594107b1db0 2024-05-09T09:39:43.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.917+0000 7f277effd640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:39:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.918+0000 7f277ffff640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2774038000 tx=0x7f2774034e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74f00a50a2abecf6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.918+0000 7f275effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:43.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.918+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277404e030 con 0x5594107b1db0 2024-05-09T09:39:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.918+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f277404a650 con 0x5594107b1db0 2024-05-09T09:39:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.918+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f2778016410 con 0x5594107b1db0 2024-05-09T09:39:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.919+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f277404a850 con 0x5594107b1db0 2024-05-09T09:39:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.919+0000 7f2788bf3740 10 client.17670 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:43.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.919+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5594104f0620 con 0x5594107b1db0 2024-05-09T09:39:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.921+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2778016410 con 0x5594107b1db0 2024-05-09T09:39:43.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.921+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f2774035c30 con 0x5594107b1db0 2024-05-09T09:39:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 10 client.17670 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:43.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 10 client.17670 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 20 client.17670 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5594107c1fc0 0x5594107c4410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 4 client.17670 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:39:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55941071da70 con 0x5594107c1fc0 2024-05-09T09:39:43.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.922+0000 7f277f7fe640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5594107c1fc0 0x5594107c4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.924+0000 7f277f7fe640 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5594107c1fc0 0x5594107c4410 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=10e07c15c0637353 server_cookie=f92eae266a9fdc5d in_seq=0 out_seq=0 2024-05-09T09:39:43.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.924+0000 7f275effd640 10 client.17670 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.925+0000 7f275effd640 1 -- 172.21.15.69:0/3402327133 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55941071da70 con 0x5594107c1fc0 2024-05-09T09:39:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.925+0000 7f275effd640 10 client.17670 handle_command_reply: tid=0 2024-05-09T09:39:43.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.925+0000 7f2788bf3740 1 client.17670 shutdown 2024-05-09T09:39:43.926 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 msgr2=0x5594107bd960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7f2774038000 tx=0x7f2774034e30 comp rx=0 tx=0).stop 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 shutdown_connections 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107b1db0 0x5594107bd960 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5594107c1fc0 0x5594107c4410 crc :-1 s=READY pgs=244 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107b6920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 --2- 172.21.15.69:0/3402327133 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 >> 172.21.15.69:0/3402327133 conn(0x5594107a78c0 msgr2=0x5594107a6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:43.927 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.926+0000 7f275effd640 0 client.17670 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.928+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 shutdown_connections 2024-05-09T09:39:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.929+0000 7f2788bf3740 1 -- 172.21.15.69:0/3402327133 wait complete. 2024-05-09T09:39:43.929 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.929+0000 7f2788bf3740 20 client.17670 trim_cache size 0 max 16384 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.948+0000 7f2788bf3740 1 Processor -- start 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.948+0000 7f2788bf3740 1 -- start start 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.948+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b56d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.948+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.948+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107b1db0 0x5594107b5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f2788bf3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f276c012530 con 0x5594107acc40 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f2788bf3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f276c00fe00 con 0x5594107ac250 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f2788bf3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f276c00dc50 con 0x5594107b1db0 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49272/0 (socket says 172.21.15.69:49272) 2024-05-09T09:39:43.950 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 -- 172.21.15.69:0/449444358 learned_addr learned my addr 172.21.15.69:0/449444358 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277f7fe640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277ffff640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107b1db0 0x5594107b5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 -- 172.21.15.69:0/449444358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107b1db0 msgr2=0x5594107b5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107b1db0 0x5594107b5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 -- 172.21.15.69:0/449444358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 msgr2=0x5594107b56d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b56d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 -- 172.21.15.69:0/449444358 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5594107a1ba0 con 0x5594107acc40 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277f7fe640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.949+0000 7f277effd640 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f277809b940 tx=0x7f27780cd0d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9c6f77b242ada04 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.950+0000 7f27657fa640 1 -- 172.21.15.69:0/449444358 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2778016410 con 0x5594107acc40 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.950+0000 7f27657fa640 1 -- 172.21.15.69:0/449444358 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27780982b0 con 0x5594107acc40 2024-05-09T09:39:43.951 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.950+0000 7f27657fa640 1 -- 172.21.15.69:0/449444358 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f277809b090 con 0x5594107acc40 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.950+0000 7f2788bf3740 1 -- 172.21.15.69:0/449444358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 msgr2=0x5594107bd920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.950+0000 7f2788bf3740 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7f277809b940 tx=0x7f27780cd0d0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- 172.21.15.69:0/449444358 shutdown_connections 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107b1db0 0x5594107b5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107acc40 0x5594107bd920 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- 172.21.15.69:0/449444358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107ac250 0x5594107b56d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- 172.21.15.69:0/449444358 >> 172.21.15.69:0/449444358 conn(0x5594107a78c0 msgr2=0x5594107adec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- 172.21.15.69:0/449444358 shutdown_connections 2024-05-09T09:39:43.955 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- 172.21.15.69:0/449444358 wait complete. 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- start start 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 Processor -- start 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107bb0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107b1db0 0x5594107bc080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f276c00e2c0 con 0x5594107ac250 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f276c00f730 con 0x5594107b1db0 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f276c00e990 con 0x5594107acc40 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.951+0000 7f2788bf3740 10 client.0 _ll_register_callbacks cb 0x7f2774042070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:49278/0 (socket says 172.21.15.69:49278) 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 -- 172.21.15.69:0/2961974040 learned_addr learned my addr 172.21.15.69:0/2961974040 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277effd640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107bb0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277ffff640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107b1db0 0x5594107bc080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 -- 172.21.15.69:0/2961974040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 msgr2=0x5594107bb0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.956 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107bb0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 -- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107b1db0 msgr2=0x5594107bc080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107b1db0 0x5594107bc080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 -- 172.21.15.69:0/2961974040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5594107bc670 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f277f7fe640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x5594107ad4a0 tx=0x7f277001c420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddef38e19156785 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f275effd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2770004440 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f27700035f0 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f27780982b0 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f277002e050 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f2788bf3740 10 client.17676 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.952+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5594107bda50 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f27780982b0 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f2770023020 con 0x5594107ac250 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 10 client.17676 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 10 client.17676 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:43.957 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 20 client.17676 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559410828af0 0x55941082af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 4 client.17676 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x559410717dc0 con 0x559410828af0 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f277effd640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559410828af0 0x55941082af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f275effd640 10 client.17676 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:43.958 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:43.953+0000 7f277effd640 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559410828af0 0x55941082af10 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=574ff94c29b195c6 server_cookie=9353c723f5215f12 in_seq=0 out_seq=0 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f275effd640 1 -- 172.21.15.69:0/2961974040 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x559410717dc0 con 0x559410828af0 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f275effd640 10 client.17676 handle_command_reply: tid=0 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 client.17676 shutdown 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 msgr2=0x5594107bab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x5594107ad4a0 tx=0x7f277001c420 comp rx=0 tx=0).stop 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 shutdown_connections 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5594107b1db0 0x5594107bc080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x559410828af0 0x55941082af10 crc :-1 s=READY pgs=245 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5594107acc40 0x5594107bb0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.796 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f275effd640 0 client.17676 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 --2- 172.21.15.69:0/2961974040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5594107ac250 0x5594107bab80 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 >> 172.21.15.69:0/2961974040 conn(0x5594107a78c0 msgr2=0x5594107b56d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 shutdown_connections 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.794+0000 7f2788bf3740 1 -- 172.21.15.69:0/2961974040 wait complete. 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2788bf3740 20 client.17676 trim_cache size 0 max 16384 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:39:44.797 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:39:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 msgr2=0x7f27680a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 0x7f27680a5330 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f277402ed80 tx=0x7f2774034ab0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.798 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 msgr2=0x7f2780070550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f277800efb0 tx=0x7f277800c800 comp rx=0 tx=0).stop 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 shutdown_connections 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f27680a2e70 0x7f27680a5330 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2780102340 0x7f27801ae330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2780101980 0x7f2780070550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 --2- 172.21.15.69:0/3180592134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2780100790 0x7f2780070010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 >> 172.21.15.69:0/3180592134 conn(0x7f27800fbd00 msgr2=0x7f2780104bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 shutdown_connections 2024-05-09T09:39:44.799 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:44.795+0000 7f2785e10640 1 -- 172.21.15.69:0/3180592134 wait complete. 2024-05-09T09:39:44.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 245368 28.05 MB/sec execute 342 sec latency 595.623 ms 2024-05-09T09:39:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:45 smithi103 ceph-mon[34421]: pgmap v995: 129 pgs: 129 active+clean; 700 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.9 MiB/s wr, 507 op/s 2024-05-09T09:39:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:45 smithi069 ceph-mon[30123]: pgmap v995: 129 pgs: 129 active+clean; 700 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.9 MiB/s wr, 507 op/s 2024-05-09T09:39:45.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:45 smithi163 ceph-mon[34354]: pgmap v995: 129 pgs: 129 active+clean; 700 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 2.9 MiB/s wr, 507 op/s 2024-05-09T09:39:45.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 245881 28.06 MB/sec execute 343 sec latency 73.613 ms 2024-05-09T09:39:46.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 246337 28.05 MB/sec execute 344 sec latency 63.857 ms 2024-05-09T09:39:47.607 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:47 smithi163 ceph-mon[34354]: pgmap v996: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 639 op/s 2024-05-09T09:39:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:47 smithi103 ceph-mon[34421]: pgmap v996: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 639 op/s 2024-05-09T09:39:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:47 smithi069 ceph-mon[30123]: pgmap v996: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 4.3 MiB/s wr, 639 op/s 2024-05-09T09:39:47.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 246804 28.03 MB/sec execute 345 sec latency 122.485 ms 2024-05-09T09:39:48.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 247185 28.00 MB/sec execute 346 sec latency 448.507 ms 2024-05-09T09:39:49.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:49 smithi103 ceph-mon[34421]: pgmap v997: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-09T09:39:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:49 smithi069 ceph-mon[30123]: pgmap v997: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-09T09:39:49.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:49 smithi163 ceph-mon[34354]: pgmap v997: 129 pgs: 129 active+clean; 682 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.8 MiB/s wr, 450 op/s 2024-05-09T09:39:49.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 247600 27.97 MB/sec execute 347 sec latency 244.979 ms 2024-05-09T09:39:50.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 248053 27.97 MB/sec execute 348 sec latency 95.171 ms 2024-05-09T09:39:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:51 smithi103 ceph-mon[34421]: pgmap v998: 129 pgs: 129 active+clean; 685 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-05-09T09:39:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:51 smithi069 ceph-mon[30123]: pgmap v998: 129 pgs: 129 active+clean; 685 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-05-09T09:39:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:51 smithi163 ceph-mon[34354]: pgmap v998: 129 pgs: 129 active+clean; 685 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 4.1 MiB/s wr, 482 op/s 2024-05-09T09:39:51.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 248759 28.02 MB/sec execute 349 sec latency 47.283 ms 2024-05-09T09:39:52.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 249393 28.03 MB/sec execute 350 sec latency 84.372 ms 2024-05-09T09:39:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:53 smithi103 ceph-mon[34421]: pgmap v999: 129 pgs: 129 active+clean; 699 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.7 MiB/s wr, 578 op/s 2024-05-09T09:39:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:53 smithi069 ceph-mon[30123]: pgmap v999: 129 pgs: 129 active+clean; 699 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.7 MiB/s wr, 578 op/s 2024-05-09T09:39:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:53 smithi163 ceph-mon[34354]: pgmap v999: 129 pgs: 129 active+clean; 699 MiB data, 9.6 GiB used, 1.0 TiB / 1.0 TiB avail; 8.6 MiB/s rd, 5.7 MiB/s wr, 578 op/s 2024-05-09T09:39:53.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 249728 28.00 MB/sec execute 351 sec latency 331.383 ms 2024-05-09T09:39:54.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 250029 27.97 MB/sec execute 352 sec latency 608.228 ms 2024-05-09T09:39:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:55 smithi103 ceph-mon[34421]: pgmap v1000: 129 pgs: 129 active+clean; 700 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.8 MiB/s wr, 556 op/s 2024-05-09T09:39:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:55 smithi069 ceph-mon[30123]: pgmap v1000: 129 pgs: 129 active+clean; 700 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.8 MiB/s wr, 556 op/s 2024-05-09T09:39:55.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:55 smithi163 ceph-mon[34354]: pgmap v1000: 129 pgs: 129 active+clean; 700 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 5.8 MiB/s wr, 556 op/s 2024-05-09T09:39:55.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 250409 27.95 MB/sec execute 353 sec latency 154.437 ms 2024-05-09T09:39:56.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 250974 27.96 MB/sec execute 354 sec latency 55.036 ms 2024-05-09T09:39:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:57 smithi103 ceph-mon[34421]: pgmap v1001: 129 pgs: 129 active+clean; 710 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-05-09T09:39:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:57 smithi069 ceph-mon[30123]: pgmap v1001: 129 pgs: 129 active+clean; 710 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-05-09T09:39:57.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:57 smithi163 ceph-mon[34354]: pgmap v1001: 129 pgs: 129 active+clean; 710 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 7.8 MiB/s wr, 677 op/s 2024-05-09T09:39:57.836 INFO:tasks.workunit.client.0.smithi069.stdout: 10 251412 27.95 MB/sec execute 355 sec latency 153.855 ms 2024-05-09T09:39:57.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.905+0000 7fb4cf47d640 1 -- 172.21.15.69:0/2439455360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 msgr2=0x7fb4c8101e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.905+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/2439455360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8101e00 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009900 tx=0x7fb4bc02a410 comp rx=0 tx=0).stop 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 -- 172.21.15.69:0/2439455360 shutdown_connections 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/2439455360 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 0x7fb4c8108bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/2439455360 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8101e00 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/2439455360 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 0x7fb4c8100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 -- 172.21.15.69:0/2439455360 >> 172.21.15.69:0/2439455360 conn(0x7fb4c80fbd00 msgr2=0x7fb4c80fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 -- 172.21.15.69:0/2439455360 shutdown_connections 2024-05-09T09:39:57.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 -- 172.21.15.69:0/2439455360 wait complete. 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 Processor -- start 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.906+0000 7fb4cf47d640 1 -- start start 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 0x7fb4c8104720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 0x7fb4c81032a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c810c410 con 0x7fb4c8101980 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c810c290 con 0x7fb4c8100790 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c810c590 con 0x7fb4c8102340 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34924/0 (socket says 172.21.15.69:34924) 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 -- 172.21.15.69:0/1588211996 learned_addr learned my addr 172.21.15.69:0/1588211996 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 0x7fb4c8104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 0x7fb4c81032a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 -- 172.21.15.69:0/1588211996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 msgr2=0x7fb4c81032a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 0x7fb4c81032a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 -- 172.21.15.69:0/1588211996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 msgr2=0x7fb4c8104720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 0x7fb4c8104720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 -- 172.21.15.69:0/1588211996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb4c8103b30 con 0x7fb4c8101980 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009230 tx=0x7fb4bc0042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4c67fc640 1 -- 172.21.15.69:0/1588211996 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc03f030 con 0x7fb4c8101980 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4c67fc640 1 -- 172.21.15.69:0/1588211996 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4bc003ce0 con 0x7fb4c8101980 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.907+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb4c81a1bb0 con 0x7fb4c8101980 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.908+0000 7fb4c67fc640 1 -- 172.21.15.69:0/1588211996 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc035070 con 0x7fb4c8101980 2024-05-09T09:39:57.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.908+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb4c81a2100 con 0x7fb4c8101980 2024-05-09T09:39:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.910+0000 7fb4d218c740 1 Processor -- start 2024-05-09T09:39:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 -- start start 2024-05-09T09:39:57.916 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf8e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf91240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebeed340 con 0x55f8ebf89250 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebe95290 con 0x55f8ebf8edb0 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4d218c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebee9300 con 0x55f8ebf89c40 2024-05-09T09:39:57.917 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4c67fc640 1 -- 172.21.15.69:0/1588211996 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb4bc042030 con 0x7fb4c8101980 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34940/0 (socket says 172.21.15.69:34940) 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 -- 172.21.15.69:0/524001680 learned_addr learned my addr 172.21.15.69:0/524001680 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cd219640 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf8e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cca18640 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf91240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 -- 172.21.15.69:0/524001680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 msgr2=0x55f8ebf8e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf8e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 -- 172.21.15.69:0/524001680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 msgr2=0x55f8ebf91240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf91240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.918 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.911+0000 7fb4cda1a640 1 -- 172.21.15.69:0/524001680 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ebeb98e0 con 0x55f8ebf89250 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.912+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009ca0 tx=0x7fb4c0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58768c15bbf521b3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.912+0000 7fb4af7fe640 1 -- 172.21.15.69:0/524001680 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c001e070 con 0x55f8ebf89250 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.912+0000 7fb4af7fe640 1 -- 172.21.15.69:0/524001680 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c0009e50 con 0x55f8ebf89250 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4af7fe640 1 -- 172.21.15.69:0/524001680 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4c000ea80 con 0x55f8ebf89250 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4c67fc640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 0x7fb4b00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 0x7fb4b00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4c67fc640 1 -- 172.21.15.69:0/1588211996 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb4bc0ef890 con 0x7fb4c8101980 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 -- 172.21.15.69:0/524001680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 msgr2=0x55f8ebf89670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 0x7fb4b00a50f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f8ebf8a040 tx=0x7fb4b8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fb4c0009ca0 tx=0x7fb4c0011810 comp rx=0 tx=0).stop 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 -- 172.21.15.69:0/524001680 shutdown_connections 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf91240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf8e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.919 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 --2- 172.21.15.69:0/524001680 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf89670 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 -- 172.21.15.69:0/524001680 >> 172.21.15.69:0/524001680 conn(0x55f8ebf848c0 msgr2=0x55f8ebf84cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.913+0000 7fb4d218c740 1 -- 172.21.15.69:0/524001680 shutdown_connections 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 -- 172.21.15.69:0/524001680 wait complete. 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 -- start start 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 Processor -- start 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf93a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf9aa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebf91d60 con 0x55f8ebf89250 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebea8790 con 0x55f8ebf8edb0 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55f8ebf91ee0 con 0x55f8ebf89c40 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4d218c740 10 client.0 _ll_register_callbacks cb 0x55f8ebee3a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:57.920 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.914+0000 7fb4cd219640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34948/0 (socket says 172.21.15.69:34948) 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 -- 172.21.15.69:0/1055559899 learned_addr learned my addr 172.21.15.69:0/1055559899 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf93a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf9aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 -- 172.21.15.69:0/1055559899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 msgr2=0x55f8ebf93a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf93a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 -- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 msgr2=0x55f8ebf9aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf9aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 -- 172.21.15.69:0/1055559899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ebc01210 con 0x55f8ebf89250 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4cd219640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8003f60 tx=0x7fb4b8004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=920f5548911521d2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4acff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800f540 con 0x55f8ebf89250 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4c0009e50 con 0x55f8ebf89250 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb4b800f6e0 con 0x55f8ebf89250 2024-05-09T09:39:57.921 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb4b8032050 con 0x55f8ebf89250 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4d218c740 10 client.17688 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.915+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f8ebccd620 con 0x55f8ebf89250 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4c0009e50 con 0x55f8ebf89250 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb4b8026030 con 0x55f8ebf89250 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 10 client.17688 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 10 client.17688 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 20 client.17688 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf9f030 0x55f8ebfa14a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 4 client.17688 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55f8ebefaa70 con 0x55f8ebf9f030 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.916+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf9f030 0x55f8ebfa14a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.922 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.917+0000 7fb4acff9640 10 client.17688 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:57.923 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.917+0000 7fb4cca18640 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf9f030 0x55f8ebfa14a0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe1aabb91999ea4 server_cookie=9ef599a1209e7bc8 in_seq=0 out_seq=0 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4acff9640 1 -- 172.21.15.69:0/1055559899 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55f8ebefaa70 con 0x55f8ebf9f030 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4acff9640 10 client.17688 handle_command_reply: tid=0 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 client.17688 shutdown 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 msgr2=0x55f8ebf934f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fb4b8003f60 tx=0x7fb4b8004ab0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 shutdown_connections 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf8edb0 0x55f8ebf9aa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf9f030 0x55f8ebfa14a0 crc :-1 s=READY pgs=246 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf93a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 --2- 172.21.15.69:0/1055559899 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf934f0 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.924 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4acff9640 0 client.17688 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 >> 172.21.15.69:0/1055559899 conn(0x55f8ebf848c0 msgr2=0x55f8ebf83a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 shutdown_connections 2024-05-09T09:39:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 1 -- 172.21.15.69:0/1055559899 wait complete. 2024-05-09T09:39:57.925 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.921+0000 7fb4d218c740 20 client.17688 trim_cache size 0 max 16384 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 Processor -- start 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 -- start start 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89250 0x55f8ebf9aa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf89c40 0x55f8ebf83170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81aee50 con 0x55f8ebf848c0 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ac720 con 0x55f8ebf89c40 2024-05-09T09:39:57.944 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.939+0000 7fb4d218c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81aa570 con 0x55f8ebf89250 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34956/0 (socket says 172.21.15.69:34956) 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 -- 172.21.15.69:0/576369986 learned_addr learned my addr 172.21.15.69:0/576369986 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cca18640 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89250 0x55f8ebf9aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf89c40 0x55f8ebf83170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 -- 172.21.15.69:0/576369986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89250 msgr2=0x55f8ebf9aa30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89250 0x55f8ebf9aa30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 -- 172.21.15.69:0/576369986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf89c40 msgr2=0x55f8ebf83170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf89c40 0x55f8ebf83170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 -- 172.21.15.69:0/576369986 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55f8ebf9d990 con 0x55f8ebf848c0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4cd219640 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb4b801dad0 tx=0x7fb4b8004420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=da60886085b43de2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4aeffd640 1 -- 172.21.15.69:0/576369986 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b8004b20 con 0x55f8ebf848c0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4aeffd640 1 -- 172.21.15.69:0/576369986 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4b8004cc0 con 0x55f8ebf848c0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.940+0000 7fb4aeffd640 1 -- 172.21.15.69:0/576369986 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4b800f870 con 0x55f8ebf848c0 2024-05-09T09:39:57.945 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- 172.21.15.69:0/576369986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 msgr2=0x55f8ebf93810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fb4b801dad0 tx=0x7fb4b8004420 comp rx=0 tx=0).stop 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- 172.21.15.69:0/576369986 shutdown_connections 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf89c40 0x55f8ebf83170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89250 0x55f8ebf9aa30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 --2- 172.21.15.69:0/576369986 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf848c0 0x55f8ebf93810 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- 172.21.15.69:0/576369986 >> 172.21.15.69:0/576369986 conn(0x55f8ebf7f9b0 msgr2=0x55f8ebf93100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- 172.21.15.69:0/576369986 shutdown_connections 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- 172.21.15.69:0/576369986 wait complete. 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 -- start start 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.941+0000 7fb4d218c740 1 Processor -- start 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf848c0 0x55f8ebf98400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf99950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.946 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81aabe0 con 0x55f8ebf89250 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ac050 con 0x55f8ebf848c0 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb4c81ab2b0 con 0x55f8ebf89c40 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4d218c740 10 client.0 _ll_register_callbacks cb 0x7fb4b8009e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:34960/0 (socket says 172.21.15.69:34960) 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 -- 172.21.15.69:0/2570645701 learned_addr learned my addr 172.21.15.69:0/2570645701 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cda1a640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf99950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cd219640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf848c0 0x55f8ebf98400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 -- 172.21.15.69:0/2570645701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 msgr2=0x55f8ebf99950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf99950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.947 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 -- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf848c0 msgr2=0x55f8ebf98400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf848c0 0x55f8ebf98400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 -- 172.21.15.69:0/2570645701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55f8ebf99f40 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4cca18640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0ecfb0 tx=0x7fb4bc0bb900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99a5fe2cdb0ee928 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4acff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.942+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb4bc049040 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb4bc0ba0c0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fb4b8004cc0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fb4bc0c00c0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 10 client.17694 fetch_fsmap learned FSMap version 47 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55f8ec0059d0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb4b8004cc0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fb4bc0f1ce0 con 0x55f8ebf89250 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 10 client.17694 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 10 client.17694 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 20 client.17694 populate_metadata read hostname 'smithi069' 2024-05-09T09:39:57.948 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf8edb0 0x55f8ec008410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:39:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 4 client.17694 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:39:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55f8ebef4dc0 con 0x55f8ebf8edb0 2024-05-09T09:39:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.943+0000 7fb4cd219640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf8edb0 0x55f8ec008410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:39:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.944+0000 7fb4acff9640 10 client.17694 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:57.944+0000 7fb4cd219640 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf8edb0 0x55f8ec008410 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cbc2cde415ae9167 server_cookie=ae7965dd8e3a8aaf in_seq=0 out_seq=0 2024-05-09T09:39:58.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 251604 27.88 MB/sec execute 356 sec latency 479.622 ms 2024-05-09T09:39:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4acff9640 1 -- 172.21.15.69:0/2570645701 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55f8ebef4dc0 con 0x55f8ebf8edb0 2024-05-09T09:39:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4acff9640 10 client.17694 handle_command_reply: tid=0 2024-05-09T09:39:58.960 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4d218c740 1 client.17694 shutdown 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 msgr2=0x55f8ebf98940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc0ecfb0 tx=0x7fb4bc0bb900 comp rx=0 tx=0).stop 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 shutdown_connections 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.958+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55f8ebf8edb0 0x55f8ec008410 crc :-1 s=READY pgs=247 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4acff9640 0 client.17694 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:39:58.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55f8ebf89c40 0x55f8ebf99950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55f8ebf89250 0x55f8ebf98940 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 --2- 172.21.15.69:0/2570645701 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55f8ebf848c0 0x55f8ebf98400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 >> 172.21.15.69:0/2570645701 conn(0x55f8ebf7f9b0 msgr2=0x55f8ebf87640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 shutdown_connections 2024-05-09T09:39:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 1 -- 172.21.15.69:0/2570645701 wait complete. 2024-05-09T09:39:58.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.959+0000 7fb4d218c740 20 client.17694 trim_cache size 0 max 16384 2024-05-09T09:39:58.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:39:58.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:39:58.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:39:58.964 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.960+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 msgr2=0x7fb4b00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.960+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 0x7fb4b00a50f0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x55f8ebf8a040 tx=0x7fb4b8009500 comp rx=0 tx=0).stop 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.960+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 msgr2=0x7fb4c8102d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.960+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7fb4bc009230 tx=0x7fb4bc0042f0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 shutdown_connections 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb4b00a2c30 0x7fb4b00a50f0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb4c8102340 0x7fb4c81032a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb4c8101980 0x7fb4c8102d60 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.965 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 --2- 172.21.15.69:0/1588211996 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb4c8100790 0x7fb4c8104720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:39:58.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.962+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 >> 172.21.15.69:0/1588211996 conn(0x7fb4c80fbd00 msgr2=0x7fb4c80fd720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:39:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.967+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 shutdown_connections 2024-05-09T09:39:58.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:39:58.967+0000 7fb4cf47d640 1 -- 172.21.15.69:0/1588211996 wait complete. 2024-05-09T09:39:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:39:59 smithi103 ceph-mon[34421]: pgmap v1002: 129 pgs: 129 active+clean; 711 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.4 MiB/s wr, 519 op/s 2024-05-09T09:39:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:39:59 smithi069 ceph-mon[30123]: pgmap v1002: 129 pgs: 129 active+clean; 711 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.4 MiB/s wr, 519 op/s 2024-05-09T09:39:59.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:39:59 smithi163 ceph-mon[34354]: pgmap v1002: 129 pgs: 129 active+clean; 711 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.4 MiB/s wr, 519 op/s 2024-05-09T09:39:59.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 251978 27.85 MB/sec execute 357 sec latency 362.138 ms 2024-05-09T09:40:00.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:39:59.999+0000 7f01b9202700 -1 log_channel(cluster) log [ERR] : overall HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:40:00.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:00 smithi103 ceph-mon[34421]: overall HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:40:00.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:00 smithi069 ceph-mon[30123]: overall HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:40:00.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:00 smithi163 ceph-mon[34354]: overall HEALTH_ERR 2 MDSs report damaged metadata 2024-05-09T09:40:00.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 252393 27.84 MB/sec execute 358 sec latency 171.416 ms 2024-05-09T09:40:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:01 smithi103 ceph-mon[34421]: pgmap v1003: 129 pgs: 129 active+clean; 707 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-05-09T09:40:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:01 smithi069 ceph-mon[30123]: pgmap v1003: 129 pgs: 129 active+clean; 707 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-05-09T09:40:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:01 smithi163 ceph-mon[34354]: pgmap v1003: 129 pgs: 129 active+clean; 707 MiB data, 9.7 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.0 MiB/s wr, 559 op/s 2024-05-09T09:40:01.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 253076 27.87 MB/sec execute 359 sec latency 133.903 ms 2024-05-09T09:40:02.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 253640 27.88 MB/sec execute 360 sec latency 70.916 ms 2024-05-09T09:40:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:03 smithi103 ceph-mon[34421]: pgmap v1004: 129 pgs: 129 active+clean; 708 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 646 op/s 2024-05-09T09:40:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:03 smithi069 ceph-mon[30123]: pgmap v1004: 129 pgs: 129 active+clean; 708 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 646 op/s 2024-05-09T09:40:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:03 smithi163 ceph-mon[34354]: pgmap v1004: 129 pgs: 129 active+clean; 708 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 7.7 MiB/s wr, 646 op/s 2024-05-09T09:40:03.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 254149 27.89 MB/sec execute 361 sec latency 105.791 ms 2024-05-09T09:40:04.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 254382 27.85 MB/sec execute 362 sec latency 268.519 ms 2024-05-09T09:40:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:05 smithi103 ceph-mon[34421]: pgmap v1005: 129 pgs: 129 active+clean; 703 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-05-09T09:40:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:05 smithi069 ceph-mon[30123]: pgmap v1005: 129 pgs: 129 active+clean; 703 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-05-09T09:40:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:05 smithi163 ceph-mon[34354]: pgmap v1005: 129 pgs: 129 active+clean; 703 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.5 MiB/s wr, 606 op/s 2024-05-09T09:40:05.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 254630 27.81 MB/sec execute 363 sec latency 196.887 ms 2024-05-09T09:40:06.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 255207 27.81 MB/sec execute 364 sec latency 58.588 ms 2024-05-09T09:40:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:07 smithi103 ceph-mon[34421]: pgmap v1006: 129 pgs: 129 active+clean; 694 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 726 op/s 2024-05-09T09:40:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:07 smithi069 ceph-mon[30123]: pgmap v1006: 129 pgs: 129 active+clean; 694 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 726 op/s 2024-05-09T09:40:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:07 smithi163 ceph-mon[34354]: pgmap v1006: 129 pgs: 129 active+clean; 694 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 8.2 MiB/s wr, 726 op/s 2024-05-09T09:40:07.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 255857 27.82 MB/sec execute 365 sec latency 80.005 ms 2024-05-09T09:40:08.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 256472 27.84 MB/sec execute 366 sec latency 59.465 ms 2024-05-09T09:40:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:09 smithi069 ceph-mon[30123]: pgmap v1007: 129 pgs: 129 active+clean; 695 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:09 smithi163 ceph-mon[34354]: pgmap v1007: 129 pgs: 129 active+clean; 695 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:09.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 256746 27.81 MB/sec execute 367 sec latency 557.715 ms 2024-05-09T09:40:09.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:09 smithi103 ceph-mon[34421]: pgmap v1007: 129 pgs: 129 active+clean; 695 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:10.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 257046 27.79 MB/sec execute 368 sec latency 82.670 ms 2024-05-09T09:40:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:11 smithi103 ceph-mon[34421]: pgmap v1008: 129 pgs: 129 active+clean; 696 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.7 MiB/s wr, 558 op/s 2024-05-09T09:40:11.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:11 smithi069 ceph-mon[30123]: pgmap v1008: 129 pgs: 129 active+clean; 696 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.7 MiB/s wr, 558 op/s 2024-05-09T09:40:11.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:11 smithi163 ceph-mon[34354]: pgmap v1008: 129 pgs: 129 active+clean; 696 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.7 MiB/s wr, 558 op/s 2024-05-09T09:40:11.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 257638 27.78 MB/sec execute 369 sec latency 72.909 ms 2024-05-09T09:40:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.109+0000 7f44d559a640 1 -- 172.21.15.69:0/4256712981 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d00319b0 con 0x7f44d8109760 2024-05-09T09:40:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 -- 172.21.15.69:0/4256712981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8109760 msgr2=0x7f44d810bbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 --2- 172.21.15.69:0/4256712981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8109760 0x7f44d810bbf0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f44d0009900 tx=0x7f44d002a3f0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 -- 172.21.15.69:0/4256712981 shutdown_connections 2024-05-09T09:40:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 --2- 172.21.15.69:0/4256712981 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8109760 0x7f44d810bbf0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 --2- 172.21.15.69:0/4256712981 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d8074290 0x7f44d8074710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 --2- 172.21.15.69:0/4256712981 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d8075870 0x7f44d8073cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 -- 172.21.15.69:0/4256712981 >> 172.21.15.69:0/4256712981 conn(0x7f44d80fcdf0 msgr2=0x7f44d80ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 -- 172.21.15.69:0/4256712981 shutdown_connections 2024-05-09T09:40:12.112 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.110+0000 7f44dcb90640 1 -- 172.21.15.69:0/4256712981 wait complete. 2024-05-09T09:40:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.111+0000 7f44dcb90640 1 Processor -- start 2024-05-09T09:40:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 -- start start 2024-05-09T09:40:12.113 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d8074290 0x7f44d819e070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d8109760 0x7f44d81a5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44d810f730 con 0x7f44d8074290 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d810f5b0 con 0x7f44d8109760 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44dcb90640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44d810f8b0 con 0x7f44d8075870 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:55086/0 (socket says 172.21.15.69:55086) 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.112+0000 7f44d5d9b640 1 -- 172.21.15.69:0/929906562 learned_addr learned my addr 172.21.15.69:0/929906562 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 -- 172.21.15.69:0/929906562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d8109760 msgr2=0x7f44d81a5620 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d8109760 0x7f44d81a5620 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.114 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 -- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d8074290 msgr2=0x7f44d819e070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d8074290 0x7f44d819e070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 -- 172.21.15.69:0/929906562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f44d81a5c90 con 0x7f44d8075870 2024-05-09T09:40:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00d950 tx=0x7f44cc00de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.115 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.113+0000 7f44c77fe640 1 -- 172.21.15.69:0/929906562 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc017070 con 0x7f44d8075870 2024-05-09T09:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.114+0000 7f44c77fe640 1 -- 172.21.15.69:0/929906562 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44cc009c30 con 0x7f44d8075870 2024-05-09T09:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.114+0000 7f44c77fe640 1 -- 172.21.15.69:0/929906562 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc00f6b0 con 0x7f44d8075870 2024-05-09T09:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.114+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f44d81a5f80 con 0x7f44d8075870 2024-05-09T09:40:12.116 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.114+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f44d81a6450 con 0x7f44d8075870 2024-05-09T09:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.118+0000 7f44c77fe640 1 -- 172.21.15.69:0/929906562 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f44cc0109d0 con 0x7f44d8075870 2024-05-09T09:40:12.119 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.119+0000 7f44df980740 1 Processor -- start 2024-05-09T09:40:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.120+0000 7f44c77fe640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 0x7f44c00a51f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.120 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.120+0000 7f44c77fe640 1 -- 172.21.15.69:0/929906562 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f44cc0cf0a0 con 0x7f44d8075870 2024-05-09T09:40:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 -- start start 2024-05-09T09:40:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998de670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 0x55a4998e6240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.121 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a499842340 con 0x55a4998dec40 2024-05-09T09:40:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a4997ea290 con 0x55a4998e3db0 2024-05-09T09:40:12.122 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44df980740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a49983e300 con 0x55a4998de250 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 0x7f44c00a51f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58118/0 (socket says 172.21.15.69:58118) 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44d5d9b640 1 -- 172.21.15.69:0/2480367497 learned_addr learned my addr 172.21.15.69:0/2480367497 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.121+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 0x55a4998e6240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.123 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d659c640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998de670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 -- 172.21.15.69:0/2480367497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 msgr2=0x55a4998de670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998de670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 -- 172.21.15.69:0/2480367497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 msgr2=0x55a4998e3740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 -- 172.21.15.69:0/2480367497 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a49980e8e0 con 0x55a4998e3db0 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 0x7f44c00a51f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f44d00096c0 tx=0x7f44d0005e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 0x55a4998e6240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44d002a910 tx=0x7f44d0043000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c2a3a6050145e496 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.124 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.122+0000 7f44c4ff9640 1 -- 172.21.15.69:0/2480367497 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d004e070 con 0x55a4998e3db0 2024-05-09T09:40:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.123+0000 7f44c4ff9640 1 -- 172.21.15.69:0/2480367497 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44d003ecb0 con 0x55a4998e3db0 2024-05-09T09:40:12.127 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.123+0000 7f44c4ff9640 1 -- 172.21.15.69:0/2480367497 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44d0049c40 con 0x55a4998e3db0 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- 172.21.15.69:0/2480367497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 msgr2=0x55a4998e6240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 0x55a4998e6240 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f44d002a910 tx=0x7f44d0043000 comp rx=0 tx=0).stop 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- 172.21.15.69:0/2480367497 shutdown_connections 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998e3db0 0x55a4998e6240 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998e3740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 --2- 172.21.15.69:0/2480367497 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998de670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- 172.21.15.69:0/2480367497 >> 172.21.15.69:0/2480367497 conn(0x55a4998d98c0 msgr2=0x55a4998d9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- 172.21.15.69:0/2480367497 shutdown_connections 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- 172.21.15.69:0/2480367497 wait complete. 2024-05-09T09:40:12.128 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 -- start start 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.124+0000 7f44df980740 1 Processor -- start 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998f99e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998e3db0 0x55a499900a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a4998f7d10 con 0x55a4998dec40 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a4997fd790 con 0x55a4998de250 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a4998f7e90 con 0x55a4998e3db0 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44df980740 10 client.0 _ll_register_callbacks cb 0x55a499838a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:59586/0 (socket says 172.21.15.69:59586) 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d659c640 1 -- 172.21.15.69:0/379503086 learned_addr learned my addr 172.21.15.69:0/379503086 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998e3db0 0x55a499900a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998f99e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d659c640 1 -- 172.21.15.69:0/379503086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998e3db0 msgr2=0x55a499900a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.125+0000 7f44d659c640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998e3db0 0x55a499900a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44d659c640 1 -- 172.21.15.69:0/379503086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 msgr2=0x55a4998f99e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44d659c640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998f99e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44d659c640 1 -- 172.21.15.69:0/379503086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a499556210 con 0x55a4998de250 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44d659c640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44c8010900 tx=0x7f44c8010dc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7b939212bc69b97f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.126+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8017070 con 0x55a4998de250 2024-05-09T09:40:12.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.127+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44d003ecb0 con 0x55a4998de250 2024-05-09T09:40:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.127+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f44c800ed80 con 0x55a4998de250 2024-05-09T09:40:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.127+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f44c8024050 con 0x55a4998de250 2024-05-09T09:40:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.127+0000 7f44df980740 10 client.26963 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:12.131 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.127+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a499622620 con 0x55a4998de250 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44d003ecb0 con 0x55a4998de250 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f44c8011440 con 0x55a4998de250 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 10 client.26963 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 10 client.26963 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 20 client.26963 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a499915ef0 0x55a499918360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 4 client.26963 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.128+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a49984fa70 con 0x55a499915ef0 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.129+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a499915ef0 0x55a499918360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.130+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a499915ef0 0x55a499918360 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ff81a3e9325db638 server_cookie=9d62c7b50ac207f3 in_seq=0 out_seq=0 2024-05-09T09:40:12.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.130+0000 7f44bdffb640 10 client.26963 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.137+0000 7f44bdffb640 1 -- 172.21.15.69:0/379503086 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a49984fa70 con 0x55a499915ef0 2024-05-09T09:40:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.137+0000 7f44bdffb640 10 client.26963 handle_command_reply: tid=0 2024-05-09T09:40:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.137+0000 7f44df980740 1 client.26963 shutdown 2024-05-09T09:40:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.137+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 msgr2=0x55a4998f94a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:12.138 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.137+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f44c8010900 tx=0x7f44c8010dc0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 shutdown_connections 2024-05-09T09:40:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a499915ef0 0x55a499918360 crc :-1 s=READY pgs=248 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998e3db0 0x55a499900a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.139 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998dec40 0x55a4998f99e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 --2- 172.21.15.69:0/379503086 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998de250 0x55a4998f94a0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 >> 172.21.15.69:0/379503086 conn(0x55a4998d98c0 msgr2=0x55a4998d8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44bdffb640 0 client.26963 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 shutdown_connections 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 1 -- 172.21.15.69:0/379503086 wait complete. 2024-05-09T09:40:12.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.138+0000 7f44df980740 20 client.26963 trim_cache size 0 max 16384 2024-05-09T09:40:12.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 Processor -- start 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 -- start start 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998d98c0 0x55a4998dcd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998d8170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806f660 con 0x55a4998de250 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806cf30 con 0x55a4998dec40 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.160+0000 7f44df980740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44d819d150 con 0x55a4998d98c0 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58142/0 (socket says 172.21.15.69:58142) 2024-05-09T09:40:12.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 -- 172.21.15.69:0/1606683305 learned_addr learned my addr 172.21.15.69:0/1606683305 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d659c640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998d98c0 0x55a4998dcd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998d8170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 -- 172.21.15.69:0/1606683305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998d98c0 msgr2=0x55a4998dcd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998d98c0 0x55a4998dcd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 -- 172.21.15.69:0/1606683305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 msgr2=0x55a4998d8170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998d8170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 -- 172.21.15.69:0/1606683305 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a4998df9b0 con 0x55a4998de250 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998d8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00dfe0 tx=0x7f44cc00bf60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=766da35cd64b919b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44bffff640 1 -- 172.21.15.69:0/1606683305 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc0b4070 con 0x55a4998de250 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44bffff640 1 -- 172.21.15.69:0/1606683305 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44cc09f8d0 con 0x55a4998de250 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.161+0000 7f44bffff640 1 -- 172.21.15.69:0/1606683305 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44cc020040 con 0x55a4998de250 2024-05-09T09:40:12.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- 172.21.15.69:0/1606683305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 msgr2=0x55a4999009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00dfe0 tx=0x7f44cc00bf60 comp rx=0 tx=0).stop 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- 172.21.15.69:0/1606683305 shutdown_connections 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998d8170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998de250 0x55a4999009e0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 --2- 172.21.15.69:0/1606683305 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998d98c0 0x55a4998dcd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- 172.21.15.69:0/1606683305 >> 172.21.15.69:0/1606683305 conn(0x55a4998e3880 msgr2=0x55a4998dc640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- 172.21.15.69:0/1606683305 shutdown_connections 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- 172.21.15.69:0/1606683305 wait complete. 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.162+0000 7f44df980740 1 -- start start 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 Processor -- start 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998fe140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998ff100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.169 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806b3f0 con 0x55a4998d98c0 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806c860 con 0x55a4998dec40 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f44d806bac0 con 0x55a4998de250 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44df980740 10 client.0 _ll_register_callbacks cb 0x7f44c80099c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:58146/0 (socket says 172.21.15.69:58146) 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d5d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998fe140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 -- 172.21.15.69:0/156959321 learned_addr learned my addr 172.21.15.69:0/156959321 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998ff100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 -- 172.21.15.69:0/156959321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 msgr2=0x55a4998fe140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998fe140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 -- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 msgr2=0x55a4998ff100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:12.170 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998ff100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 -- 172.21.15.69:0/156959321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a4998ff6f0 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d6d9d640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44d659c640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f44c8002230 tx=0x7f44c8011670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26cb9c0e67b732c9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.163+0000 7f44bdffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f44c8017040 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f44cc09f8d0 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f44c8011aa0 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f44c8039050 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 10 client.17709 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a499900a70 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f44cc09f8d0 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f44c8004040 con 0x55a4998d98c0 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 10 client.17709 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:12.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 10 client.17709 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 20 client.17709 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a4998e3db0 0x55a49997ea80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 4 client.17709 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.164+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a499849dc0 con 0x55a4998e3db0 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.165+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a4998e3db0 0x55a49997ea80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.165+0000 7f44d5d9b640 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a4998e3db0 0x55a49997ea80 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82f248e455580fc4 server_cookie=a3168558aac1898f in_seq=0 out_seq=0 2024-05-09T09:40:12.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:12.165+0000 7f44bdffb640 10 client.17709 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:12.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 257848 27.74 MB/sec execute 370 sec latency 588.126 ms 2024-05-09T09:40:13.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44bdffb640 1 -- 172.21.15.69:0/156959321 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a499849dc0 con 0x55a4998e3db0 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44bdffb640 10 client.17709 handle_command_reply: tid=0 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 client.17709 shutdown 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 msgr2=0x55a4998fdc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7f44c8002230 tx=0x7f44c8011670 comp rx=0 tx=0).stop 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 shutdown_connections 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a4998dec40 0x55a4998ff100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.007+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a4998e3db0 0x55a49997ea80 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44bdffb640 0 client.17709 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a4998de250 0x55a4998fe140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 1 --2- 172.21.15.69:0/156959321 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a4998d98c0 0x55a4998fdc00 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 >> 172.21.15.69:0/156959321 conn(0x55a4998e3880 msgr2=0x55a4998d8a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 shutdown_connections 2024-05-09T09:40:13.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 1 -- 172.21.15.69:0/156959321 wait complete. 2024-05-09T09:40:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.008+0000 7f44df980740 20 client.17709 trim_cache size 0 max 16384 2024-05-09T09:40:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:40:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:40:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:40:13.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:40:13.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 msgr2=0x7f44c00a51f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 0x7f44c00a51f0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7f44d00096c0 tx=0x7f44d0005e80 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 msgr2=0x7f44d819e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f44cc00d950 tx=0x7f44cc00de10 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 shutdown_connections 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f44c00a2d30 0x7f44c00a51f0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f44d8109760 0x7f44d81a5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f44d8075870 0x7f44d819e5b0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 --2- 172.21.15.69:0/929906562 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f44d8074290 0x7f44d819e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.009+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 >> 172.21.15.69:0/929906562 conn(0x7f44d80fcdf0 msgr2=0x7f44d8101250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.010+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 shutdown_connections 2024-05-09T09:40:13.012 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:13.010+0000 7f44dcb90640 1 -- 172.21.15.69:0/929906562 wait complete. 2024-05-09T09:40:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:13 smithi069 ceph-mon[30123]: pgmap v1009: 129 pgs: 129 active+clean; 706 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-05-09T09:40:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:13 smithi163 ceph-mon[34354]: pgmap v1009: 129 pgs: 129 active+clean; 706 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-05-09T09:40:13.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 258455 27.75 MB/sec execute 371 sec latency 631.431 ms 2024-05-09T09:40:13.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:13 smithi103 ceph-mon[34421]: pgmap v1009: 129 pgs: 129 active+clean; 706 MiB data, 9.8 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 7.6 MiB/s wr, 595 op/s 2024-05-09T09:40:14.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 258738 27.73 MB/sec execute 372 sec latency 357.548 ms 2024-05-09T09:40:15.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:15 smithi069 ceph-mon[30123]: pgmap v1010: 129 pgs: 129 active+clean; 711 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 536 op/s 2024-05-09T09:40:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:15 smithi163 ceph-mon[34354]: pgmap v1010: 129 pgs: 129 active+clean; 711 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 536 op/s 2024-05-09T09:40:15.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 259037 27.69 MB/sec execute 373 sec latency 177.352 ms 2024-05-09T09:40:15.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:15 smithi103 ceph-mon[34421]: pgmap v1010: 129 pgs: 129 active+clean; 711 MiB data, 9.9 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.0 MiB/s wr, 536 op/s 2024-05-09T09:40:16.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 259654 27.72 MB/sec execute 374 sec latency 58.774 ms 2024-05-09T09:40:17.669 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:17 smithi103 ceph-mon[34421]: pgmap v1011: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.4 MiB/s wr, 667 op/s 2024-05-09T09:40:17.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:17 smithi069 ceph-mon[30123]: pgmap v1011: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.4 MiB/s wr, 667 op/s 2024-05-09T09:40:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:17 smithi163 ceph-mon[34354]: pgmap v1011: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 8.4 MiB/s wr, 667 op/s 2024-05-09T09:40:17.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 260135 27.72 MB/sec execute 375 sec latency 117.923 ms 2024-05-09T09:40:18.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 260754 27.71 MB/sec execute 376 sec latency 80.067 ms 2024-05-09T09:40:19.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:19 smithi069 ceph-mon[30123]: pgmap v1012: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:40:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:19 smithi163 ceph-mon[34354]: pgmap v1012: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:40:19.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 261045 27.67 MB/sec execute 377 sec latency 386.541 ms 2024-05-09T09:40:19.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:19 smithi103 ceph-mon[34421]: pgmap v1012: 129 pgs: 129 active+clean; 700 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:40:20.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 261472 27.67 MB/sec execute 378 sec latency 89.423 ms 2024-05-09T09:40:21.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:21 smithi069 ceph-mon[30123]: pgmap v1013: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 638 op/s 2024-05-09T09:40:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:21 smithi163 ceph-mon[34354]: pgmap v1013: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 638 op/s 2024-05-09T09:40:21.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 262290 27.71 MB/sec execute 379 sec latency 60.574 ms 2024-05-09T09:40:21.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:21 smithi103 ceph-mon[34421]: pgmap v1013: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 7.2 MiB/s wr, 638 op/s 2024-05-09T09:40:22.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 262937 27.75 MB/sec execute 380 sec latency 57.695 ms 2024-05-09T09:40:23.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:23 smithi069 ceph-mon[30123]: pgmap v1014: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.5 MiB/s wr, 698 op/s 2024-05-09T09:40:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:23 smithi163 ceph-mon[34354]: pgmap v1014: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.5 MiB/s wr, 698 op/s 2024-05-09T09:40:23.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 263326 27.74 MB/sec execute 381 sec latency 70.441 ms 2024-05-09T09:40:23.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:23 smithi103 ceph-mon[34421]: pgmap v1014: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.5 MiB/s wr, 698 op/s 2024-05-09T09:40:24.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 263510 27.68 MB/sec execute 382 sec latency 536.030 ms 2024-05-09T09:40:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:25 smithi069 ceph-mon[30123]: pgmap v1015: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-05-09T09:40:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:25 smithi163 ceph-mon[34354]: pgmap v1015: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-05-09T09:40:25.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 263769 27.64 MB/sec execute 383 sec latency 109.764 ms 2024-05-09T09:40:25.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:25 smithi103 ceph-mon[34421]: pgmap v1015: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 622 op/s 2024-05-09T09:40:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 -- 172.21.15.69:0/2644044782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc103460 msgr2=0x7f58fc109cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 --2- 172.21.15.69:0/2644044782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc103460 0x7f58fc109cf0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f58f4009be0 tx=0x7f58f402ead0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 -- 172.21.15.69:0/2644044782 shutdown_connections 2024-05-09T09:40:26.158 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 --2- 172.21.15.69:0/2644044782 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc103460 0x7f58fc109cf0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 --2- 172.21.15.69:0/2644044782 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc102aa0 0x7f58fc102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 --2- 172.21.15.69:0/2644044782 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc1018b0 0x7f58fc101cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.159 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.157+0000 7f59026bd640 1 -- 172.21.15.69:0/2644044782 >> 172.21.15.69:0/2644044782 conn(0x7f58fc0fce20 msgr2=0x7f58fc0ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.159+0000 7f59026bd640 1 -- 172.21.15.69:0/2644044782 shutdown_connections 2024-05-09T09:40:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.159+0000 7f59026bd640 1 -- 172.21.15.69:0/2644044782 wait complete. 2024-05-09T09:40:26.160 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.160+0000 7f59026bd640 1 Processor -- start 2024-05-09T09:40:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.160+0000 7f59026bd640 1 -- start start 2024-05-09T09:40:26.161 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc102aa0 0x7f58fc19e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc103460 0x7f58fc1a5630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc113b60 con 0x7f58fc1018b0 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1139e0 con 0x7f58fc102aa0 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f59026bd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc113ce0 con 0x7f58fc103460 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47470/0 (socket says 172.21.15.69:47470) 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 -- 172.21.15.69:0/3962436259 learned_addr learned my addr 172.21.15.69:0/3962436259 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:26.162 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fb7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc102aa0 0x7f58fc19e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f5900c5a640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc103460 0x7f58fc1a5630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 -- 172.21.15.69:0/3962436259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc103460 msgr2=0x7f58fc1a5630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc103460 0x7f58fc1a5630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 -- 172.21.15.69:0/3962436259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc102aa0 msgr2=0x7f58fc19e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc102aa0 0x7f58fc19e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.163 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.161+0000 7f58fbfff640 1 -- 172.21.15.69:0/3962436259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f58fc1a5ca0 con 0x7f58fc1018b0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.162+0000 7f58fbfff640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e9c0 tx=0x7f58f000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.162+0000 7f58f97fa640 1 -- 172.21.15.69:0/3962436259 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f000cd80 con 0x7f58fc1018b0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.162+0000 7f58f97fa640 1 -- 172.21.15.69:0/3962436259 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f00035f0 con 0x7f58fc1018b0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.162+0000 7f58f97fa640 1 -- 172.21.15.69:0/3962436259 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0010690 con 0x7f58fc1018b0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.163+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f58fc1a5f90 con 0x7f58fc1018b0 2024-05-09T09:40:26.164 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.163+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f58fc1a6450 con 0x7f58fc1018b0 2024-05-09T09:40:26.167 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.167+0000 7f58f97fa640 1 -- 172.21.15.69:0/3962436259 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f58f00037a0 con 0x7f58fc1018b0 2024-05-09T09:40:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.168+0000 7f58f97fa640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 0x7f58e40a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.168+0000 7f58f97fa640 1 -- 172.21.15.69:0/3962436259 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f58f00cf750 con 0x7f58fc1018b0 2024-05-09T09:40:26.168 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.168+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 0x7f58e40a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.168+0000 7f59053d0740 1 Processor -- start 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 -- start start 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985f2670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985f7740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 0x7f58e40a5170 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f58fc19f590 tx=0x7f58ec009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985f2670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985f2670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36622/0 (socket says 172.21.15.69:36622) 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f58fbfff640 1 -- 172.21.15.69:0/1818081870 learned_addr learned my addr 172.21.15.69:0/1818081870 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9985fa240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985f7740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a998556340 con 0x55a9985f2c40 2024-05-09T09:40:26.171 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.169+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a9984fe290 con 0x55a9985f2250 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a998552300 con 0x55a9985f7db0 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f5900c5a640 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9985fa240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 -- 172.21.15.69:0/1818081870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 msgr2=0x55a9985fa240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9985fa240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 -- 172.21.15.69:0/1818081870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 msgr2=0x55a9985f2670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985f2670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 -- 172.21.15.69:0/1818081870 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9985228e0 con 0x55a9985f2c40 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985f7740 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f58ec005fd0 tx=0x7f58ec014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1cc1885f58ebe5d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58e27fc640 1 -- 172.21.15.69:0/1818081870 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec02b070 con 0x55a9985f2c40 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58e27fc640 1 -- 172.21.15.69:0/1818081870 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58ec01ce20 con 0x55a9985f2c40 2024-05-09T09:40:26.172 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.170+0000 7f58e27fc640 1 -- 172.21.15.69:0/1818081870 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58ec00ea90 con 0x55a9985f2c40 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.171+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 msgr2=0x55a9985f7740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.171+0000 7f59053d0740 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985f7740 secure :-1 s=READY pgs=1075 cs=0 l=1 rev1=1 crypto rx=0x7f58ec005fd0 tx=0x7f58ec014000 comp rx=0 tx=0).stop 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 shutdown_connections 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9985fa240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985f7740 unknown :-1 s=CLOSED pgs=1075 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 --2- 172.21.15.69:0/1818081870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985f2670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 >> 172.21.15.69:0/1818081870 conn(0x55a9985ed8c0 msgr2=0x55a9985edcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 shutdown_connections 2024-05-09T09:40:26.177 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 -- 172.21.15.69:0/1818081870 wait complete. 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 -- start start 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.172+0000 7f59053d0740 1 Processor -- start 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985fc980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9986039c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a9985fad60 con 0x55a9985f2c40 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a998511790 con 0x55a9985f2250 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a9985faee0 con 0x55a9985f7db0 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f59053d0740 10 client.0 _ll_register_callbacks cb 0x55a99854ca20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:26.178 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:36624/0 (socket says 172.21.15.69:36624) 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f58fbfff640 1 -- 172.21.15.69:0/1293311919 learned_addr learned my addr 172.21.15.69:0/1293311919 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985fc980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.173+0000 7f5900c5a640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9986039c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 -- 172.21.15.69:0/1293311919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 msgr2=0x55a9986039c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9986039c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 -- 172.21.15.69:0/1293311919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 msgr2=0x55a9985fc980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985fc980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 -- 172.21.15.69:0/1293311919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a99826a210 con 0x55a9985f2250 2024-05-09T09:40:26.179 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985fc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58f00058c0 tx=0x7f58f009c390 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8fddeb342583f825 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f00a4840 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f58f0003b30 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.174+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58ec01ce20 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f58f0081050 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 10 client.26981 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a998336620 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f00cf420 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f58f0097a20 con 0x55a9985f2250 2024-05-09T09:40:26.180 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 10 client.26981 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 10 client.26981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 20 client.26981 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a998607f20 0x55a99860a350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 4 client.26981 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.175+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f58ec01ce20 con 0x55a998607f20 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.176+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a998607f20 0x55a99860a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.176+0000 7f58db7fe640 10 client.26981 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:26.181 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.176+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a998607f20 0x55a99860a350 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a239e7ca865d18a server_cookie=9a5ca7bd08a09d83 in_seq=0 out_seq=0 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f58db7fe640 1 -- 172.21.15.69:0/1293311919 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f58ec01ce20 con 0x55a998607f20 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f58db7fe640 10 client.26981 handle_command_reply: tid=0 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 client.26981 shutdown 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 msgr2=0x55a9985fc440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f58f00058c0 tx=0x7f58f009c390 comp rx=0 tx=0).stop 2024-05-09T09:40:26.183 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 shutdown_connections 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a998607f20 0x55a99860a350 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a9986039c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2c40 0x55a9985fc980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f58db7fe640 0 client.26981 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 --2- 172.21.15.69:0/1293311919 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2250 0x55a9985fc440 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 >> 172.21.15.69:0/1293311919 conn(0x55a9985ed8c0 msgr2=0x55a9985eca00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 shutdown_connections 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 1 -- 172.21.15.69:0/1293311919 wait complete. 2024-05-09T09:40:26.184 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.182+0000 7f59053d0740 20 client.26981 trim_cache size 0 max 16384 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.204+0000 7f59053d0740 1 Processor -- start 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.204+0000 7f59053d0740 1 -- start start 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.204+0000 7f59053d0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a9985f2630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.204+0000 7f59053d0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.204+0000 7f59053d0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f7db0 0x55a9985fc8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.205 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f59053d0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1b3310 con 0x55a9985f2250 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f59053d0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1b0be0 con 0x55a9985f7db0 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fb7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:58918/0 (socket says 172.21.15.69:58918) 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fb7fe640 1 -- 172.21.15.69:0/1676940645 learned_addr learned my addr 172.21.15.69:0/1676940645 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f5900c5a640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f7db0 0x55a9985fc8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a9985f2630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.206 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1aea30 con 0x55a9985f2c40 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 -- 172.21.15.69:0/1676940645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 msgr2=0x55a998603980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 -- 172.21.15.69:0/1676940645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f7db0 msgr2=0x55a9985fc8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f7db0 0x55a9985fc8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fbfff640 1 -- 172.21.15.69:0/1676940645 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a9985ebf00 con 0x55a9985f2250 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.205+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.206+0000 7f58fbfff640 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a9985f2630 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f58f00985a0 tx=0x7f58f009a0e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=147a461d7d39b3af server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.206+0000 7f58e1ffb640 1 -- 172.21.15.69:0/1676940645 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f00a8020 con 0x55a9985f2250 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.206+0000 7f58e1ffb640 1 -- 172.21.15.69:0/1676940645 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f00a4840 con 0x55a9985f2250 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.206+0000 7f58e1ffb640 1 -- 172.21.15.69:0/1676940645 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f0003b30 con 0x55a9985f2250 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 msgr2=0x55a9985f2630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a9985f2630 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 crypto rx=0x7f58f00985a0 tx=0x7f58f009a0e0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 shutdown_connections 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f7db0 0x55a9985fc8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f2c40 0x55a998603980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 --2- 172.21.15.69:0/1676940645 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a9985f2630 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 >> 172.21.15.69:0/1676940645 conn(0x55a9985ed8c0 msgr2=0x55a9985e8620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 shutdown_connections 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- 172.21.15.69:0/1676940645 wait complete. 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.207+0000 7f59053d0740 1 -- start start 2024-05-09T09:40:26.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 Processor -- start 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2c40 0x55a998601460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a998602470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1af0a0 con 0x55a9985f2250 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1b0510 con 0x55a9985f2c40 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f58fc1af770 con 0x55a9985f7db0 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f59053d0740 10 client.0 _ll_register_callbacks cb 0x7f58f0009790 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f58fbfff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47510/0 (socket says 172.21.15.69:47510) 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f58fbfff640 1 -- 172.21.15.69:0/2266697535 learned_addr learned my addr 172.21.15.69:0/2266697535 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.208+0000 7f5900c5a640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a998602470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2c40 0x55a998601460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 -- 172.21.15.69:0/2266697535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 msgr2=0x55a998602470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a998602470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 -- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2c40 msgr2=0x55a998601460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2c40 0x55a998601460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 -- 172.21.15.69:0/2266697535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a998602a60 con 0x55a9985f2250 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58fbfff640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f58f009bb60 tx=0x7f58f000e940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=72333c34baa043b5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58db7fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f58f00a8070 con 0x55a9985f2250 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f58f0097920 con 0x55a9985f2250 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f58f00a4840 con 0x55a9985f2250 2024-05-09T09:40:26.214 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f58f0097b60 con 0x55a9985f2250 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f59053d0740 10 client.17727 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.209+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55a99866e4d0 con 0x55a9985f2250 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f58f009b020 con 0x55a9985f2250 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f58f00a12b0 con 0x55a9985f2250 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 10 client.17727 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 10 client.17727 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 20 client.17727 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a99866ea70 0x55a998670eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 4 client.17727 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7f58f00a4840 con 0x55a99866ea70 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a99866ea70 0x55a998670eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:26.215 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f58db7fe640 10 client.17727 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:26.216 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.210+0000 7f58fb7fe640 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a99866ea70 0x55a998670eb0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6f59f8af3a6f7a1 server_cookie=e87e06e0b1f69f02 in_seq=0 out_seq=0 2024-05-09T09:40:26.838 INFO:tasks.workunit.client.0.smithi069.stdout: 10 264000 27.58 MB/sec execute 384 sec latency 311.782 ms 2024-05-09T09:40:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f58db7fe640 1 -- 172.21.15.69:0/2266697535 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7f58f00a4840 con 0x55a99866ea70 2024-05-09T09:40:26.867 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f58db7fe640 10 client.17727 handle_command_reply: tid=0 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 client.17727 shutdown 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 msgr2=0x55a998600f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 secure :-1 s=READY pgs=1077 cs=0 l=1 rev1=1 crypto rx=0x7f58f009bb60 tx=0x7f58f000e940 comp rx=0 tx=0).stop 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 shutdown_connections 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a99866ea70 0x55a998670eb0 crc :-1 s=READY pgs=251 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f58db7fe640 0 client.17727 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a9985f7db0 0x55a998602470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a9985f2c40 0x55a998601460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 --2- 172.21.15.69:0/2266697535 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a9985f2250 0x55a998600f20 unknown :-1 s=CLOSED pgs=1077 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 >> 172.21.15.69:0/2266697535 conn(0x55a9985ed8c0 msgr2=0x55a9985fc030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 shutdown_connections 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 1 -- 172.21.15.69:0/2266697535 wait complete. 2024-05-09T09:40:26.868 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.866+0000 7f59053d0740 20 client.17727 trim_cache size 0 max 16384 2024-05-09T09:40:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:40:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:40:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:40:26.869 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:40:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 msgr2=0x7f58e40a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 0x7f58e40a5170 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f58fc19f590 tx=0x7f58ec009260 comp rx=0 tx=0).stop 2024-05-09T09:40:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 msgr2=0x7f58fc19e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 secure :-1 s=READY pgs=1074 cs=0 l=1 rev1=1 crypto rx=0x7f58f000e9c0 tx=0x7f58f000ee80 comp rx=0 tx=0).stop 2024-05-09T09:40:26.870 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 shutdown_connections 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f58e40a2cb0 0x7f58e40a5170 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f58fc103460 0x7f58fc1a5630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f58fc102aa0 0x7f58fc19e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 --2- 172.21.15.69:0/3962436259 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f58fc1018b0 0x7f58fc19e080 unknown :-1 s=CLOSED pgs=1074 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 >> 172.21.15.69:0/3962436259 conn(0x7f58fc0fce20 msgr2=0x7f58fc105cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 shutdown_connections 2024-05-09T09:40:26.871 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:26.867+0000 7f59026bd640 1 -- 172.21.15.69:0/3962436259 wait complete. 2024-05-09T09:40:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:27 smithi069 ceph-mon[30123]: pgmap v1016: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 662 op/s 2024-05-09T09:40:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:27 smithi163 ceph-mon[34354]: pgmap v1016: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 662 op/s 2024-05-09T09:40:27.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 264630 27.59 MB/sec execute 385 sec latency 171.013 ms 2024-05-09T09:40:27.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:27 smithi103 ceph-mon[34421]: pgmap v1016: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.4 MiB/s rd, 8.1 MiB/s wr, 662 op/s 2024-05-09T09:40:28.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:28 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:40:28.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:28 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:40:28.837 INFO:tasks.workunit.client.0.smithi069.stdout: 10 265478 27.63 MB/sec execute 386 sec latency 76.239 ms 2024-05-09T09:40:28.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:28 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:40:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:29 smithi069 ceph-mon[30123]: pgmap v1017: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 472 op/s 2024-05-09T09:40:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:29 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:29 smithi163 ceph-mon[34354]: pgmap v1017: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 472 op/s 2024-05-09T09:40:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:29 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.838 INFO:tasks.workunit.client.0.smithi069.stdout: 10 265817 27.63 MB/sec execute 387 sec latency 424.332 ms 2024-05-09T09:40:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:29 smithi103 ceph-mon[34421]: pgmap v1017: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 6.3 MiB/s wr, 472 op/s 2024-05-09T09:40:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:29.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:29 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:30.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 266277 27.64 MB/sec execute 388 sec latency 73.979 ms 2024-05-09T09:40:31.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:30 smithi069 ceph-mon[30123]: pgmap v1018: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.4 MiB/s wr, 458 op/s 2024-05-09T09:40:31.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.011 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:30 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:30 smithi163 ceph-mon[34354]: pgmap v1018: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.4 MiB/s wr, 458 op/s 2024-05-09T09:40:31.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:30 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:30 smithi103 ceph-mon[34421]: pgmap v1018: 129 pgs: 129 active+clean; 717 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 6.4 MiB/s wr, 458 op/s 2024-05-09T09:40:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:30 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:31.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 266698 27.64 MB/sec execute 389 sec latency 96.370 ms 2024-05-09T09:40:32.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:40:32.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:40:32.056 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:40:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:40:32.118 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:40:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:40:32.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:40:32.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 267130 27.62 MB/sec execute 390 sec latency 64.076 ms 2024-05-09T09:40:33.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:32 smithi103 ceph-mon[34421]: pgmap v1019: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:32 smithi069 ceph-mon[30123]: pgmap v1019: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:32 smithi163 ceph-mon[34354]: pgmap v1019: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:33.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 267509 27.58 MB/sec execute 391 sec latency 110.759 ms 2024-05-09T09:40:34.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 267835 27.54 MB/sec execute 392 sec latency 359.915 ms 2024-05-09T09:40:35.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:34 smithi103 ceph-mon[34421]: pgmap v1020: 129 pgs: 129 active+clean; 687 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.4 MiB/s wr, 558 op/s 2024-05-09T09:40:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:34 smithi069 ceph-mon[30123]: pgmap v1020: 129 pgs: 129 active+clean; 687 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.4 MiB/s wr, 558 op/s 2024-05-09T09:40:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:34 smithi163 ceph-mon[34354]: pgmap v1020: 129 pgs: 129 active+clean; 687 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.4 MiB/s wr, 558 op/s 2024-05-09T09:40:35.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 268196 27.54 MB/sec execute 393 sec latency 394.260 ms 2024-05-09T09:40:36.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 268847 27.55 MB/sec execute 394 sec latency 88.610 ms 2024-05-09T09:40:37.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:36 smithi103 ceph-mon[34421]: pgmap v1021: 129 pgs: 129 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.0 MiB/s wr, 645 op/s 2024-05-09T09:40:37.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:36 smithi069 ceph-mon[30123]: pgmap v1021: 129 pgs: 129 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.0 MiB/s wr, 645 op/s 2024-05-09T09:40:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:36 smithi163 ceph-mon[34354]: pgmap v1021: 129 pgs: 129 active+clean; 701 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.8 MiB/s rd, 8.0 MiB/s wr, 645 op/s 2024-05-09T09:40:37.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 269538 27.57 MB/sec execute 395 sec latency 44.720 ms 2024-05-09T09:40:38.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 270066 27.59 MB/sec execute 396 sec latency 71.888 ms 2024-05-09T09:40:39.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:38 smithi103 ceph-mon[34421]: pgmap v1022: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:39.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:38 smithi069 ceph-mon[30123]: pgmap v1022: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:38 smithi163 ceph-mon[34354]: pgmap v1022: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.7 MiB/s wr, 541 op/s 2024-05-09T09:40:39.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 270408 27.58 MB/sec execute 397 sec latency 216.513 ms 2024-05-09T09:40:40.015 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 -- 172.21.15.69:0/507074219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d01007a0 msgr2=0x7f35d0100bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.016 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 --2- 172.21.15.69:0/507074219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d01007a0 0x7f35d0100bc0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 crypto rx=0x7f35c8009ca0 tx=0x7f35c802a8d0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 -- 172.21.15.69:0/507074219 shutdown_connections 2024-05-09T09:40:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 --2- 172.21.15.69:0/507074219 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0102350 0x7f35d0108be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 --2- 172.21.15.69:0/507074219 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d0101990 0x7f35d0101e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 --2- 172.21.15.69:0/507074219 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d01007a0 0x7f35d0100bc0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.017 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.016+0000 7f35d61f9640 1 -- 172.21.15.69:0/507074219 >> 172.21.15.69:0/507074219 conn(0x7f35d00fbd10 msgr2=0x7f35d00fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:40.018 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.018+0000 7f35d61f9640 1 -- 172.21.15.69:0/507074219 shutdown_connections 2024-05-09T09:40:40.019 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.018+0000 7f35d61f9640 1 -- 172.21.15.69:0/507074219 wait complete. 2024-05-09T09:40:40.020 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.019+0000 7f35d61f9640 1 Processor -- start 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.022+0000 7f35d61f9640 1 -- start start 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.022+0000 7f35d61f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.022+0000 7f35d61f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0101990 0x7f35d0195e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.022+0000 7f35d61f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d0102350 0x7f35d01ae2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.022+0000 7f35d61f9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010b3b0 con 0x7f35d0102350 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35d61f9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010b230 con 0x7f35d01007a0 2024-05-09T09:40:40.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35d61f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35d010b530 con 0x7f35d0101990 2024-05-09T09:40:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:48866/0 (socket says 172.21.15.69:48866) 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 -- 172.21.15.69:0/4012969445 learned_addr learned my addr 172.21.15.69:0/4012969445 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35ceffd640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0101990 0x7f35d0195e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cffff640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d0102350 0x7f35d01ae2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 -- 172.21.15.69:0/4012969445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0101990 msgr2=0x7f35d0195e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0101990 0x7f35d0195e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 -- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d0102350 msgr2=0x7f35d01ae2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d0102350 0x7f35d01ae2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.023+0000 7f35cf7fe640 1 -- 172.21.15.69:0/4012969445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f35d0198fd0 con 0x7f35d01007a0 2024-05-09T09:40:40.024 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.025+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f35c802aec0 tx=0x7f35c802f850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.028+0000 7f35ccff9640 1 -- 172.21.15.69:0/4012969445 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80305d0 con 0x7f35d01007a0 2024-05-09T09:40:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.028+0000 7f35ccff9640 1 -- 172.21.15.69:0/4012969445 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c8030c50 con 0x7f35d01007a0 2024-05-09T09:40:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.028+0000 7f35ccff9640 1 -- 172.21.15.69:0/4012969445 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c8046c70 con 0x7f35d01007a0 2024-05-09T09:40:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.029+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f35d01ae810 con 0x7f35d01007a0 2024-05-09T09:40:40.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.029+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f35d01aecf0 con 0x7f35d01007a0 2024-05-09T09:40:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.033+0000 7f35d8fe7740 1 Processor -- start 2024-05-09T09:40:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.034+0000 7f35ccff9640 1 -- 172.21.15.69:0/4012969445 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f35c8030770 con 0x7f35d01007a0 2024-05-09T09:40:40.034 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.034+0000 7f35d8fe7740 1 -- start start 2024-05-09T09:40:40.041 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1bc40 0x562311e20740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.046 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e23240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562311d7f340 con 0x562311e1b250 2024-05-09T09:40:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562311d27290 con 0x562311e20db0 2024-05-09T09:40:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562311d7b300 con 0x562311e1bc40 2024-05-09T09:40:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.049 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.035+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57492/0 (socket says 172.21.15.69:57492) 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.036+0000 7f35cffff640 1 -- 172.21.15.69:0/2488048411 learned_addr learned my addr 172.21.15.69:0/2488048411 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.038+0000 7f35ccff9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 0x7f35b80a5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.042+0000 7f35ceffd640 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e23240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.050+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1bc40 0x562311e20740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.050+0000 7f35cffff640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 0x7f35b80a5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.051+0000 7f35cffff640 1 -- 172.21.15.69:0/2488048411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1bc40 msgr2=0x562311e20740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.050 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.051+0000 7f35cffff640 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1bc40 0x562311e20740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.051+0000 7f35cffff640 1 -- 172.21.15.69:0/2488048411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 msgr2=0x562311e23240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.051+0000 7f35cffff640 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e23240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.051 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.051+0000 7f35cffff640 1 -- 172.21.15.69:0/2488048411 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562311d4b8e0 con 0x562311e1b250 2024-05-09T09:40:40.053 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.053+0000 7f35ccff9640 1 -- 172.21.15.69:0/4012969445 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f35c80f0b60 con 0x7f35d01007a0 2024-05-09T09:40:40.056 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.057+0000 7f35cffff640 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009cf0 tx=0x7f35c0012640 comp rx=0 tx=0).ready entity=mon.0 client_cookie=337c45b8799a48f1 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.062+0000 7f35cffff640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 0x7f35b80a5290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f35c0019000 tx=0x7f35c001b000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.061 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.062+0000 7f35b5ffb640 1 -- 172.21.15.69:0/2488048411 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c0023070 con 0x562311e1b250 2024-05-09T09:40:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.062+0000 7f35b5ffb640 1 -- 172.21.15.69:0/2488048411 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c0015d10 con 0x562311e1b250 2024-05-09T09:40:40.062 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.062+0000 7f35b5ffb640 1 -- 172.21.15.69:0/2488048411 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c00145b0 con 0x562311e1b250 2024-05-09T09:40:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 -- 172.21.15.69:0/2488048411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 msgr2=0x562311e1b670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 crypto rx=0x7f35c0009cf0 tx=0x7f35c0012640 comp rx=0 tx=0).stop 2024-05-09T09:40:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 -- 172.21.15.69:0/2488048411 shutdown_connections 2024-05-09T09:40:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e23240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.063 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1bc40 0x562311e20740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/2488048411 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1b250 0x562311e1b670 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.064 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.064+0000 7f35d8fe7740 1 -- 172.21.15.69:0/2488048411 >> 172.21.15.69:0/2488048411 conn(0x562311e168c0 msgr2=0x562311e16cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.068+0000 7f35d8fe7740 1 -- 172.21.15.69:0/2488048411 shutdown_connections 2024-05-09T09:40:40.068 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.069+0000 7f35d8fe7740 1 -- 172.21.15.69:0/2488048411 wait complete. 2024-05-09T09:40:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.069+0000 7f35d8fe7740 1 -- start start 2024-05-09T09:40:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.069+0000 7f35d8fe7740 1 Processor -- start 2024-05-09T09:40:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e25580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e25ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e2ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562311e23d60 con 0x562311e20db0 2024-05-09T09:40:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562311d3a790 con 0x562311e1bc40 2024-05-09T09:40:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562311e23ee0 con 0x562311e1b250 2024-05-09T09:40:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.070+0000 7f35d8fe7740 10 client.0 _ll_register_callbacks cb 0x562311d75a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e2ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e2ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57508/0 (socket says 172.21.15.69:57508) 2024-05-09T09:40:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cffff640 1 -- 172.21.15.69:0/1925037896 learned_addr learned my addr 172.21.15.69:0/1925037896 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e25580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 -- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 msgr2=0x562311e25ac0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e25ac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 -- 172.21.15.69:0/1925037896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 msgr2=0x562311e2ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e2ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.073+0000 7f35cf7fe640 1 -- 172.21.15.69:0/1925037896 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562311a93210 con 0x562311e1b250 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.074+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e25580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b99b0 tx=0x7f35c80f4190 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e39e4b754c626a6f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.074+0000 7f35aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:40:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.074+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80bc940 con 0x562311e1b250 2024-05-09T09:40:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.074+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c0015d10 con 0x562311e1b250 2024-05-09T09:40:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.074+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f35c80b7b50 con 0x562311e1b250 2024-05-09T09:40:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.076+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f35c80098d0 con 0x562311e1b250 2024-05-09T09:40:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.078+0000 7f35d8fe7740 10 client.26920 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.078+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562311b5f620 con 0x562311e1b250 2024-05-09T09:40:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.080+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c0015d10 con 0x562311e1b250 2024-05-09T09:40:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.081+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f35c804e030 con 0x562311e1b250 2024-05-09T09:40:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 10 client.26920 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 10 client.26920 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 20 client.26920 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:40.081 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311e31020 0x562311e33430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 4 client.26920 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:40:40.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.082+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562311d8ca70 con 0x562311e31020 2024-05-09T09:40:40.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.085+0000 7f35ceffd640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311e31020 0x562311e33430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.089+0000 7f35ceffd640 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311e31020 0x562311e33430 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=188262fcce586eed server_cookie=dd8c843e724da94f in_seq=0 out_seq=0 2024-05-09T09:40:40.089 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.090+0000 7f35aeffd640 10 client.26920 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:40.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.093+0000 7f35aeffd640 1 -- 172.21.15.69:0/1925037896 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562311d8ca70 con 0x562311e31020 2024-05-09T09:40:40.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.093+0000 7f35aeffd640 10 client.26920 handle_command_reply: tid=0 2024-05-09T09:40:40.093 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.093+0000 7f35d8fe7740 1 client.26920 shutdown 2024-05-09T09:40:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.099+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 msgr2=0x562311e25580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.099+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e25580 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f35c80b99b0 tx=0x7f35c80f4190 comp rx=0 tx=0).stop 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 shutdown_connections 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e2ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e25ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311e31020 0x562311e33430 crc :-1 s=READY pgs=252 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1925037896 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e25580 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 >> 172.21.15.69:0/1925037896 conn(0x562311e168c0 msgr2=0x562311e15a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.101+0000 7f35aeffd640 0 client.26920 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:40.101 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.102+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 shutdown_connections 2024-05-09T09:40:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.102+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1925037896 wait complete. 2024-05-09T09:40:40.102 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.102+0000 7f35d8fe7740 20 client.26920 trim_cache size 0 max 16384 2024-05-09T09:40:40.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 Processor -- start 2024-05-09T09:40:40.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 -- start start 2024-05-09T09:40:40.129 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e1a100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e2ca20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e258a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01bb980 con 0x562311e20db0 2024-05-09T09:40:40.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01b9250 con 0x562311e1bc40 2024-05-09T09:40:40.130 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.130+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01b7150 con 0x562311e1b250 2024-05-09T09:40:40.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.131+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e258a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.131+0000 7f35cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e258a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57518/0 (socket says 172.21.15.69:57518) 2024-05-09T09:40:40.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.131+0000 7f35cffff640 1 -- 172.21.15.69:0/715263070 learned_addr learned my addr 172.21.15.69:0/715263070 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:40.132 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.132+0000 7f35ceffd640 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e2ca20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.134+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e1a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.136+0000 7f35cf7fe640 1 -- 172.21.15.69:0/715263070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 msgr2=0x562311e2ca20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.136+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e2ca20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.136+0000 7f35cf7fe640 1 -- 172.21.15.69:0/715263070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 msgr2=0x562311e258a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.136+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e258a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.136 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.136+0000 7f35cf7fe640 1 -- 172.21.15.69:0/715263070 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562311e1ae90 con 0x562311e1b250 2024-05-09T09:40:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.137+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e1a100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35c80bcce0 tx=0x7f35c80bb4f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1085727c1028de71 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.138+0000 7f35b57fa640 1 -- 172.21.15.69:0/715263070 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c80bc940 con 0x562311e1b250 2024-05-09T09:40:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.138+0000 7f35b57fa640 1 -- 172.21.15.69:0/715263070 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c8036050 con 0x562311e1b250 2024-05-09T09:40:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.138+0000 7f35d8fe7740 1 -- 172.21.15.69:0/715263070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 msgr2=0x562311e1a100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:40.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.139+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e1a100 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f35c80bcce0 tx=0x7f35c80bb4f0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.141+0000 7f35d8fe7740 1 -- 172.21.15.69:0/715263070 shutdown_connections 2024-05-09T09:40:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.141+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e20db0 0x562311e258a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.141+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e1bc40 0x562311e2ca20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.141+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/715263070 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e1a100 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.141+0000 7f35d8fe7740 1 -- 172.21.15.69:0/715263070 >> 172.21.15.69:0/715263070 conn(0x562311e168c0 msgr2=0x562311e25190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.143+0000 7f35d8fe7740 1 -- 172.21.15.69:0/715263070 shutdown_connections 2024-05-09T09:40:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.144+0000 7f35d8fe7740 1 -- 172.21.15.69:0/715263070 wait complete. 2024-05-09T09:40:40.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.144+0000 7f35d8fe7740 1 -- start start 2024-05-09T09:40:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 Processor -- start 2024-05-09T09:40:40.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e2a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e2b710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01b7710 con 0x562311e1bc40 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01b8b80 con 0x562311e20db0 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f35d01b7de0 con 0x562311e1b250 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35d8fe7740 10 client.0 _ll_register_callbacks cb 0x7f35c80f4290 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:40.145 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.145+0000 7f35ceffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57522/0 (socket says 172.21.15.69:57522) 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 -- 172.21.15.69:0/1634906229 learned_addr learned my addr 172.21.15.69:0/1634906229 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e2a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35cffff640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e2b710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 -- 172.21.15.69:0/1634906229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 msgr2=0x562311e2a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e2a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 -- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 msgr2=0x562311e2b710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e2b710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35ceffd640 1 -- 172.21.15.69:0/1634906229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562311e2bd00 con 0x562311e1bc40 2024-05-09T09:40:40.146 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.146+0000 7f35cffff640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e2b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.150+0000 7f35aeffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.150+0000 7f35ceffd640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x562311e17830 tx=0x7f35c40108c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ad4016b0e025e111 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.150+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f35c401d890 con 0x562311e1bc40 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.150+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f35c401d690 con 0x562311e1bc40 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.151+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f35c8036050 con 0x562311e1bc40 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.151+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f35c401d890 con 0x562311e1bc40 2024-05-09T09:40:40.151 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.152+0000 7f35d8fe7740 10 client.17745 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:40.152 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.152+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562311ea8750 con 0x562311e1bc40 2024-05-09T09:40:40.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.153+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f35c8036050 con 0x562311e1bc40 2024-05-09T09:40:40.154 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.153+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f35c400f420 con 0x562311e1bc40 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 10 client.17745 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 10 client.17745 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 20 client.17745 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311ea9130 0x562311eab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 4 client.17745 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:40:40.155 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.154+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562311d86dc0 con 0x562311ea9130 2024-05-09T09:40:40.156 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.157+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311ea9130 0x562311eab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.166+0000 7f35cf7fe640 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311ea9130 0x562311eab550 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3d84fc7d55a5f72e server_cookie=78a1447964a9bb9a in_seq=0 out_seq=0 2024-05-09T09:40:40.166 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:40.166+0000 7f35aeffd640 10 client.17745 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:40.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 270538 27.53 MB/sec execute 398 sec latency 554.525 ms 2024-05-09T09:40:41.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.106+0000 7f35aeffd640 1 -- 172.21.15.69:0/1634906229 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562311d86dc0 con 0x562311ea9130 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.106+0000 7f35aeffd640 10 client.17745 handle_command_reply: tid=0 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.106+0000 7f35d8fe7740 1 client.17745 shutdown 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 msgr2=0x562311e2a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 secure :-1 s=READY pgs=1080 cs=0 l=1 rev1=1 crypto rx=0x562311e17830 tx=0x7f35c40108c0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 shutdown_connections 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562311e20db0 0x562311e2b710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562311e1bc40 0x562311e2a700 unknown :-1 s=CLOSED pgs=1080 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562311ea9130 0x562311eab550 crc :-1 s=READY pgs=253 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 --2- 172.21.15.69:0/1634906229 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562311e1b250 0x562311e2a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 >> 172.21.15.69:0/1634906229 conn(0x562311e168c0 msgr2=0x562311e19f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35aeffd640 0 client.17745 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 shutdown_connections 2024-05-09T09:40:41.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 1 -- 172.21.15.69:0/1634906229 wait complete. 2024-05-09T09:40:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.107+0000 7f35d8fe7740 20 client.17745 trim_cache size 0 max 16384 2024-05-09T09:40:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:40:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:40:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:40:41.108 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.108+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 msgr2=0x7f35b80a5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.108+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 0x7f35b80a5290 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f35c0019000 tx=0x7f35c001b000 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.108+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 msgr2=0x7f35d0195910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.108+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f35c802aec0 tx=0x7f35c802f850 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 shutdown_connections 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f35b80a2dd0 0x7f35b80a5290 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f35d0102350 0x7f35d01ae2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f35d0101990 0x7f35d0195e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 --2- 172.21.15.69:0/4012969445 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f35d01007a0 0x7f35d0195910 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 >> 172.21.15.69:0/4012969445 conn(0x7f35d00fbd10 msgr2=0x7f35d0104bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:41.110 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 shutdown_connections 2024-05-09T09:40:41.111 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:41.109+0000 7f35d61f9640 1 -- 172.21.15.69:0/4012969445 wait complete. 2024-05-09T09:40:41.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:40 smithi103 ceph-mon[34421]: pgmap v1023: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 532 op/s 2024-05-09T09:40:41.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:40 smithi069 ceph-mon[30123]: pgmap v1023: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 532 op/s 2024-05-09T09:40:41.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:40 smithi163 ceph-mon[34354]: pgmap v1023: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 532 op/s 2024-05-09T09:40:41.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 270901 27.50 MB/sec execute 399 sec latency 585.902 ms 2024-05-09T09:40:42.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 271370 27.48 MB/sec execute 400 sec latency 66.307 ms 2024-05-09T09:40:43.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:42 smithi103 ceph-mon[34421]: pgmap v1024: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:42 smithi069 ceph-mon[30123]: pgmap v1024: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:43.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:42 smithi163 ceph-mon[34354]: pgmap v1024: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 7.2 MiB/s wr, 618 op/s 2024-05-09T09:40:43.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 271882 27.48 MB/sec execute 401 sec latency 111.660 ms 2024-05-09T09:40:44.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 272479 27.50 MB/sec execute 402 sec latency 139.603 ms 2024-05-09T09:40:45.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:44 smithi103 ceph-mon[34421]: pgmap v1025: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.1 MiB/s wr, 460 op/s 2024-05-09T09:40:45.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:44 smithi069 ceph-mon[30123]: pgmap v1025: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.1 MiB/s wr, 460 op/s 2024-05-09T09:40:45.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:44 smithi163 ceph-mon[34354]: pgmap v1025: 129 pgs: 129 active+clean; 705 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.1 MiB/s wr, 460 op/s 2024-05-09T09:40:45.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 272918 27.48 MB/sec execute 403 sec latency 312.777 ms 2024-05-09T09:40:46.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 273346 27.46 MB/sec execute 404 sec latency 109.605 ms 2024-05-09T09:40:47.106 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:46 smithi163 ceph-mon[34354]: pgmap v1026: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 674 op/s 2024-05-09T09:40:47.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:46 smithi103 ceph-mon[34421]: pgmap v1026: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 674 op/s 2024-05-09T09:40:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:46 smithi069 ceph-mon[30123]: pgmap v1026: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 674 op/s 2024-05-09T09:40:47.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 274063 27.51 MB/sec execute 405 sec latency 48.550 ms 2024-05-09T09:40:48.839 INFO:tasks.workunit.client.0.smithi069.stdout: 10 274642 27.52 MB/sec execute 406 sec latency 60.401 ms 2024-05-09T09:40:49.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:48 smithi103 ceph-mon[34421]: pgmap v1027: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 538 op/s 2024-05-09T09:40:49.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:48 smithi069 ceph-mon[30123]: pgmap v1027: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 538 op/s 2024-05-09T09:40:49.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:48 smithi163 ceph-mon[34354]: pgmap v1027: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.2 MiB/s wr, 538 op/s 2024-05-09T09:40:49.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 275093 27.52 MB/sec execute 407 sec latency 145.756 ms 2024-05-09T09:40:50.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 275318 27.48 MB/sec execute 408 sec latency 467.566 ms 2024-05-09T09:40:51.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:50 smithi103 ceph-mon[34421]: pgmap v1028: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 528 op/s 2024-05-09T09:40:51.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:50 smithi069 ceph-mon[30123]: pgmap v1028: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 528 op/s 2024-05-09T09:40:51.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:50 smithi163 ceph-mon[34354]: pgmap v1028: 129 pgs: 129 active+clean; 703 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 5.1 MiB/s wr, 528 op/s 2024-05-09T09:40:51.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 275550 27.44 MB/sec execute 409 sec latency 93.028 ms 2024-05-09T09:40:52.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 276207 27.44 MB/sec execute 410 sec latency 43.853 ms 2024-05-09T09:40:53.117 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:52 smithi103 ceph-mon[34421]: pgmap v1029: 129 pgs: 129 active+clean; 711 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 665 op/s 2024-05-09T09:40:53.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:52 smithi069 ceph-mon[30123]: pgmap v1029: 129 pgs: 129 active+clean; 711 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 665 op/s 2024-05-09T09:40:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:52 smithi163 ceph-mon[34354]: pgmap v1029: 129 pgs: 129 active+clean; 711 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.3 MiB/s wr, 665 op/s 2024-05-09T09:40:53.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 276756 27.44 MB/sec execute 411 sec latency 66.734 ms 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 -- 172.21.15.69:0/1548300910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 msgr2=0x7fd2d8102f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/1548300910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8102f30 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0009900 tx=0x7fd2c002a410 comp rx=0 tx=0).stop 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 -- 172.21.15.69:0/1548300910 shutdown_connections 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/1548300910 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8109d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/1548300910 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8102f30 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/1548300910 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d8101ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 -- 172.21.15.69:0/1548300910 >> 172.21.15.69:0/1548300910 conn(0x7fd2d80fce50 msgr2=0x7fd2d80ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:54.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 -- 172.21.15.69:0/1548300910 shutdown_connections 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.241+0000 7fd2dda5d640 1 -- 172.21.15.69:0/1548300910 wait complete. 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 Processor -- start 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 -- start start 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d80755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810d540 con 0x7fd2d8102ab0 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810d3c0 con 0x7fd2d8103470 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2dda5d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d810d6c0 con 0x7fd2d81018c0 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2d6ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d80755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2d6ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d80755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51140/0 (socket says 172.21.15.69:51140) 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2d6ffd640 1 -- 172.21.15.69:0/377474538 learned_addr learned my addr 172.21.15.69:0/377474538 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.242+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 -- 172.21.15.69:0/377474538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 msgr2=0x7fd2d80755e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d80755e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 -- 172.21.15.69:0/377474538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 msgr2=0x7fd2d8074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 -- 172.21.15.69:0/377474538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd2d80749f0 con 0x7fd2d8102ab0 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2cffff640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8073c20 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0031440 tx=0x7fd2c0031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2dca5b640 1 -- 172.21.15.69:0/377474538 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c003f070 con 0x7fd2d8102ab0 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2dca5b640 1 -- 172.21.15.69:0/377474538 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2c0003cd0 con 0x7fd2d8102ab0 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd2d81a1be0 con 0x7fd2d8102ab0 2024-05-09T09:40:54.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd2d81a2130 con 0x7fd2d8102ab0 2024-05-09T09:40:54.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.243+0000 7fd2dca5b640 1 -- 172.21.15.69:0/377474538 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0035070 con 0x7fd2d8102ab0 2024-05-09T09:40:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.245+0000 7fd2e076b740 1 Processor -- start 2024-05-09T09:40:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.245+0000 7fd2dca5b640 1 -- 172.21.15.69:0/377474538 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd2c003b560 con 0x7fd2d8102ab0 2024-05-09T09:40:54.256 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.245+0000 7fd2e076b740 1 -- start start 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562c37100340 con 0x562c371a1db0 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c370a8290 con 0x562c3719cc40 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2e076b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c370fc300 con 0x562c3719c250 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51152/0 (socket says 172.21.15.69:51152) 2024-05-09T09:40:54.257 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33620/0 (socket says 172.21.15.69:33620) 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2373738733 learned_addr learned my addr 172.21.15.69:0/2373738733 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2dca5b640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 0x7fd2c40a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2373738733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 msgr2=0x562c371a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2373738733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 msgr2=0x562c371a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2373738733 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c370cc8e0 con 0x562c3719c250 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2dca5b640 1 -- 172.21.15.69:0/377474538 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd2c00ef7d0 con 0x7fd2d8102ab0 2024-05-09T09:40:54.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.246+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 0x7fd2c40a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0010540 tx=0x7fd2d0010a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3d19420696176cff server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2cffff640 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 0x7fd2c40a50f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0021000 tx=0x7fd2d001e000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2cdffb640 1 -- 172.21.15.69:0/2373738733 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d00134c0 con 0x562c3719c250 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2cdffb640 1 -- 172.21.15.69:0/2373738733 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d001d8e0 con 0x562c3719c250 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.247+0000 7fd2cdffb640 1 -- 172.21.15.69:0/2373738733 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0011800 con 0x562c3719c250 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- 172.21.15.69:0/2373738733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 msgr2=0x562c3719c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.259 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0010540 tx=0x7fd2d0010a00 comp rx=0 tx=0).stop 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- 172.21.15.69:0/2373738733 shutdown_connections 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2373738733 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c3719c670 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- 172.21.15.69:0/2373738733 >> 172.21.15.69:0/2373738733 conn(0x562c371978c0 msgr2=0x562c37197cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- 172.21.15.69:0/2373738733 shutdown_connections 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- 172.21.15.69:0/2373738733 wait complete. 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- start start 2024-05-09T09:40:54.260 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 Processor -- start 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371a6540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x562c371a4db0 con 0x562c371a1db0 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x562c370bb790 con 0x562c3719cc40 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.248+0000 7fd2e076b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x562c371a4f30 con 0x562c3719c250 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2e076b740 10 client.0 _ll_register_callbacks cb 0x562c370f6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33626/0 (socket says 172.21.15.69:33626) 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 -- 172.21.15.69:0/3482494827 learned_addr learned my addr 172.21.15.69:0/3482494827 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:54.261 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2cffff640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371a6540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 -- 172.21.15.69:0/3482494827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 msgr2=0x562c371a6540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371a6540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 -- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 msgr2=0x562c371a6a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a6a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 -- 172.21.15.69:0/3482494827 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c36e14210 con 0x562c371a1db0 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2cffff640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00098e0 tx=0x7fd2d0005a10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d04183f9b9938ec server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.262 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d001dd60 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd2d001d400 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2d001d8e0 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd2d003d050 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2e076b740 10 client.17757 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.249+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562c36ee0620 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d001d8e0 con 0x562c371a1db0 2024-05-09T09:40:54.263 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fd2d002e070 con 0x562c371a1db0 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 10 client.17757 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 10 client.17757 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 20 client.17757 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c371b1f40 0x562c371b4330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 4 client.17757 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x562c3710da70 con 0x562c371b1f40 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c371b1f40 0x562c371b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2b6ffd640 10 client.17757 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.250+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c371b1f40 0x562c371b4330 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=66ad1c5a438df194 server_cookie=ea8069e77f53d618 in_seq=0 out_seq=0 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.254+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/3482494827 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x562c3710da70 con 0x562c371b1f40 2024-05-09T09:40:54.264 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.254+0000 7fd2b6ffd640 10 client.17757 handle_command_reply: tid=0 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.254+0000 7fd2e076b740 1 client.17757 shutdown 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.254+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 msgr2=0x562c371adac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 crypto rx=0x7fd2d00098e0 tx=0x7fd2d0005a10 comp rx=0 tx=0).stop 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 shutdown_connections 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371adac0 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719cc40 0x562c371a6a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c371b1f40 0x562c371b4330 crc :-1 s=READY pgs=254 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 --2- 172.21.15.69:0/3482494827 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371a6540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2b6ffd640 0 client.17757 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 >> 172.21.15.69:0/3482494827 conn(0x562c371978c0 msgr2=0x562c37196a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 shutdown_connections 2024-05-09T09:40:54.265 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 1 -- 172.21.15.69:0/3482494827 wait complete. 2024-05-09T09:40:54.266 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.255+0000 7fd2e076b740 20 client.17757 trim_cache size 0 max 16384 2024-05-09T09:40:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 Processor -- start 2024-05-09T09:40:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 -- start start 2024-05-09T09:40:54.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 0x562c3719ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719cc40 0x562c3719b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81aee90 con 0x562c371a1db0 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81ac760 con 0x562c3719c250 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2e076b740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81aa5b0 con 0x562c3719cc40 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.274+0000 7fd2d77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33638/0 (socket says 172.21.15.69:33638) 2024-05-09T09:40:54.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2618635093 learned_addr learned my addr 172.21.15.69:0/2618635093 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 0x562c3719ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2cffff640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719cc40 0x562c3719b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2618635093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719cc40 msgr2=0x562c3719b290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719cc40 0x562c3719b290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2618635093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 msgr2=0x562c3719ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 0x562c3719ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 -- 172.21.15.69:0/2618635093 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x562c371b49b0 con 0x562c371a1db0 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 0x562c3719ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009f10 tx=0x7fd2d0010a00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c945e35e719073c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2cd7fa640 1 -- 172.21.15.69:0/2618635093 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0046070 con 0x562c371a1db0 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2cd7fa640 1 -- 172.21.15.69:0/2618635093 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0034070 con 0x562c371a1db0 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.275+0000 7fd2cd7fa640 1 -- 172.21.15.69:0/2618635093 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2d0011160 con 0x562c371a1db0 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- 172.21.15.69:0/2618635093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 msgr2=0x562c371ada80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0009f10 tx=0x7fd2d0010a00 comp rx=0 tx=0).stop 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- 172.21.15.69:0/2618635093 shutdown_connections 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c371a1db0 0x562c371ada80 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719cc40 0x562c3719b290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 --2- 172.21.15.69:0/2618635093 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c3719c250 0x562c3719ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- 172.21.15.69:0/2618635093 >> 172.21.15.69:0/2618635093 conn(0x562c371978c0 msgr2=0x562c3719a640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- 172.21.15.69:0/2618635093 shutdown_connections 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- 172.21.15.69:0/2618635093 wait complete. 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.276+0000 7fd2e076b740 1 -- start start 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 Processor -- start 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371aadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c371a1db0 0x562c371ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33650/0 (socket says 172.21.15.69:33650) 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 -- 172.21.15.69:0/1387865327 learned_addr learned my addr 172.21.15.69:0/1387865327 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371aadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81aac20 con 0x562c3719cc40 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81ac090 con 0x562c371a1db0 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd2d81ab2f0 con 0x562c3719c250 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2e076b740 10 client.0 _ll_register_callbacks cb 0x7fd2d0020020 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:40:54.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2d77fe640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c371a1db0 0x562c371ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 -- 172.21.15.69:0/1387865327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 msgr2=0x562c371aadf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371aadf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 -- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c371a1db0 msgr2=0x562c371ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c371a1db0 0x562c371ac2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.277+0000 7fd2cffff640 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x562c371ac8e0 con 0x562c3719cc40 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2cffff640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006010 tx=0x7fd2c00b8060 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56f5f56d7fb159f3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:40:54.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd2c0048040 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd2c00c4690 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd2d0034070 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd2c00bb610 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 10 client.17763 fetch_fsmap learned FSMap version 47 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x562c371adca0 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd2d0034070 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fd2c004b060 con 0x562c3719cc40 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 10 client.17763 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:40:54.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 10 client.17763 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 20 client.17763 populate_metadata read hostname 'smithi069' 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c37218be0 0x562c3721b000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 4 client.17763 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x562c37107dc0 con 0x562c37218be0 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.278+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c37218be0 0x562c3721b000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.279+0000 7fd2b6ffd640 10 client.17763 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:54.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:54.279+0000 7fd2d6ffd640 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c37218be0 0x562c3721b000 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ec10494a1239d1ab server_cookie=57a0d596a1cb6322 in_seq=0 out_seq=0 2024-05-09T09:40:54.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 277040 27.41 MB/sec execute 412 sec latency 468.720 ms 2024-05-09T09:40:55.261 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:54 smithi069 ceph-mon[30123]: pgmap v1030: 129 pgs: 129 active+clean; 715 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 576 op/s 2024-05-09T09:40:55.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:54 smithi163 ceph-mon[34354]: pgmap v1030: 129 pgs: 129 active+clean; 715 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 576 op/s 2024-05-09T09:40:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:54 smithi103 ceph-mon[34421]: pgmap v1030: 129 pgs: 129 active+clean; 715 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 576 op/s 2024-05-09T09:40:55.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.410+0000 7fd2b6ffd640 1 -- 172.21.15.69:0/1387865327 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x562c37107dc0 con 0x562c37218be0 2024-05-09T09:40:55.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.410+0000 7fd2b6ffd640 10 client.17763 handle_command_reply: tid=0 2024-05-09T09:40:55.412 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.410+0000 7fd2e076b740 1 client.17763 shutdown 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.410+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 msgr2=0x562c371ab330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.410+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0006010 tx=0x7fd2c00b8060 comp rx=0 tx=0).stop 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 shutdown_connections 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x562c371a1db0 0x562c371ac2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x562c3719cc40 0x562c371ab330 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x562c37218be0 0x562c3721b000 crc :-1 s=READY pgs=255 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2b6ffd640 0 client.17763 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 --2- 172.21.15.69:0/1387865327 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x562c3719c250 0x562c371aadf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 >> 172.21.15.69:0/1387865327 conn(0x562c371978c0 msgr2=0x562c3719aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 shutdown_connections 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 1 -- 172.21.15.69:0/1387865327 wait complete. 2024-05-09T09:40:55.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.411+0000 7fd2e076b740 20 client.17763 trim_cache size 0 max 16384 2024-05-09T09:40:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:40:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:40:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:40:55.414 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 msgr2=0x7fd2c40a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 0x7fd2c40a50f0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0021000 tx=0x7fd2d001e000 comp rx=0 tx=0).stop 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 msgr2=0x7fd2d8073c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8073c20 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 crypto rx=0x7fd2c0031440 tx=0x7fd2c0031b60 comp rx=0 tx=0).stop 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 shutdown_connections 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd2c40a2c30 0x7fd2c40a50f0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.415 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd2d8103470 0x7fd2d8074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd2d8102ab0 0x7fd2d8073c20 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 --2- 172.21.15.69:0/377474538 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd2d81018c0 0x7fd2d80755e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 >> 172.21.15.69:0/377474538 conn(0x7fd2d80fce50 msgr2=0x7fd2d8105db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 shutdown_connections 2024-05-09T09:40:55.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:40:55.412+0000 7fd2dda5d640 1 -- 172.21.15.69:0/377474538 wait complete. 2024-05-09T09:40:55.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 277338 27.38 MB/sec execute 413 sec latency 968.487 ms 2024-05-09T09:40:56.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 277704 27.38 MB/sec execute 414 sec latency 184.045 ms 2024-05-09T09:40:57.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:56 smithi069 ceph-mon[30123]: pgmap v1031: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.5 MiB/s wr, 760 op/s 2024-05-09T09:40:57.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:56 smithi163 ceph-mon[34354]: pgmap v1031: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.5 MiB/s wr, 760 op/s 2024-05-09T09:40:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:56 smithi103 ceph-mon[34421]: pgmap v1031: 129 pgs: 129 active+clean; 718 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.5 MiB/s wr, 760 op/s 2024-05-09T09:40:57.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 278328 27.40 MB/sec execute 415 sec latency 39.527 ms 2024-05-09T09:40:58.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 278802 27.41 MB/sec execute 416 sec latency 91.373 ms 2024-05-09T09:40:59.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:40:58 smithi069 ceph-mon[30123]: pgmap v1032: 129 pgs: 129 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 525 op/s 2024-05-09T09:40:59.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:40:58 smithi163 ceph-mon[34354]: pgmap v1032: 129 pgs: 129 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 525 op/s 2024-05-09T09:40:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:40:58 smithi103 ceph-mon[34421]: pgmap v1032: 129 pgs: 129 active+clean; 714 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.8 MiB/s wr, 525 op/s 2024-05-09T09:40:59.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 279161 27.39 MB/sec execute 417 sec latency 99.630 ms 2024-05-09T09:41:00.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 279344 27.34 MB/sec execute 418 sec latency 485.188 ms 2024-05-09T09:41:01.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:00 smithi069 ceph-mon[30123]: pgmap v1033: 129 pgs: 129 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 543 op/s 2024-05-09T09:41:01.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:00 smithi163 ceph-mon[34354]: pgmap v1033: 129 pgs: 129 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 543 op/s 2024-05-09T09:41:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:00 smithi103 ceph-mon[34421]: pgmap v1033: 129 pgs: 129 active+clean; 710 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 6.0 MiB/s wr, 543 op/s 2024-05-09T09:41:01.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 279627 27.30 MB/sec execute 419 sec latency 139.550 ms 2024-05-09T09:41:02.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 280291 27.31 MB/sec execute 420 sec latency 66.497 ms 2024-05-09T09:41:03.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:02 smithi069 ceph-mon[30123]: pgmap v1034: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-05-09T09:41:03.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:02 smithi163 ceph-mon[34354]: pgmap v1034: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-05-09T09:41:03.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:02 smithi103 ceph-mon[34421]: pgmap v1034: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 8.6 MiB/s wr, 700 op/s 2024-05-09T09:41:03.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 281081 27.34 MB/sec execute 421 sec latency 57.809 ms 2024-05-09T09:41:04.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 281751 27.39 MB/sec execute 422 sec latency 94.974 ms 2024-05-09T09:41:05.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:04 smithi069 ceph-mon[30123]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.7 MiB/s wr, 583 op/s 2024-05-09T09:41:05.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:04 smithi163 ceph-mon[34354]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.7 MiB/s wr, 583 op/s 2024-05-09T09:41:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:04 smithi103 ceph-mon[34421]: pgmap v1035: 129 pgs: 129 active+clean; 696 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 6.7 MiB/s wr, 583 op/s 2024-05-09T09:41:05.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 282077 27.37 MB/sec execute 423 sec latency 344.745 ms 2024-05-09T09:41:06.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 282441 27.36 MB/sec execute 424 sec latency 71.832 ms 2024-05-09T09:41:07.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:06 smithi069 ceph-mon[30123]: pgmap v1036: 129 pgs: 129 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 661 op/s 2024-05-09T09:41:07.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:06 smithi163 ceph-mon[34354]: pgmap v1036: 129 pgs: 129 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 661 op/s 2024-05-09T09:41:07.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:06 smithi103 ceph-mon[34421]: pgmap v1036: 129 pgs: 129 active+clean; 699 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 7.4 MiB/s wr, 661 op/s 2024-05-09T09:41:07.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 282823 27.33 MB/sec execute 425 sec latency 109.793 ms 2024-05-09T09:41:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 -- 172.21.15.69:0/348902925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c105680 msgr2=0x7fa31c10bf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 --2- 172.21.15.69:0/348902925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c105680 0x7fa31c10bf10 secure :-1 s=READY pgs=1086 cs=0 l=1 rev1=1 crypto rx=0x7fa314009be0 tx=0x7fa31402a6c0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 -- 172.21.15.69:0/348902925 shutdown_connections 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 --2- 172.21.15.69:0/348902925 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c105680 0x7fa31c10bf10 unknown :-1 s=CLOSED pgs=1086 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 --2- 172.21.15.69:0/348902925 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c104cc0 0x7fa31c105140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 --2- 172.21.15.69:0/348902925 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 0x7fa31c103ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 -- 172.21.15.69:0/348902925 >> 172.21.15.69:0/348902925 conn(0x7fa31c0ff020 msgr2=0x7fa31c101480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 -- 172.21.15.69:0/348902925 shutdown_connections 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 -- 172.21.15.69:0/348902925 wait complete. 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.524+0000 7fa321763640 1 Processor -- start 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 -- start start 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 0x7fa31c199cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c105680 0x7fa31c1a1250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f730 con 0x7fa31c104cc0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f5b0 con 0x7fa31c103ad0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa321763640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c10f8b0 con 0x7fa31c105680 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35868/0 (socket says 172.21.15.69:35868) 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 -- 172.21.15.69:0/604275934 learned_addr learned my addr 172.21.15.69:0/604275934 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31affd640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 0x7fa31c199cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c105680 0x7fa31c1a1250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 -- 172.21.15.69:0/604275934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c105680 msgr2=0x7fa31c1a1250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c105680 0x7fa31c1a1250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 -- 172.21.15.69:0/604275934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 msgr2=0x7fa31c199cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 0x7fa31c199cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 -- 172.21.15.69:0/604275934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa31c1a18f0 con 0x7fa31c104cc0 2024-05-09T09:41:08.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.525+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa31000e9f0 tx=0x7fa31000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.526+0000 7fa303fff640 1 -- 172.21.15.69:0/604275934 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31000cd80 con 0x7fa31c104cc0 2024-05-09T09:41:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.526+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa31c1a1be0 con 0x7fa31c104cc0 2024-05-09T09:41:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.526+0000 7fa303fff640 1 -- 172.21.15.69:0/604275934 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa310010910 con 0x7fa31c104cc0 2024-05-09T09:41:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.526+0000 7fa303fff640 1 -- 172.21.15.69:0/604275934 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa3100188f0 con 0x7fa31c104cc0 2024-05-09T09:41:08.529 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.526+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa31c1a2130 con 0x7fa31c104cc0 2024-05-09T09:41:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 Processor -- start 2024-05-09T09:41:08.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa303fff640 1 -- 172.21.15.69:0/604275934 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa310018b70 con 0x7fa31c104cc0 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 -- start start 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a94740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 0x5640d1a97240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640d19f3340 con 0x5640d1a94db0 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5640d199b290 con 0x5640d1a8fc40 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa324476740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5640d19ef300 con 0x5640d1a8f250 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.529+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 0x5640d1a97240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53724/0 (socket says 172.21.15.69:53724) 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 -- 172.21.15.69:0/3465998905 learned_addr learned my addr 172.21.15.69:0/3465998905 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 0x5640d1a97240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35870/0 (socket says 172.21.15.69:35870) 2024-05-09T09:41:08.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31affd640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a94740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa303fff640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 0x7fa3040a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 -- 172.21.15.69:0/3465998905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 msgr2=0x5640d1a94740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a94740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 -- 172.21.15.69:0/3465998905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 msgr2=0x5640d1a97240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 0x5640d1a97240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 -- 172.21.15.69:0/3465998905 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d19bf8e0 con 0x5640d1a8f250 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31affd640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a94740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa303fff640 1 -- 172.21.15.69:0/604275934 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa3100cf740 con 0x7fa31c104cc0 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 0x7fa3040a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.530+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa314009be0 tx=0x7fa314005eb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec479b7c71476bfd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.531+0000 7fa3017fa640 1 -- 172.21.15.69:0/3465998905 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314041040 con 0x5640d1a8f250 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.531+0000 7fa3017fa640 1 -- 172.21.15.69:0/3465998905 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa314030db0 con 0x5640d1a8f250 2024-05-09T09:41:08.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.531+0000 7fa3017fa640 1 -- 172.21.15.69:0/3465998905 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314036040 con 0x5640d1a8f250 2024-05-09T09:41:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 -- 172.21.15.69:0/3465998905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 msgr2=0x5640d1a8f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa314009be0 tx=0x7fa314005eb0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 0x7fa3040a50f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa314049000 tx=0x7fa31404c000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 -- 172.21.15.69:0/3465998905 shutdown_connections 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a94db0 0x5640d1a97240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a94740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 --2- 172.21.15.69:0/3465998905 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a8f670 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 -- 172.21.15.69:0/3465998905 >> 172.21.15.69:0/3465998905 conn(0x5640d1a8a8c0 msgr2=0x5640d1a8acc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 -- 172.21.15.69:0/3465998905 shutdown_connections 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.532+0000 7fa324476740 1 -- 172.21.15.69:0/3465998905 wait complete. 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 -- start start 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 Processor -- start 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a99a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 0x5640d1aa0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5640d1a97db0 con 0x5640d1a8f250 2024-05-09T09:41:08.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5640d19ae790 con 0x5640d1a8fc40 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5640d1a97f30 con 0x5640d1a94db0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa324476740 10 client.0 _ll_register_callbacks cb 0x5640d19e9a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa31affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa31b7fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 0x5640d1aa0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa31affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35874/0 (socket says 172.21.15.69:35874) 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa31affd640 1 -- 172.21.15.69:0/2356319358 learned_addr learned my addr 172.21.15.69:0/2356319358 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.533+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a99a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 -- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 msgr2=0x5640d1a99a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a99a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 -- 172.21.15.69:0/2356319358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 msgr2=0x5640d1a99510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 -- 172.21.15.69:0/2356319358 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d1707210 con 0x5640d1a94db0 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31affd640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:08.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 0x5640d1aa0a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa314009b30 tx=0x7fa3140046a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9a19245968e0b7fb server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa2feff5640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa314057070 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa3140303a0 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.534+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa314030db0 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa314065050 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 10 client.26956 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5640d17d3620 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa314030db0 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa31403d6c0 con 0x5640d1a94db0 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 10 client.26956 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 10 client.26956 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 20 client.26956 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1aa6fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 4 client.26956 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:41:08.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.535+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5640d1a00a70 con 0x5640d1aa4bf0 2024-05-09T09:41:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.536+0000 7fa31affd640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1aa6fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.536+0000 7fa31affd640 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1aa6fe0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fee9560f9680e83d server_cookie=696d3926ddd32ea9 in_seq=0 out_seq=0 2024-05-09T09:41:08.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.536+0000 7fa2feff5640 10 client.26956 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa2feff5640 1 -- 172.21.15.69:0/2356319358 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5640d1a00a70 con 0x5640d1aa4bf0 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa2feff5640 10 client.26956 handle_command_reply: tid=0 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 client.26956 shutdown 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 msgr2=0x5640d1aa0a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 0x5640d1aa0a90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa314009b30 tx=0x7fa3140046a0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 shutdown_connections 2024-05-09T09:41:08.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1aa6fe0 crc :-1 s=READY pgs=256 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a94db0 0x5640d1aa0a90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a8fc40 0x5640d1a99a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa2feff5640 0 client.26956 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 --2- 172.21.15.69:0/2356319358 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a99510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 >> 172.21.15.69:0/2356319358 conn(0x5640d1a8a8c0 msgr2=0x5640d1a89a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 shutdown_connections 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 1 -- 172.21.15.69:0/2356319358 wait complete. 2024-05-09T09:41:08.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.542+0000 7fa324476740 20 client.26956 trim_cache size 0 max 16384 2024-05-09T09:41:08.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.561+0000 7fa324476740 1 Processor -- start 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.561+0000 7fa324476740 1 -- start start 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a971a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a8d840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1aef60 con 0x5640d1a8fc40 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1ac830 con 0x5640d1a94db0 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa324476740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1aa680 con 0x5640d1a8f250 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35888/0 (socket says 172.21.15.69:35888) 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 -- 172.21.15.69:0/2496131728 learned_addr learned my addr 172.21.15.69:0/2496131728 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a8d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31affd640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a971a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 -- 172.21.15.69:0/2496131728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 msgr2=0x5640d1a971a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a971a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 -- 172.21.15.69:0/2496131728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 msgr2=0x5640d1a8d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a8d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 -- 172.21.15.69:0/2496131728 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5640d1aa7350 con 0x5640d1a8fc40 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31affd640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cce60 tx=0x7fa310009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1c346293759657b8 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.562+0000 7fa300ff9640 1 -- 172.21.15.69:0/2496131728 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa310018e60 con 0x5640d1a8fc40 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa300ff9640 1 -- 172.21.15.69:0/2496131728 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31009bcd0 con 0x5640d1a8fc40 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa300ff9640 1 -- 172.21.15.69:0/2496131728 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa31009b120 con 0x5640d1a8fc40 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 -- 172.21.15.69:0/2496131728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 msgr2=0x5640d1aa0a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 secure :-1 s=READY pgs=1088 cs=0 l=1 rev1=1 crypto rx=0x7fa3100cce60 tx=0x7fa310009700 comp rx=0 tx=0).stop 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 -- 172.21.15.69:0/2496131728 shutdown_connections 2024-05-09T09:41:08.568 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a8d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8fc40 0x5640d1aa0a50 unknown :-1 s=CLOSED pgs=1088 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 --2- 172.21.15.69:0/2496131728 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8f250 0x5640d1a971a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 -- 172.21.15.69:0/2496131728 >> 172.21.15.69:0/2496131728 conn(0x5640d1a8a8c0 msgr2=0x5640d1a8aca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 -- 172.21.15.69:0/2496131728 shutdown_connections 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.563+0000 7fa324476740 1 -- 172.21.15.69:0/2496131728 wait complete. 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 -- start start 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 Processor -- start 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8fc40 0x5640d1a9e280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a9f290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1aacf0 con 0x5640d1a8f250 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1ac160 con 0x5640d1a94db0 2024-05-09T09:41:08.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa31c1ab3c0 con 0x5640d1a8fc40 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa324476740 10 client.0 _ll_register_callbacks cb 0x7fa3100d5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35898/0 (socket says 172.21.15.69:35898) 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 -- 172.21.15.69:0/2239939501 learned_addr learned my addr 172.21.15.69:0/2239939501 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8fc40 0x5640d1a9e280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31b7fe640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a9f290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 -- 172.21.15.69:0/2239939501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8fc40 msgr2=0x5640d1a9e280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8fc40 0x5640d1a9e280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 -- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 msgr2=0x5640d1a9f290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a9f290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.564+0000 7fa31affd640 1 -- 172.21.15.69:0/2239939501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5640d1a9f880 con 0x5640d1a8f250 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa31affd640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa31c104b20 tx=0x7fa30c009d80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e410cc010543375d server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:08.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa2feff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa30c022030 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa30c00faa0 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa31009bcd0 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fa30c02f050 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa324476740 10 client.17781 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5640d1aa0c60 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.565+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa31009bcd0 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fa30c027410 con 0x5640d1a8f250 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 10 client.17781 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 10 client.17781 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 20 client.17781 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1b0d820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 4 client.17781 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:41:08.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5640d19fadc0 con 0x5640d1aa4bf0 2024-05-09T09:41:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1b0d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa31a7fc640 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1b0d820 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a8403d8d821d6a3 server_cookie=fc51098ae752b821 in_seq=0 out_seq=0 2024-05-09T09:41:08.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:08.566+0000 7fa2feff5640 10 client.17781 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:08.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 283157 27.29 MB/sec execute 426 sec latency 252.065 ms 2024-05-09T09:41:09.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:08 smithi069 ceph-mon[30123]: pgmap v1037: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-05-09T09:41:09.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:08 smithi163 ceph-mon[34354]: pgmap v1037: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-05-09T09:41:09.307 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.305+0000 7fa2feff5640 1 -- 172.21.15.69:0/2239939501 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5640d19fadc0 con 0x5640d1aa4bf0 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.305+0000 7fa2feff5640 10 client.17781 handle_command_reply: tid=0 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.305+0000 7fa324476740 1 client.17781 shutdown 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.305+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 msgr2=0x5640d1a9dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.305+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 secure :-1 s=READY pgs=1089 cs=0 l=1 rev1=1 crypto rx=0x7fa31c104b20 tx=0x7fa30c009d80 comp rx=0 tx=0).stop 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 shutdown_connections 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5640d1a94db0 0x5640d1a9f290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5640d1aa4bf0 0x5640d1b0d820 crc :-1 s=READY pgs=257 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.308 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5640d1a8fc40 0x5640d1a9e280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa2feff5640 0 client.17781 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 --2- 172.21.15.69:0/2239939501 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5640d1a8f250 0x5640d1a9dd40 unknown :-1 s=CLOSED pgs=1089 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 >> 172.21.15.69:0/2239939501 conn(0x5640d1a8a8c0 msgr2=0x5640d1a8d840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 shutdown_connections 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 1 -- 172.21.15.69:0/2239939501 wait complete. 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa324476740 20 client.17781 trim_cache size 0 max 16384 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:41:09.309 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:41:09.310 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 msgr2=0x7fa3040a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 0x7fa3040a50f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fa314049000 tx=0x7fa31404c000 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 msgr2=0x7fa31c19a1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 secure :-1 s=READY pgs=1087 cs=0 l=1 rev1=1 crypto rx=0x7fa31000e9f0 tx=0x7fa31000eeb0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 shutdown_connections 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa3040a2c30 0x7fa3040a50f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa31c105680 0x7fa31c1a1250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa31c104cc0 0x7fa31c19a1f0 unknown :-1 s=CLOSED pgs=1087 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 --2- 172.21.15.69:0/604275934 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa31c103ad0 0x7fa31c199cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.306+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 >> 172.21.15.69:0/604275934 conn(0x7fa31c0ff020 msgr2=0x7fa31c107ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.307+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 shutdown_connections 2024-05-09T09:41:09.311 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:09.307+0000 7fa321763640 1 -- 172.21.15.69:0/604275934 wait complete. 2024-05-09T09:41:09.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:08 smithi103 ceph-mon[34421]: pgmap v1037: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.1 MiB/s wr, 475 op/s 2024-05-09T09:41:09.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 283633 27.28 MB/sec execute 427 sec latency 617.818 ms 2024-05-09T09:41:10.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 284037 27.26 MB/sec execute 428 sec latency 481.333 ms 2024-05-09T09:41:11.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:10 smithi069 ceph-mon[30123]: pgmap v1038: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-05-09T09:41:11.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:10 smithi163 ceph-mon[34354]: pgmap v1038: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-05-09T09:41:11.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:10 smithi103 ceph-mon[34421]: pgmap v1038: 129 pgs: 129 active+clean; 697 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.1 MiB/s wr, 476 op/s 2024-05-09T09:41:11.840 INFO:tasks.workunit.client.0.smithi069.stdout: 10 284623 27.28 MB/sec execute 429 sec latency 84.126 ms 2024-05-09T09:41:12.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 285225 27.31 MB/sec execute 430 sec latency 69.416 ms 2024-05-09T09:41:13.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:12 smithi069 ceph-mon[30123]: pgmap v1039: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.0 MiB/s wr, 684 op/s 2024-05-09T09:41:13.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:12 smithi163 ceph-mon[34354]: pgmap v1039: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.0 MiB/s wr, 684 op/s 2024-05-09T09:41:13.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:12 smithi103 ceph-mon[34421]: pgmap v1039: 129 pgs: 129 active+clean; 702 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 8.0 MiB/s wr, 684 op/s 2024-05-09T09:41:13.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 285680 27.31 MB/sec execute 431 sec latency 68.836 ms 2024-05-09T09:41:14.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 286050 27.29 MB/sec execute 432 sec latency 194.985 ms 2024-05-09T09:41:15.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:14 smithi069 ceph-mon[30123]: pgmap v1040: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-05-09T09:41:15.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:14 smithi163 ceph-mon[34354]: pgmap v1040: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-05-09T09:41:15.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:14 smithi103 ceph-mon[34421]: pgmap v1040: 129 pgs: 129 active+clean; 695 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 5.8 MiB/s wr, 568 op/s 2024-05-09T09:41:15.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 286457 27.28 MB/sec execute 433 sec latency 315.344 ms 2024-05-09T09:41:16.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 286854 27.25 MB/sec execute 434 sec latency 106.009 ms 2024-05-09T09:41:17.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:16 smithi069 ceph-mon[30123]: pgmap v1041: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-05-09T09:41:17.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:16 smithi163 ceph-mon[34354]: pgmap v1041: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-05-09T09:41:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:16 smithi103 ceph-mon[34421]: pgmap v1041: 129 pgs: 129 active+clean; 698 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 5.9 MiB/s wr, 590 op/s 2024-05-09T09:41:17.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 287398 27.25 MB/sec execute 435 sec latency 102.887 ms 2024-05-09T09:41:18.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 288063 27.30 MB/sec execute 436 sec latency 60.188 ms 2024-05-09T09:41:19.168 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:18 smithi103 ceph-mon[34421]: pgmap v1042: 129 pgs: 129 active+clean; 680 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.1 MiB/s wr, 572 op/s 2024-05-09T09:41:19.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:18 smithi069 ceph-mon[30123]: pgmap v1042: 129 pgs: 129 active+clean; 680 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.1 MiB/s wr, 572 op/s 2024-05-09T09:41:19.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:18 smithi163 ceph-mon[34354]: pgmap v1042: 129 pgs: 129 active+clean; 680 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.1 MiB/s wr, 572 op/s 2024-05-09T09:41:19.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 288605 27.27 MB/sec execute 437 sec latency 78.543 ms 2024-05-09T09:41:20.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 288903 27.25 MB/sec execute 438 sec latency 299.815 ms 2024-05-09T09:41:21.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:20 smithi069 ceph-mon[30123]: pgmap v1043: 129 pgs: 129 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 553 op/s 2024-05-09T09:41:21.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:20 smithi163 ceph-mon[34354]: pgmap v1043: 129 pgs: 129 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 553 op/s 2024-05-09T09:41:21.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:20 smithi103 ceph-mon[34421]: pgmap v1043: 129 pgs: 129 active+clean; 683 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 5.8 MiB/s wr, 553 op/s 2024-05-09T09:41:21.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 289266 27.24 MB/sec execute 439 sec latency 445.656 ms 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 -- 172.21.15.69:0/3758450917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c108830 msgr2=0x7fd58c108c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3758450917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c108830 0x7fd58c108c50 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd57c009900 tx=0x7fd57c02a400 comp rx=0 tx=0).stop 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 -- 172.21.15.69:0/3758450917 shutdown_connections 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3758450917 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 0x7fd58c1124a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3758450917 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c109190 0x7fd58c10fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3758450917 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c108830 0x7fd58c108c50 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 -- 172.21.15.69:0/3758450917 >> 172.21.15.69:0/3758450917 conn(0x7fd58c06e8a0 msgr2=0x7fd58c070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:22.449 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 -- 172.21.15.69:0/3758450917 shutdown_connections 2024-05-09T09:41:22.450 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.449+0000 7fd5938a5640 1 -- 172.21.15.69:0/3758450917 wait complete. 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.450+0000 7fd5938a5640 1 Processor -- start 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.450+0000 7fd5938a5640 1 -- start start 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c108830 0x7fd58c1a0310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 0x7fd58c1a78c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c115d90 con 0x7fd58c1100b0 2024-05-09T09:41:22.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c115c10 con 0x7fd58c108830 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd5938a5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd58c115f10 con 0x7fd58c109190 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c108830 0x7fd58c1a0310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51608/0 (socket says 172.21.15.69:51608) 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd590e40640 1 -- 172.21.15.69:0/3779406419 learned_addr learned my addr 172.21.15.69:0/3779406419 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd591e42640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 0x7fd58c1a78c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd590e40640 1 -- 172.21.15.69:0/3779406419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c108830 msgr2=0x7fd58c1a0310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.451+0000 7fd590e40640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c108830 0x7fd58c1a0310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd590e40640 1 -- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 msgr2=0x7fd58c1a78c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd590e40640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 0x7fd58c1a78c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd590e40640 1 -- 172.21.15.69:0/3779406419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd58c1099c0 con 0x7fd58c109190 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd590e40640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b4a0 tx=0x7fd58800b960 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd5827fc640 1 -- 172.21.15.69:0/3779406419 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588014070 con 0x7fd58c109190 2024-05-09T09:41:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd58c109cb0 con 0x7fd58c109190 2024-05-09T09:41:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd58c10a2d0 con 0x7fd58c109190 2024-05-09T09:41:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd5827fc640 1 -- 172.21.15.69:0/3779406419 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd588010af0 con 0x7fd58c109190 2024-05-09T09:41:22.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.452+0000 7fd5827fc640 1 -- 172.21.15.69:0/3779406419 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd58801e060 con 0x7fd58c109190 2024-05-09T09:41:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.455+0000 7fd5965b1740 1 Processor -- start 2024-05-09T09:41:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 -- start start 2024-05-09T09:41:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5827fc640 1 -- 172.21.15.69:0/3779406419 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd588017070 con 0x7fd58c109190 2024-05-09T09:41:22.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15250 0x55cf88f15670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f1d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf88e79340 con 0x55cf88f1adb0 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf88e21290 con 0x55cf88f15c40 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.456+0000 7fd5965b1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf88e75300 con 0x55cf88f15250 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.457+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.457+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:51396/0 (socket says 172.21.15.69:51396) 2024-05-09T09:41:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.457+0000 7fd591641640 1 -- 172.21.15.69:0/3857849452 learned_addr learned my addr 172.21.15.69:0/3857849452 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.457+0000 7fd5827fc640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 0x7fd5780a5380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.457+0000 7fd591e42640 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15250 0x55cf88f15670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd5827fc640 1 -- 172.21.15.69:0/3779406419 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd5880ce140 con 0x7fd58c109190 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 -- 172.21.15.69:0/3857849452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15250 msgr2=0x55cf88f15670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15250 0x55cf88f15670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 -- 172.21.15.69:0/3857849452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 msgr2=0x55cf88f1d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f1d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 -- 172.21.15.69:0/3857849452 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf88e458e0 con 0x55cf88f15c40 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591641640 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0314e0 tx=0x7fd57c004170 comp rx=0 tx=0).ready entity=mon.2 client_cookie=df74acd6cc2f70d9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.458+0000 7fd591e42640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 0x7fd5780a5380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.459+0000 7fd5777fe640 1 -- 172.21.15.69:0/3857849452 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c035040 con 0x55cf88f15c40 2024-05-09T09:41:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.459+0000 7fd5777fe640 1 -- 172.21.15.69:0/3857849452 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c005520 con 0x55cf88f15c40 2024-05-09T09:41:22.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.459+0000 7fd5777fe640 1 -- 172.21.15.69:0/3857849452 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd57c031910 con 0x55cf88f15c40 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 -- 172.21.15.69:0/3857849452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 msgr2=0x55cf88f1a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fd57c0314e0 tx=0x7fd57c004170 comp rx=0 tx=0).stop 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd591e42640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 0x7fd5780a5380 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55cf88f164a0 tx=0x7fd584004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 -- 172.21.15.69:0/3857849452 shutdown_connections 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f1d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15c40 0x55cf88f1a740 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3857849452 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15250 0x55cf88f15670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.460+0000 7fd5965b1740 1 -- 172.21.15.69:0/3857849452 >> 172.21.15.69:0/3857849452 conn(0x55cf88f108c0 msgr2=0x55cf88f10cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:22.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.461+0000 7fd5965b1740 1 -- 172.21.15.69:0/3857849452 shutdown_connections 2024-05-09T09:41:22.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.462+0000 7fd5965b1740 1 -- 172.21.15.69:0/3857849452 wait complete. 2024-05-09T09:41:22.462 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.463+0000 7fd5965b1740 1 -- start start 2024-05-09T09:41:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.463+0000 7fd5965b1740 1 Processor -- start 2024-05-09T09:41:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15250 0x55cf88f1f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 0x55cf88f26ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55cf88f1dd00 con 0x55cf88f15250 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55cf88e34790 con 0x55cf88f1adb0 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55cf88f1de80 con 0x55cf88f15c40 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.464+0000 7fd5965b1740 10 client.0 _ll_register_callbacks cb 0x55cf88e6fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51636/0 (socket says 172.21.15.69:51636) 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 -- 172.21.15.69:0/1341662841 learned_addr learned my addr 172.21.15.69:0/1341662841 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:22.464 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd591e42640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 0x55cf88f26ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.465 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd591641640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15250 0x55cf88f1f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 -- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 msgr2=0x55cf88f26ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 0x55cf88f26ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 -- 172.21.15.69:0/1341662841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15250 msgr2=0x55cf88f1f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15250 0x55cf88f1f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 -- 172.21.15.69:0/1341662841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf88b8d210 con 0x55cf88f15c40 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd591e42640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 0x55cf88f26ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.465+0000 7fd590e40640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd58809a660 tx=0x7fd58809a310 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f31ce1160d8d6cc server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.466+0000 7fd574ff9640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.466+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588014070 con 0x55cf88f15c40 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.466+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd588096d40 con 0x55cf88f15c40 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.466+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd57c005520 con 0x55cf88f15c40 2024-05-09T09:41:22.466 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.466+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd5880850f0 con 0x55cf88f15c40 2024-05-09T09:41:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.468+0000 7fd5965b1740 10 client.26974 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:22.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.468+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cf88c59620 con 0x55cf88f15c40 2024-05-09T09:41:22.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.468+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd57c005520 con 0x55cf88f15c40 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fd588099b20 con 0x55cf88f15c40 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 10 client.26974 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 10 client.26974 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 20 client.26974 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f2afd0 0x55cf88f2d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 4 client.26974 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:41:22.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.469+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55cf88e86a70 con 0x55cf88f2afd0 2024-05-09T09:41:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.472+0000 7fd591641640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f2afd0 0x55cf88f2d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.473+0000 7fd574ff9640 10 client.26974 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:22.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.473+0000 7fd591641640 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f2afd0 0x55cf88f2d400 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8267eaa8df53a94 server_cookie=1810893deb02c042 in_seq=0 out_seq=0 2024-05-09T09:41:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.478+0000 7fd574ff9640 1 -- 172.21.15.69:0/1341662841 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55cf88e86a70 con 0x55cf88f2afd0 2024-05-09T09:41:22.478 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.478+0000 7fd574ff9640 10 client.26974 handle_command_reply: tid=0 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.478+0000 7fd5965b1740 1 client.26974 shutdown 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 msgr2=0x55cf88f1fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fd58809a660 tx=0x7fd58809a310 comp rx=0 tx=0).stop 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 shutdown_connections 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f1adb0 0x55cf88f26ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f2afd0 0x55cf88f2d400 crc :-1 s=READY pgs=258 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f1fa90 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1341662841 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15250 0x55cf88f1f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd574ff9640 0 client.26974 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 >> 172.21.15.69:0/1341662841 conn(0x55cf88f108c0 msgr2=0x55cf88f0fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 shutdown_connections 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 1 -- 172.21.15.69:0/1341662841 wait complete. 2024-05-09T09:41:22.479 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.479+0000 7fd5965b1740 20 client.26974 trim_cache size 0 max 16384 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 Processor -- start 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 -- start start 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f13f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f31d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd5700123f0 con 0x55cf88f1adb0 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd57000fcc0 con 0x55cf88f15250 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd5965b1740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd57000db10 con 0x55cf88f15c40 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f13f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.502 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd591e42640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f31d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd590e40640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:51638/0 (socket says 172.21.15.69:51638) 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd590e40640 1 -- 172.21.15.69:0/3591524946 learned_addr learned my addr 172.21.15.69:0/3591524946 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.500+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f13f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:51402/0 (socket says 172.21.15.69:51402) 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 -- 172.21.15.69:0/3591524946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 msgr2=0x55cf88f13f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f13f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 -- 172.21.15.69:0/3591524946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 msgr2=0x55cf88f31d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f31d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 -- 172.21.15.69:0/3591524946 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55cf88f1a1d0 con 0x55cf88f15c40 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd591e42640 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f31d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd590e40640 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b470 tx=0x7fd588098be0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8013be96203e2847 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd576ffd640 1 -- 172.21.15.69:0/3591524946 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588014050 con 0x55cf88f15c40 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd576ffd640 1 -- 172.21.15.69:0/3591524946 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5880a3290 con 0x55cf88f15c40 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.501+0000 7fd576ffd640 1 -- 172.21.15.69:0/3591524946 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588096d40 con 0x55cf88f15c40 2024-05-09T09:41:22.503 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- 172.21.15.69:0/3591524946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 msgr2=0x55cf88f26a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b470 tx=0x7fd588098be0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- 172.21.15.69:0/3591524946 shutdown_connections 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f1adb0 0x55cf88f31d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f15c40 0x55cf88f26a90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 --2- 172.21.15.69:0/3591524946 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f13f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- 172.21.15.69:0/3591524946 >> 172.21.15.69:0/3591524946 conn(0x55cf88f108c0 msgr2=0x55cf88f13870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- 172.21.15.69:0/3591524946 shutdown_connections 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- 172.21.15.69:0/3591524946 wait complete. 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 -- start start 2024-05-09T09:41:22.504 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.502+0000 7fd5965b1740 1 Processor -- start 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f24590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f1adb0 0x55cf88f25ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd57000e180 con 0x55cf88f15c40 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd591641640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f24590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:50798/0 (socket says 172.21.15.69:50798) 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 -- 172.21.15.69:0/1989784146 learned_addr learned my addr 172.21.15.69:0/1989784146 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:22.507 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd57000f5f0 con 0x55cf88f15250 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd57000e850 con 0x55cf88f1adb0 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd591e42640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f1adb0 0x55cf88f25ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd5965b1740 10 client.0 _ll_register_callbacks cb 0x7fd5880092f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 -- 172.21.15.69:0/1989784146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f1adb0 msgr2=0x55cf88f25ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f1adb0 0x55cf88f25ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 -- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 msgr2=0x55cf88f24590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f24590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55cf88f260d0 con 0x55cf88f15c40 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd590e40640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd5880970a0 tx=0x7fd588000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fd5fcc6cc350b48b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.503+0000 7fd574ff9640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd588014070 con 0x55cf88f15c40 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd5880997a0 con 0x55cf88f15c40 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fd5880a3290 con 0x55cf88f15c40 2024-05-09T09:41:22.508 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7fd58809e9d0 con 0x55cf88f15c40 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd5965b1740 10 client.17796 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55cf88f91930 con 0x55cf88f15c40 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd5880ba2d0 con 0x55cf88f15c40 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.504+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7fd588017a10 con 0x55cf88f15c40 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 10 client.17796 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 10 client.17796 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 20 client.17796 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f91eb0 0x55cf88f94310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 4 client.17796 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x7fd5880a3290 con 0x55cf88f91eb0 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd591641640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f91eb0 0x55cf88f94310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd574ff9640 10 client.17796 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:22.509 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:22.505+0000 7fd591641640 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f91eb0 0x55cf88f94310 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbe48fb3459e38e7 server_cookie=2016c30bec63e809 in_seq=0 out_seq=0 2024-05-09T09:41:22.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 289555 27.21 MB/sec execute 440 sec latency 276.757 ms 2024-05-09T09:41:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd574ff9640 1 -- 172.21.15.69:0/1989784146 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x7fd5880a3290 con 0x55cf88f91eb0 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd574ff9640 10 client.17796 handle_command_reply: tid=0 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd5965b1740 1 client.17796 shutdown 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 msgr2=0x55cf88f24ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 crypto rx=0x7fd5880970a0 tx=0x7fd588000bc0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.209+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 shutdown_connections 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55cf88f91eb0 0x55cf88f94310 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55cf88f1adb0 0x55cf88f25ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55cf88f15c40 0x55cf88f24ad0 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 --2- 172.21.15.69:0/1989784146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55cf88f15250 0x55cf88f24590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 >> 172.21.15.69:0/1989784146 conn(0x55cf88f108c0 msgr2=0x55cf88f0f170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd574ff9640 0 client.17796 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 shutdown_connections 2024-05-09T09:41:23.210 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 1 -- 172.21.15.69:0/1989784146 wait complete. 2024-05-09T09:41:23.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.210+0000 7fd5965b1740 20 client.17796 trim_cache size 0 max 16384 2024-05-09T09:41:23.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:41:23.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:41:23.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:41:23.211 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 msgr2=0x7fd5780a5380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 0x7fd5780a5380 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x55cf88f164a0 tx=0x7fd584004a40 comp rx=0 tx=0).stop 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 msgr2=0x7fd58c1a0850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd58800b4a0 tx=0x7fd58800b960 comp rx=0 tx=0).stop 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 shutdown_connections 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd5780a2ec0 0x7fd5780a5380 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.212 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd58c1100b0 0x7fd58c1a78c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd58c109190 0x7fd58c1a0850 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 --2- 172.21.15.69:0/3779406419 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd58c108830 0x7fd58c1a0310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 >> 172.21.15.69:0/3779406419 conn(0x7fd58c06e8a0 msgr2=0x7fd58c070440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 shutdown_connections 2024-05-09T09:41:23.213 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:23.211+0000 7fd5938a5640 1 -- 172.21.15.69:0/3779406419 wait complete. 2024-05-09T09:41:23.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:22 smithi069 ceph-mon[30123]: pgmap v1044: 129 pgs: 129 active+clean; 693 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-05-09T09:41:23.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:22 smithi163 ceph-mon[34354]: pgmap v1044: 129 pgs: 129 active+clean; 693 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-05-09T09:41:23.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:22 smithi103 ceph-mon[34421]: pgmap v1044: 129 pgs: 129 active+clean; 693 MiB data, 10 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.5 MiB/s wr, 677 op/s 2024-05-09T09:41:23.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 290059 27.21 MB/sec execute 441 sec latency 596.491 ms 2024-05-09T09:41:24.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 290741 27.24 MB/sec execute 442 sec latency 50.513 ms 2024-05-09T09:41:25.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:24 smithi069 ceph-mon[30123]: pgmap v1045: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.9 MiB/s wr, 492 op/s 2024-05-09T09:41:25.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:24 smithi163 ceph-mon[34354]: pgmap v1045: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.9 MiB/s wr, 492 op/s 2024-05-09T09:41:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:24 smithi103 ceph-mon[34421]: pgmap v1045: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 5.9 MiB/s wr, 492 op/s 2024-05-09T09:41:25.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 291126 27.24 MB/sec execute 443 sec latency 180.446 ms 2024-05-09T09:41:26.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 291398 27.21 MB/sec execute 444 sec latency 383.249 ms 2024-05-09T09:41:27.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:26 smithi069 ceph-mon[30123]: pgmap v1046: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.6 MiB/s wr, 468 op/s 2024-05-09T09:41:27.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:26 smithi163 ceph-mon[34354]: pgmap v1046: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.6 MiB/s wr, 468 op/s 2024-05-09T09:41:27.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:26 smithi103 ceph-mon[34421]: pgmap v1046: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 5.6 MiB/s wr, 468 op/s 2024-05-09T09:41:27.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 291833 27.19 MB/sec execute 445 sec latency 88.094 ms 2024-05-09T09:41:28.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 292387 27.18 MB/sec execute 446 sec latency 49.346 ms 2024-05-09T09:41:29.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:28 smithi069 ceph-mon[30123]: pgmap v1047: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.3 MiB/s wr, 653 op/s 2024-05-09T09:41:29.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:28 smithi163 ceph-mon[34354]: pgmap v1047: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.3 MiB/s wr, 653 op/s 2024-05-09T09:41:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:28 smithi103 ceph-mon[34421]: pgmap v1047: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.3 MiB/s wr, 653 op/s 2024-05-09T09:41:29.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 292979 27.20 MB/sec execute 447 sec latency 66.560 ms 2024-05-09T09:41:30.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 293755 27.23 MB/sec execute 448 sec latency 66.134 ms 2024-05-09T09:41:31.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:30 smithi069 ceph-mon[30123]: pgmap v1048: 129 pgs: 129 active+clean; 690 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-05-09T09:41:31.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:30 smithi163 ceph-mon[34354]: pgmap v1048: 129 pgs: 129 active+clean; 690 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-05-09T09:41:31.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:30 smithi103 ceph-mon[34421]: pgmap v1048: 129 pgs: 129 active+clean; 690 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 6.2 MiB/s wr, 580 op/s 2024-05-09T09:41:31.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 294019 27.21 MB/sec execute 449 sec latency 324.034 ms 2024-05-09T09:41:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:41:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:31 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:41:32.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.306 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:31 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:41:32.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:31 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:32.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 294348 27.18 MB/sec execute 450 sec latency 187.586 ms 2024-05-09T09:41:33.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:32 smithi069 ceph-mon[30123]: pgmap v1049: 129 pgs: 129 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.6 MiB/s wr, 661 op/s 2024-05-09T09:41:33.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:32 smithi163 ceph-mon[34354]: pgmap v1049: 129 pgs: 129 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.6 MiB/s wr, 661 op/s 2024-05-09T09:41:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:32 smithi103 ceph-mon[34421]: pgmap v1049: 129 pgs: 129 active+clean; 697 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.6 MiB/s wr, 661 op/s 2024-05-09T09:41:33.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 294886 27.20 MB/sec execute 451 sec latency 71.034 ms 2024-05-09T09:41:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:41:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:41:34.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:33 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:41:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:41:34.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:33 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:41:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:41:34.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:33 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:41:34.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 295448 27.20 MB/sec execute 452 sec latency 61.121 ms 2024-05-09T09:41:35.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:34 smithi069 ceph-mon[30123]: pgmap v1050: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.5 MiB/s wr, 590 op/s 2024-05-09T09:41:35.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:34 smithi163 ceph-mon[34354]: pgmap v1050: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.5 MiB/s wr, 590 op/s 2024-05-09T09:41:35.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:34 smithi103 ceph-mon[34421]: pgmap v1050: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 6.5 MiB/s wr, 590 op/s 2024-05-09T09:41:35.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 295960 27.19 MB/sec execute 453 sec latency 63.157 ms 2024-05-09T09:41:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.386+0000 7f680ad61640 1 -- 172.21.15.69:0/3897704794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f680410b1a0 msgr2=0x7f680410d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.388 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.386+0000 7f680ad61640 1 --2- 172.21.15.69:0/3897704794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f680410b1a0 0x7f680410d610 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 crypto rx=0x7f67fc009ca0 tx=0x7f67fc02a8f0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.393 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.394+0000 7f680ad61640 1 -- 172.21.15.69:0/3897704794 shutdown_connections 2024-05-09T09:41:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.394+0000 7f680ad61640 1 --2- 172.21.15.69:0/3897704794 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410db50 0x7f680410dfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.394+0000 7f680ad61640 1 --2- 172.21.15.69:0/3897704794 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f680410b1a0 0x7f680410d610 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.394+0000 7f680ad61640 1 --2- 172.21.15.69:0/3897704794 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6804108830 0x7f680410ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.394 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.394+0000 7f680ad61640 1 -- 172.21.15.69:0/3897704794 >> 172.21.15.69:0/3897704794 conn(0x7f680406e8a0 msgr2=0x7f6804070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:36.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.399+0000 7f680ad61640 1 -- 172.21.15.69:0/3897704794 shutdown_connections 2024-05-09T09:41:36.399 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.400+0000 7f680ad61640 1 -- 172.21.15.69:0/3897704794 wait complete. 2024-05-09T09:41:36.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.401+0000 7f680ad61640 1 Processor -- start 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 -- start start 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 0x7f68041a6990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 0x7f68041a6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f680410db50 0x7f68041adef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f680411c3b0 con 0x7f6804108830 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f680411c230 con 0x7f680410b1a0 2024-05-09T09:41:36.404 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.404+0000 7f680ad61640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f680411c530 con 0x7f680410db50 2024-05-09T09:41:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.407+0000 7f6803fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 0x7f68041a6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.407+0000 7f6803fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 0x7f68041a6ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:53776/0 (socket says 172.21.15.69:53776) 2024-05-09T09:41:36.406 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.407+0000 7f6803fff640 1 -- 172.21.15.69:0/1340527473 learned_addr learned my addr 172.21.15.69:0/1340527473 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.407+0000 7f68092fe640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f680410db50 0x7f68041adef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.407 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 0x7f68041a6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 -- 172.21.15.69:0/1340527473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f680410db50 msgr2=0x7f68041adef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f680410db50 0x7f68041adef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 -- 172.21.15.69:0/1340527473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 msgr2=0x7f68041a6ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 0x7f68041a6ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.408+0000 7f6808afd640 1 -- 172.21.15.69:0/1340527473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f68041ae560 con 0x7f6804108830 2024-05-09T09:41:36.408 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.409+0000 7f6808afd640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 0x7f68041a6990 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f67ec005fd0 tx=0x7f67ec00dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.410+0000 7f6801ffb640 1 -- 172.21.15.69:0/1340527473 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec009c30 con 0x7f6804108830 2024-05-09T09:41:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.410+0000 7f6801ffb640 1 -- 172.21.15.69:0/1340527473 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67ec009dd0 con 0x7f6804108830 2024-05-09T09:41:36.409 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.410+0000 7f6801ffb640 1 -- 172.21.15.69:0/1340527473 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec010460 con 0x7f6804108830 2024-05-09T09:41:36.413 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.414+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f68041ae850 con 0x7f6804108830 2024-05-09T09:41:36.416 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.414+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f68041aed20 con 0x7f6804108830 2024-05-09T09:41:36.417 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.417+0000 7f680da6e740 1 Processor -- start 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 -- start start 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf51e740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf51edb0 0x55b7cf521240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf47d340 con 0x55b7cf519250 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf425290 con 0x55b7cf519c40 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.423+0000 7f680da6e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf479300 con 0x55b7cf51edb0 2024-05-09T09:41:36.423 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.424+0000 7f6801ffb640 1 -- 172.21.15.69:0/1340527473 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f67ec003f60 con 0x7f6804108830 2024-05-09T09:41:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.424+0000 7f68092fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.425 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.424+0000 7f68092fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:36100/0 (socket says 172.21.15.69:36100) 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.424+0000 7f68092fe640 1 -- 172.21.15.69:0/2651517040 learned_addr learned my addr 172.21.15.69:0/2651517040 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.424+0000 7f6808afd640 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf51e740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 -- 172.21.15.69:0/2651517040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf51edb0 msgr2=0x55b7cf521240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf51edb0 0x55b7cf521240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 -- 172.21.15.69:0/2651517040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 msgr2=0x55b7cf51e740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf51e740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 -- 172.21.15.69:0/2651517040 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7cf4498e0 con 0x55b7cf519250 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.425+0000 7f68092fe640 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f67f40099c0 tx=0x7f67f4009e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6f6f66a5b7bc16e3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.426+0000 7f67faffd640 1 -- 172.21.15.69:0/2651517040 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67f4017070 con 0x55b7cf519250 2024-05-09T09:41:36.426 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.426+0000 7f67faffd640 1 -- 172.21.15.69:0/2651517040 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67f4013470 con 0x55b7cf519250 2024-05-09T09:41:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.427+0000 7f680da6e740 1 -- 172.21.15.69:0/2651517040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 msgr2=0x55b7cf519670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.427+0000 7f680da6e740 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 crypto rx=0x7f67f40099c0 tx=0x7f67f4009e80 comp rx=0 tx=0).stop 2024-05-09T09:41:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.427+0000 7f6801ffb640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 0x7f67f00a5330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.427 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.428+0000 7f6801ffb640 1 -- 172.21.15.69:0/1340527473 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67ec0cedb0 con 0x7f6804108830 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f680da6e740 1 -- 172.21.15.69:0/2651517040 shutdown_connections 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f680da6e740 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf51edb0 0x55b7cf521240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f680da6e740 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf51e740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f680da6e740 1 --2- 172.21.15.69:0/2651517040 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519250 0x55b7cf519670 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f680da6e740 1 -- 172.21.15.69:0/2651517040 >> 172.21.15.69:0/2651517040 conn(0x55b7cf5148c0 msgr2=0x55b7cf514cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:36.431 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.430+0000 7f68092fe640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 0x7f67f00a5330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.436 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.437+0000 7f680da6e740 1 -- 172.21.15.69:0/2651517040 shutdown_connections 2024-05-09T09:41:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.444+0000 7f680da6e740 1 -- 172.21.15.69:0/2651517040 wait complete. 2024-05-09T09:41:36.443 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.444+0000 7f680da6e740 1 -- start start 2024-05-09T09:41:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.445+0000 7f680da6e740 1 Processor -- start 2024-05-09T09:41:36.444 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.445+0000 7f68092fe640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 0x7f67f00a5330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f67f400e5a0 tx=0x7f67f4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 0x55b7cf523450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 0x55b7cf523990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf52a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf521db0 con 0x55b7cf51edb0 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf438790 con 0x55b7cf519250 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55b7cf521f30 con 0x55b7cf519c40 2024-05-09T09:41:36.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.455+0000 7f680da6e740 10 client.0 _ll_register_callbacks cb 0x55b7cf473a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.457+0000 7f6803fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 0x55b7cf523990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.457+0000 7f6808afd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 0x55b7cf523450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.457+0000 7f6803fff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 0x55b7cf523990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:36358/0 (socket says 172.21.15.69:36358) 2024-05-09T09:41:36.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.457+0000 7f6803fff640 1 -- 172.21.15.69:0/4206674247 learned_addr learned my addr 172.21.15.69:0/4206674247 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.458+0000 7f6808afd640 1 -- 172.21.15.69:0/4206674247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 msgr2=0x55b7cf523990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f6808afd640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 0x55b7cf523990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f6808afd640 1 -- 172.21.15.69:0/4206674247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 msgr2=0x55b7cf52a9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f68092fe640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf52a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f6808afd640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf52a9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f6808afd640 1 -- 172.21.15.69:0/4206674247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7cf191210 con 0x55b7cf519250 2024-05-09T09:41:36.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f68092fe640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf52a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:41:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.459+0000 7f6808afd640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 0x55b7cf523450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55b7cf51a6b0 tx=0x7f67ec09b140 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d6dbd2169aca77 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.460+0000 7f67e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:41:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.460+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec09b2b0 con 0x55b7cf519250 2024-05-09T09:41:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.460+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67f4013470 con 0x55b7cf519250 2024-05-09T09:41:36.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.460+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67ec0035f0 con 0x55b7cf519250 2024-05-09T09:41:36.463 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.464+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67ec09f660 con 0x55b7cf519250 2024-05-09T09:41:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.468+0000 7f680da6e740 10 client.27056 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:36.467 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.468+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b7cf25d620 con 0x55b7cf519250 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67f4013470 con 0x55b7cf519250 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67ec0a3a90 con 0x55b7cf519250 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 10 client.27056 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 10 client.27056 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 20 client.27056 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf52f3e0 0x55b7cf531850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.468 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 4 client.27056 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:41:36.469 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.469+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55b7cf48aa70 con 0x55b7cf52f3e0 2024-05-09T09:41:36.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.472+0000 7f6803fff640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf52f3e0 0x55b7cf531850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.474+0000 7f6803fff640 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf52f3e0 0x55b7cf531850 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e437d49489ef2ca8 server_cookie=897a0a2b307d49fd in_seq=0 out_seq=0 2024-05-09T09:41:36.474 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.474+0000 7f67e3fff640 10 client.27056 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:36.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.476+0000 7f67e3fff640 1 -- 172.21.15.69:0/4206674247 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55b7cf48aa70 con 0x55b7cf52f3e0 2024-05-09T09:41:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.476+0000 7f67e3fff640 10 client.27056 handle_command_reply: tid=0 2024-05-09T09:41:36.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.476+0000 7f680da6e740 1 client.27056 shutdown 2024-05-09T09:41:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.483+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 msgr2=0x55b7cf523450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.483+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 0x55b7cf523450 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x55b7cf51a6b0 tx=0x7f67ec09b140 comp rx=0 tx=0).stop 2024-05-09T09:41:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 shutdown_connections 2024-05-09T09:41:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf52a9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.484 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf52f3e0 0x55b7cf531850 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519c40 0x55b7cf523990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 --2- 172.21.15.69:0/4206674247 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519250 0x55b7cf523450 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 >> 172.21.15.69:0/4206674247 conn(0x55b7cf5148c0 msgr2=0x55b7cf513a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:36.485 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.485+0000 7f67e3fff640 0 client.27056 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:36.489 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.489+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 shutdown_connections 2024-05-09T09:41:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.491+0000 7f680da6e740 1 -- 172.21.15.69:0/4206674247 wait complete. 2024-05-09T09:41:36.491 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.492+0000 7f680da6e740 20 client.27056 trim_cache size 0 max 16384 2024-05-09T09:41:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.527+0000 7f680da6e740 1 Processor -- start 2024-05-09T09:41:36.526 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.527+0000 7f680da6e740 1 -- start start 2024-05-09T09:41:36.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.527+0000 7f680da6e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf517d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.527+0000 7f680da6e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 0x55b7cf52a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f680da6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 0x55b7cf523060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f680da6e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68041bbb80 con 0x55b7cf519c40 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f680da6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68041b9450 con 0x55b7cf51edb0 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f680da6e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f68041b72a0 con 0x55b7cf519250 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f68092fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 0x55b7cf523060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f68092fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 0x55b7cf523060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:53794/0 (socket says 172.21.15.69:53794) 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.528+0000 7f68092fe640 1 -- 172.21.15.69:0/3188468906 learned_addr learned my addr 172.21.15.69:0/3188468906 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:36.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.529+0000 7f6803fff640 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 0x55b7cf52a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.531+0000 7f6803fff640 1 -- 172.21.15.69:0/3188468906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 msgr2=0x55b7cf517d50 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:41:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.531+0000 7f6803fff640 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf517d50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.531+0000 7f6803fff640 1 -- 172.21.15.69:0/3188468906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 msgr2=0x55b7cf523060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.531+0000 7f6803fff640 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 0x55b7cf523060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.531+0000 7f6803fff640 1 -- 172.21.15.69:0/3188468906 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55b7cf51a4d0 con 0x55b7cf519c40 2024-05-09T09:41:36.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.536+0000 7f6803fff640 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 0x55b7cf52a990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f67fc02f570 tx=0x7f67fc031480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6a8e3b54957b1978 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.538+0000 7f67fa7fc640 1 -- 172.21.15.69:0/3188468906 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc003970 con 0x55b7cf519c40 2024-05-09T09:41:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.538+0000 7f67fa7fc640 1 -- 172.21.15.69:0/3188468906 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67fc036040 con 0x55b7cf519c40 2024-05-09T09:41:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.538+0000 7f67fa7fc640 1 -- 172.21.15.69:0/3188468906 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67fc0043c0 con 0x55b7cf519c40 2024-05-09T09:41:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.541+0000 7f680da6e740 1 -- 172.21.15.69:0/3188468906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 msgr2=0x55b7cf52a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:36.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.541+0000 7f680da6e740 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 0x55b7cf52a990 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 crypto rx=0x7f67fc02f570 tx=0x7f67fc031480 comp rx=0 tx=0).stop 2024-05-09T09:41:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.545+0000 7f680da6e740 1 -- 172.21.15.69:0/3188468906 shutdown_connections 2024-05-09T09:41:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.545+0000 7f680da6e740 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf51edb0 0x55b7cf523060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.545+0000 7f680da6e740 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf519c40 0x55b7cf52a990 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.545+0000 7f680da6e740 1 --2- 172.21.15.69:0/3188468906 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf517d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.545+0000 7f680da6e740 1 -- 172.21.15.69:0/3188468906 >> 172.21.15.69:0/3188468906 conn(0x55b7cf5148c0 msgr2=0x55b7cf517640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:36.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.546+0000 7f680da6e740 1 -- 172.21.15.69:0/3188468906 shutdown_connections 2024-05-09T09:41:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.549+0000 7f680da6e740 1 -- 172.21.15.69:0/3188468906 wait complete. 2024-05-09T09:41:36.548 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.549+0000 7f680da6e740 1 -- start start 2024-05-09T09:41:36.552 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.549+0000 7f680da6e740 1 Processor -- start 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf527f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf528f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f68041b7910 con 0x55b7cf51edb0 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f68041b8d80 con 0x55b7cf519c40 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f68041b7fe0 con 0x55b7cf519250 2024-05-09T09:41:36.554 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.554+0000 7f680da6e740 10 client.0 _ll_register_callbacks cb 0x7f67ec005c00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.557+0000 7f6808afd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.557+0000 7f6808afd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:36368/0 (socket says 172.21.15.69:36368) 2024-05-09T09:41:36.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.557+0000 7f6808afd640 1 -- 172.21.15.69:0/1584279997 learned_addr learned my addr 172.21.15.69:0/1584279997 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.558+0000 7f6803fff640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf527f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.558+0000 7f68092fe640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf528f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.560+0000 7f6808afd640 1 -- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 msgr2=0x55b7cf527f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.560+0000 7f6808afd640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf527f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.560+0000 7f6808afd640 1 -- 172.21.15.69:0/1584279997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 msgr2=0x55b7cf528f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.560+0000 7f6808afd640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf528f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:36.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.560+0000 7f6808afd640 1 -- 172.21.15.69:0/1584279997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55b7cf529530 con 0x55b7cf519250 2024-05-09T09:41:36.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.562+0000 7f6808afd640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f67ec09f820 tx=0x7f67ec005b90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1fcc9322e0971cd server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:36.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.564+0000 7f67e3fff640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:41:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.564+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67ec09a870 con 0x55b7cf519250 2024-05-09T09:41:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.564+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67fc036040 con 0x55b7cf519250 2024-05-09T09:41:36.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.564+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67ec097920 con 0x55b7cf519250 2024-05-09T09:41:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.567+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67ec09b2b0 con 0x55b7cf519250 2024-05-09T09:41:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.567+0000 7f680da6e740 10 client.26992 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:36.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.567+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x55b7cf52a8b0 con 0x55b7cf519250 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.570+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67fc036040 con 0x55b7cf519250 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.570+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67ec0971f0 con 0x55b7cf519250 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.571+0000 7f680da6e740 10 client.26992 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.571+0000 7f680da6e740 10 client.26992 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.571+0000 7f680da6e740 20 client.26992 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.571+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf5a6d40 0x55b7cf5a9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.572+0000 7f680da6e740 4 client.26992 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:41:36.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.572+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55b7cf484dc0 con 0x55b7cf5a6d40 2024-05-09T09:41:36.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.578+0000 7f6803fff640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf5a6d40 0x55b7cf5a9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.585+0000 7f6803fff640 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf5a6d40 0x55b7cf5a9160 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=105e29f7eab25c21 server_cookie=66682d2a693b7f13 in_seq=0 out_seq=0 2024-05-09T09:41:36.585 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:36.586+0000 7f67e3fff640 10 client.26992 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:36.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 296092 27.14 MB/sec execute 454 sec latency 631.181 ms 2024-05-09T09:41:37.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:36 smithi069 ceph-mon[30123]: pgmap v1051: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.3 MiB/s wr, 549 op/s 2024-05-09T09:41:37.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:36 smithi163 ceph-mon[34354]: pgmap v1051: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.3 MiB/s wr, 549 op/s 2024-05-09T09:41:37.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:36 smithi103 ceph-mon[34421]: pgmap v1051: 129 pgs: 129 active+clean; 703 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.3 MiB/s wr, 549 op/s 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f67e3fff640 1 -- 172.21.15.69:0/1584279997 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55b7cf484dc0 con 0x55b7cf5a6d40 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f67e3fff640 10 client.26992 handle_command_reply: tid=0 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 client.26992 shutdown 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 msgr2=0x55b7cf5279f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f67ec09f820 tx=0x7f67ec005b90 comp rx=0 tx=0).stop 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 shutdown_connections 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55b7cf51edb0 0x55b7cf528f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55b7cf519c40 0x55b7cf527f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.533 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55b7cf5a6d40 0x55b7cf5a9160 crc :-1 s=READY pgs=261 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 --2- 172.21.15.69:0/1584279997 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55b7cf519250 0x55b7cf5279f0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.532+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 >> 172.21.15.69:0/1584279997 conn(0x55b7cf5148c0 msgr2=0x55b7cf523010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.533+0000 7f67e3fff640 0 client.26992 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.533+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 shutdown_connections 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.533+0000 7f680da6e740 1 -- 172.21.15.69:0/1584279997 wait complete. 2024-05-09T09:41:37.534 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.533+0000 7f680da6e740 20 client.26992 trim_cache size 0 max 16384 2024-05-09T09:41:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:41:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:41:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:41:37.535 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:41:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 msgr2=0x7f67f00a5330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 0x7f67f00a5330 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f67f400e5a0 tx=0x7f67f4008040 comp rx=0 tx=0).stop 2024-05-09T09:41:37.536 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 msgr2=0x7f68041a6990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 0x7f68041a6990 secure :-1 s=READY pgs=1092 cs=0 l=1 rev1=1 crypto rx=0x7f67ec005fd0 tx=0x7f67ec00dde0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 shutdown_connections 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67f00a2e70 0x7f67f00a5330 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f680410db50 0x7f68041adef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f680410b1a0 0x7f68041a6ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 --2- 172.21.15.69:0/1340527473 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6804108830 0x7f68041a6990 unknown :-1 s=CLOSED pgs=1092 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:37.537 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 >> 172.21.15.69:0/1340527473 conn(0x7f680406e8a0 msgr2=0x7f680410a990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 shutdown_connections 2024-05-09T09:41:37.538 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:37.534+0000 7f680ad61640 1 -- 172.21.15.69:0/1340527473 wait complete. 2024-05-09T09:41:37.841 INFO:tasks.workunit.client.0.smithi069.stdout: 10 296197 27.10 MB/sec execute 455 sec latency 590.646 ms 2024-05-09T09:41:38.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 296737 27.10 MB/sec execute 456 sec latency 165.827 ms 2024-05-09T09:41:39.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:38 smithi069 ceph-mon[30123]: pgmap v1052: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.7 MiB/s wr, 644 op/s 2024-05-09T09:41:39.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:38 smithi163 ceph-mon[34354]: pgmap v1052: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.7 MiB/s wr, 644 op/s 2024-05-09T09:41:39.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:38 smithi103 ceph-mon[34421]: pgmap v1052: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.5 MiB/s rd, 7.7 MiB/s wr, 644 op/s 2024-05-09T09:41:39.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 297428 27.13 MB/sec execute 457 sec latency 52.775 ms 2024-05-09T09:41:40.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 298065 27.16 MB/sec execute 458 sec latency 48.323 ms 2024-05-09T09:41:41.262 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:40 smithi069 ceph-mon[30123]: pgmap v1053: 129 pgs: 129 active+clean; 705 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.3 MiB/s wr, 482 op/s 2024-05-09T09:41:41.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:40 smithi163 ceph-mon[34354]: pgmap v1053: 129 pgs: 129 active+clean; 705 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.3 MiB/s wr, 482 op/s 2024-05-09T09:41:41.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:40 smithi103 ceph-mon[34421]: pgmap v1053: 129 pgs: 129 active+clean; 705 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.3 MiB/s wr, 482 op/s 2024-05-09T09:41:41.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 298392 27.14 MB/sec execute 459 sec latency 303.940 ms 2024-05-09T09:41:42.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 298666 27.11 MB/sec execute 460 sec latency 214.434 ms 2024-05-09T09:41:43.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:42 smithi069 ceph-mon[30123]: pgmap v1054: 129 pgs: 129 active+clean; 704 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-05-09T09:41:43.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:43 smithi163 ceph-mon[34354]: pgmap v1054: 129 pgs: 129 active+clean; 704 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-05-09T09:41:43.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:43 smithi103 ceph-mon[34421]: pgmap v1054: 129 pgs: 129 active+clean; 704 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 7.8 MiB/s wr, 647 op/s 2024-05-09T09:41:43.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 299129 27.10 MB/sec execute 461 sec latency 121.238 ms 2024-05-09T09:41:44.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 299662 27.09 MB/sec execute 462 sec latency 63.690 ms 2024-05-09T09:41:45.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:45 smithi069 ceph-mon[30123]: pgmap v1055: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-05-09T09:41:45.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:45 smithi163 ceph-mon[34354]: pgmap v1055: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-05-09T09:41:45.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:45 smithi103 ceph-mon[34421]: pgmap v1055: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 6.5 MiB/s wr, 601 op/s 2024-05-09T09:41:45.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 300194 27.09 MB/sec execute 463 sec latency 76.983 ms 2024-05-09T09:41:46.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 300606 27.08 MB/sec execute 464 sec latency 468.150 ms 2024-05-09T09:41:47.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:47 smithi069 ceph-mon[30123]: pgmap v1056: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 542 op/s 2024-05-09T09:41:47.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:47 smithi163 ceph-mon[34354]: pgmap v1056: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 542 op/s 2024-05-09T09:41:47.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:47 smithi103 ceph-mon[34421]: pgmap v1056: 129 pgs: 129 active+clean; 706 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 542 op/s 2024-05-09T09:41:47.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 300969 27.07 MB/sec execute 465 sec latency 121.560 ms 2024-05-09T09:41:48.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 301541 27.09 MB/sec execute 466 sec latency 64.284 ms 2024-05-09T09:41:49.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:49 smithi069 ceph-mon[30123]: pgmap v1057: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 691 op/s 2024-05-09T09:41:49.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:49 smithi163 ceph-mon[34354]: pgmap v1057: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 691 op/s 2024-05-09T09:41:49.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:49 smithi103 ceph-mon[34421]: pgmap v1057: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.3 MiB/s rd, 7.4 MiB/s wr, 691 op/s 2024-05-09T09:41:49.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 302146 27.09 MB/sec execute 467 sec latency 58.595 ms 2024-05-09T09:41:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.659+0000 7f72ae542640 1 -- 172.21.15.69:0/2328874605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81019a0 msgr2=0x7f72a8101e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.659+0000 7f72ae542640 1 --2- 172.21.15.69:0/2328874605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81019a0 0x7f72a8101e20 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 crypto rx=0x7f729c009900 tx=0x7f729c02a410 comp rx=0 tx=0).stop 2024-05-09T09:41:50.660 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 -- 172.21.15.69:0/2328874605 shutdown_connections 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 --2- 172.21.15.69:0/2328874605 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72a8102360 0x7f72a8108bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 --2- 172.21.15.69:0/2328874605 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81019a0 0x7f72a8101e20 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 --2- 172.21.15.69:0/2328874605 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81007b0 0x7f72a8100bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 -- 172.21.15.69:0/2328874605 >> 172.21.15.69:0/2328874605 conn(0x7f72a80fbd20 msgr2=0x7f72a80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 -- 172.21.15.69:0/2328874605 shutdown_connections 2024-05-09T09:41:50.661 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 -- 172.21.15.69:0/2328874605 wait complete. 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 Processor -- start 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.660+0000 7f72ae542640 1 -- start start 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81019a0 0x7f72a819a240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72a8102360 0x7f72a81a1270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810c430 con 0x7f72a81007b0 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810c2b0 con 0x7f72a81019a0 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72ae542640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72a810c5b0 con 0x7f72a8102360 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40884/0 (socket says 172.21.15.69:40884) 2024-05-09T09:41:50.663 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 -- 172.21.15.69:0/4286462787 learned_addr learned my addr 172.21.15.69:0/4286462787 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a77fe640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81019a0 0x7f72a819a240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 -- 172.21.15.69:0/4286462787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72a8102360 msgr2=0x7f72a81a1270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72a8102360 0x7f72a81a1270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 -- 172.21.15.69:0/4286462787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81019a0 msgr2=0x7f72a819a240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81019a0 0x7f72a819a240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 -- 172.21.15.69:0/4286462787 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f72a81a1910 con 0x7f72a81007b0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a7fff640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9c0 tx=0x7f729800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.661+0000 7f72a57fa640 1 -- 172.21.15.69:0/4286462787 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729800cd80 con 0x7f72a81007b0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.662+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f72a81a1c00 con 0x7f72a81007b0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.662+0000 7f72a57fa640 1 -- 172.21.15.69:0/4286462787 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f72980035f0 con 0x7f72a81007b0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.662+0000 7f72a57fa640 1 -- 172.21.15.69:0/4286462787 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f7298010690 con 0x7f72a81007b0 2024-05-09T09:41:50.664 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.662+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f72a81a21a0 con 0x7f72a81007b0 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.664+0000 7f72a57fa640 1 -- 172.21.15.69:0/4286462787 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f72980037a0 con 0x7f72a81007b0 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 Processor -- start 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 -- start start 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e890b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e890e240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5576e886a340 con 0x5576e8906250 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5576e8812290 con 0x5576e890bdb0 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72b124d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5576e8866300 con 0x5576e8906c40 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72acadf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72acadf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40892/0 (socket says 172.21.15.69:40892) 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72acadf640 1 -- 172.21.15.69:0/1567588044 learned_addr learned my addr 172.21.15.69:0/1567588044 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:50.667 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e890e240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e890b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.665+0000 7f72a57fa640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 0x7f72900a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 -- 172.21.15.69:0/1567588044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 msgr2=0x5576e890b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e890b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 -- 172.21.15.69:0/1567588044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 msgr2=0x5576e8906670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 -- 172.21.15.69:0/1567588044 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576e88368e0 con 0x5576e890bdb0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72acadf640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72acadf640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 0x7f72900a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a57fa640 1 -- 172.21.15.69:0/4286462787 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f72980cf930 con 0x7f72a81007b0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e890e240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f729c031440 tx=0x7f729c005e00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e172c33b65175971 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f728e7fc640 1 -- 172.21.15.69:0/1567588044 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c040070 con 0x5576e890bdb0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f72acadf640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 0x7f72900a5170 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x5576e89074a0 tx=0x7f72a0004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.668 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f728e7fc640 1 -- 172.21.15.69:0/1567588044 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004030 con 0x5576e890bdb0 2024-05-09T09:41:50.669 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.666+0000 7f728e7fc640 1 -- 172.21.15.69:0/1567588044 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c035040 con 0x5576e890bdb0 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- 172.21.15.69:0/1567588044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 msgr2=0x5576e890e240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e890e240 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f729c031440 tx=0x7f729c005e00 comp rx=0 tx=0).stop 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- 172.21.15.69:0/1567588044 shutdown_connections 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e890e240 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e890b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 --2- 172.21.15.69:0/1567588044 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e8906670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.672 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- 172.21.15.69:0/1567588044 >> 172.21.15.69:0/1567588044 conn(0x5576e89018c0 msgr2=0x5576e8901cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- 172.21.15.69:0/1567588044 shutdown_connections 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- 172.21.15.69:0/1567588044 wait complete. 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.667+0000 7f72b124d740 1 -- start start 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 Processor -- start 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8910920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8917960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5576e890ed00 con 0x5576e8906250 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40898/0 (socket says 172.21.15.69:40898) 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 -- 172.21.15.69:0/1169286120 learned_addr learned my addr 172.21.15.69:0/1169286120 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5576e8825790 con 0x5576e890bdb0 2024-05-09T09:41:50.673 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5576e890ee80 con 0x5576e8906c40 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8910920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72b124d740 10 client.0 _ll_register_callbacks cb 0x5576e8860a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72acadf640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8917960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 -- 172.21.15.69:0/1169286120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 msgr2=0x5576e8910920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8910920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 -- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 msgr2=0x5576e8917960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8917960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a7fff640 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576e857e210 con 0x5576e8906250 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.668+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8910920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f729800c6a0 tx=0x7f729809b970 comp rx=0 tx=0).ready entity=mon.0 client_cookie=668f3d1c71aef12f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729809b400 con 0x5576e8906250 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f7298097a70 con 0x5576e8906250 2024-05-09T09:41:50.674 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f729c004030 con 0x5576e8906250 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f729809bdb0 con 0x5576e8906250 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 10 client.17823 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5576e864a620 con 0x5576e8906250 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004030 con 0x5576e8906250 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f7298063080 con 0x5576e8906250 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 10 client.17823 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 10 client.17823 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 20 client.17823 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e891e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 4 client.17823 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:41:50.675 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.669+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x5576e8877a70 con 0x5576e891bf30 2024-05-09T09:41:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.670+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e891e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.670+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e891e3c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fe1d5083f4b0a8f3 server_cookie=85349c3ced2ab4d0 in_seq=0 out_seq=0 2024-05-09T09:41:50.676 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.670+0000 7f72877fe640 10 client.17823 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72877fe640 1 -- 172.21.15.69:0/1169286120 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x5576e8877a70 con 0x5576e891bf30 2024-05-09T09:41:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72877fe640 10 client.17823 handle_command_reply: tid=0 2024-05-09T09:41:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 client.17823 shutdown 2024-05-09T09:41:50.679 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 msgr2=0x5576e89103e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 crypto rx=0x7f729800c6a0 tx=0x7f729809b970 comp rx=0 tx=0).stop 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 shutdown_connections 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8917960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e891e3c0 crc :-1 s=READY pgs=262 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8910920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72877fe640 0 client.17823 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 --2- 172.21.15.69:0/1169286120 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89103e0 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 >> 172.21.15.69:0/1169286120 conn(0x5576e89018c0 msgr2=0x5576e8900a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 shutdown_connections 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.678+0000 7f72b124d740 1 -- 172.21.15.69:0/1169286120 wait complete. 2024-05-09T09:41:50.680 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.679+0000 7f72b124d740 20 client.17823 trim_cache size 0 max 16384 2024-05-09T09:41:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.697+0000 7f72b124d740 1 Processor -- start 2024-05-09T09:41:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 -- start start 2024-05-09T09:41:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e8906250 0x5576e8904f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.700 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e890bdb0 0x5576e8924210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81aedc0 con 0x5576e8906c40 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81ac690 con 0x5576e8906250 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72b124d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81aa4e0 con 0x5576e890bdb0 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40902/0 (socket says 172.21.15.69:40902) 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 -- 172.21.15.69:0/373585532 learned_addr learned my addr 172.21.15.69:0/373585532 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a7fff640 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e8906250 0x5576e8904f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72acadf640 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e890bdb0 0x5576e8924210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 -- 172.21.15.69:0/373585532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e890bdb0 msgr2=0x5576e8924210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e890bdb0 0x5576e8924210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 -- 172.21.15.69:0/373585532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e8906250 msgr2=0x5576e8904f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e8906250 0x5576e8904f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 -- 172.21.15.69:0/373585532 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5576e88fff00 con 0x5576e8906c40 2024-05-09T09:41:50.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.698+0000 7f72a77fe640 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f729c009900 tx=0x7f729c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=116ca452a17e0adf server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f728dffb640 1 -- 172.21.15.69:0/373585532 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c035040 con 0x5576e8906c40 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f728dffb640 1 -- 172.21.15.69:0/373585532 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004040 con 0x5576e8906c40 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f728dffb640 1 -- 172.21.15.69:0/373585532 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f729c040040 con 0x5576e8906c40 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 -- 172.21.15.69:0/373585532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 msgr2=0x5576e8923cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 crypto rx=0x7f729c009900 tx=0x7f729c000bc0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 -- 172.21.15.69:0/373585532 shutdown_connections 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e890bdb0 0x5576e8924210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906c40 0x5576e8923cd0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 --2- 172.21.15.69:0/373585532 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e8906250 0x5576e8904f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 -- 172.21.15.69:0/373585532 >> 172.21.15.69:0/373585532 conn(0x5576e89018c0 msgr2=0x5576e8904870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.699+0000 7f72b124d740 1 -- 172.21.15.69:0/373585532 shutdown_connections 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 -- 172.21.15.69:0/373585532 wait complete. 2024-05-09T09:41:50.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 -- start start 2024-05-09T09:41:50.703 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 Processor -- start 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8915cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81aab50 con 0x5576e8906250 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81abfc0 con 0x5576e890bdb0 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f72a81ab220 con 0x5576e8906c40 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72b124d740 10 client.0 _ll_register_callbacks cb 0x7f729809bbe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72a77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72acadf640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8915cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72a77fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:42966/0 (socket says 172.21.15.69:42966) 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72a7fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40912/0 (socket says 172.21.15.69:40912) 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.700+0000 7f72a77fe640 1 -- 172.21.15.69:0/1638642237 learned_addr learned my addr 172.21.15.69:0/1638642237 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 -- 172.21.15.69:0/1638642237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 msgr2=0x5576e8914ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 -- 172.21.15.69:0/1638642237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 msgr2=0x5576e89147a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:50.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 -- 172.21.15.69:0/1638642237 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5576e89162e0 con 0x5576e890bdb0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72a77fe640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72acadf640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8915cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009420 tx=0x7f72a0009500 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d09f6a38e17345f6 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72877fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f72a002d060 con 0x5576e890bdb0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f729c004040 con 0x5576e890bdb0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f72a001d470 con 0x5576e890bdb0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f72a0012650 con 0x5576e890bdb0 2024-05-09T09:41:50.707 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72b124d740 10 client.27080 fetch_fsmap learned FSMap version 47 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.701+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x5576e898b050 con 0x5576e890bdb0 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f729c004040 con 0x5576e890bdb0 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f72a0027030 con 0x5576e890bdb0 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 10 client.27080 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 10 client.27080 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 20 client.27080 populate_metadata read hostname 'smithi069' 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e898da70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 4 client.27080 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5576e8871dc0 con 0x5576e891bf30 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e898da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72877fe640 10 client.27080 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:50.708 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:50.702+0000 7f72a7fff640 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e898da70 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d7cdc6f3ae4dcc62 server_cookie=955e2f525f7450c in_seq=0 out_seq=0 2024-05-09T09:41:50.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 302691 27.11 MB/sec execute 468 sec latency 71.927 ms 2024-05-09T09:41:51.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:51 smithi163 ceph-mon[34354]: pgmap v1058: 129 pgs: 129 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-05-09T09:41:51.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:51 smithi103 ceph-mon[34421]: pgmap v1058: 129 pgs: 129 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-05-09T09:41:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:51 smithi069 ceph-mon[30123]: pgmap v1058: 129 pgs: 129 active+clean; 692 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 6.3 MiB/s wr, 579 op/s 2024-05-09T09:41:51.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 302791 27.07 MB/sec execute 469 sec latency 958.116 ms 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.842+0000 7f72877fe640 1 -- 172.21.15.69:0/1638642237 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5576e8871dc0 con 0x5576e891bf30 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.842+0000 7f72877fe640 10 client.27080 handle_command_reply: tid=0 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.842+0000 7f72b124d740 1 client.27080 shutdown 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.842+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 msgr2=0x5576e8915cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8915cf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f72a0009420 tx=0x7f72a0009500 comp rx=0 tx=0).stop 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 shutdown_connections 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5576e890bdb0 0x5576e8915cf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.844 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5576e891bf30 0x5576e898da70 crc :-1 s=READY pgs=263 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5576e8906c40 0x5576e8914ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72877fe640 0 client.27080 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 --2- 172.21.15.69:0/1638642237 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5576e8906250 0x5576e89147a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 >> 172.21.15.69:0/1638642237 conn(0x5576e89018c0 msgr2=0x5576e8904870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 shutdown_connections 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.843+0000 7f72b124d740 1 -- 172.21.15.69:0/1638642237 wait complete. 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.844+0000 7f72b124d740 20 client.27080 trim_cache size 0 max 16384 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:41:51.845 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:41:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:41:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.844+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 msgr2=0x7f72900a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.844+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 0x7f72900a5170 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x5576e89074a0 tx=0x7f72a0004a40 comp rx=0 tx=0).stop 2024-05-09T09:41:51.846 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.844+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 msgr2=0x7f72a8199d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.844+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 crypto rx=0x7f729800e9c0 tx=0x7f729800ee80 comp rx=0 tx=0).stop 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 shutdown_connections 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f72900a2cb0 0x7f72900a5170 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f72a8102360 0x7f72a81a1270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f72a81019a0 0x7f72a819a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 --2- 172.21.15.69:0/4286462787 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f72a81007b0 0x7f72a8199d00 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 >> 172.21.15.69:0/4286462787 conn(0x7f72a80fbd20 msgr2=0x7f72a80fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 shutdown_connections 2024-05-09T09:41:51.847 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:41:51.845+0000 7f72ae542640 1 -- 172.21.15.69:0/4286462787 wait complete. 2024-05-09T09:41:52.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 303081 27.04 MB/sec execute 470 sec latency 714.091 ms 2024-05-09T09:41:53.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:53 smithi163 ceph-mon[34354]: pgmap v1059: 129 pgs: 129 active+clean; 693 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.0 MiB/s wr, 540 op/s 2024-05-09T09:41:53.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:53 smithi103 ceph-mon[34421]: pgmap v1059: 129 pgs: 129 active+clean; 693 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.0 MiB/s wr, 540 op/s 2024-05-09T09:41:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:53 smithi069 ceph-mon[30123]: pgmap v1059: 129 pgs: 129 active+clean; 693 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 6.0 MiB/s wr, 540 op/s 2024-05-09T09:41:53.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 303573 27.03 MB/sec execute 471 sec latency 117.677 ms 2024-05-09T09:41:54.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 304196 27.04 MB/sec execute 472 sec latency 93.964 ms 2024-05-09T09:41:55.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:55 smithi163 ceph-mon[34354]: pgmap v1060: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 564 op/s 2024-05-09T09:41:55.311 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:55 smithi069 ceph-mon[30123]: pgmap v1060: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 564 op/s 2024-05-09T09:41:55.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:55 smithi103 ceph-mon[34421]: pgmap v1060: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.0 MiB/s wr, 564 op/s 2024-05-09T09:41:55.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 304900 27.07 MB/sec execute 473 sec latency 43.576 ms 2024-05-09T09:41:56.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 305347 27.05 MB/sec execute 474 sec latency 290.827 ms 2024-05-09T09:41:57.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:57 smithi163 ceph-mon[34354]: pgmap v1061: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 513 op/s 2024-05-09T09:41:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:57 smithi103 ceph-mon[34421]: pgmap v1061: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 513 op/s 2024-05-09T09:41:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:57 smithi069 ceph-mon[30123]: pgmap v1061: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.9 MiB/s wr, 513 op/s 2024-05-09T09:41:57.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 305674 27.04 MB/sec execute 475 sec latency 111.920 ms 2024-05-09T09:41:58.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 306246 27.06 MB/sec execute 476 sec latency 87.844 ms 2024-05-09T09:41:59.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:41:59 smithi163 ceph-mon[34354]: pgmap v1062: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 686 op/s 2024-05-09T09:41:59.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:41:59 smithi103 ceph-mon[34421]: pgmap v1062: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 686 op/s 2024-05-09T09:41:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:41:59 smithi069 ceph-mon[30123]: pgmap v1062: 129 pgs: 129 active+clean; 696 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 7.4 MiB/s wr, 686 op/s 2024-05-09T09:41:59.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 306774 27.05 MB/sec execute 477 sec latency 53.799 ms 2024-05-09T09:42:00.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 307270 27.05 MB/sec execute 478 sec latency 55.772 ms 2024-05-09T09:42:01.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:01 smithi163 ceph-mon[34354]: pgmap v1063: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:42:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:01 smithi103 ceph-mon[34421]: pgmap v1063: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:42:01.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:01 smithi069 ceph-mon[30123]: pgmap v1063: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.4 MiB/s wr, 579 op/s 2024-05-09T09:42:01.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 307738 27.05 MB/sec execute 479 sec latency 285.868 ms 2024-05-09T09:42:02.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 308017 27.02 MB/sec execute 480 sec latency 427.608 ms 2024-05-09T09:42:03.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:03 smithi163 ceph-mon[34354]: pgmap v1064: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.4 MiB/s wr, 589 op/s 2024-05-09T09:42:03.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:03 smithi103 ceph-mon[34421]: pgmap v1064: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.4 MiB/s wr, 589 op/s 2024-05-09T09:42:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:03 smithi069 ceph-mon[30123]: pgmap v1064: 129 pgs: 129 active+clean; 688 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 6.4 MiB/s wr, 589 op/s 2024-05-09T09:42:03.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 308522 27.01 MB/sec execute 481 sec latency 89.661 ms 2024-05-09T09:42:04.842 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309194 27.05 MB/sec execute 482 sec latency 50.792 ms 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- 172.21.15.69:0/1494652487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c4104cc0 msgr2=0x7f67c41070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1494652487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c4104cc0 0x7f67c41070b0 secure :-1 s=READY pgs=1099 cs=0 l=1 rev1=1 crypto rx=0x7f67b4009900 tx=0x7f67b402a410 comp rx=0 tx=0).stop 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- 172.21.15.69:0/1494652487 shutdown_connections 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1494652487 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c41075f0 0x7f67c41099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1494652487 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c4104cc0 0x7f67c41070b0 unknown :-1 s=CLOSED pgs=1099 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1494652487 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4102390 0x7f67c4104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- 172.21.15.69:0/1494652487 >> 172.21.15.69:0/1494652487 conn(0x7f67c40fbd20 msgr2=0x7f67c40fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- 172.21.15.69:0/1494652487 shutdown_connections 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- 172.21.15.69:0/1494652487 wait complete. 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 Processor -- start 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.959+0000 7f67cc04d640 1 -- start start 2024-05-09T09:42:04.961 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 0x7f67c4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 0x7f67c4198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 0x7f67c419f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d500 con 0x7f67c41075f0 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d380 con 0x7f67c4104cc0 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67c410d680 con 0x7f67c4102390 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 0x7f67c4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 0x7f67c419f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 0x7f67c4198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 0x7f67c4197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:50096/0 (socket says 172.21.15.69:50096) 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67c9de9640 1 -- 172.21.15.69:0/1498050594 learned_addr learned my addr 172.21.15.69:0/1498050594 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 -- 172.21.15.69:0/1498050594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 msgr2=0x7f67c4197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 0x7f67c4197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 -- 172.21.15.69:0/1498050594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 msgr2=0x7f67c4198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 0x7f67c4198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.962 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 -- 172.21.15.69:0/1498050594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f67c419f740 con 0x7f67c41075f0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 0x7f67c4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67ca5ea640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 0x7f67c419f0d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00da10 tx=0x7f67bc00ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67baffd640 1 -- 172.21.15.69:0/1498050594 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc00bd70 con 0x7f67c41075f0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67baffd640 1 -- 172.21.15.69:0/1498050594 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67bc003e50 con 0x7f67c41075f0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.960+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f67c419fa30 con 0x7f67c41075f0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.961+0000 7f67baffd640 1 -- 172.21.15.69:0/1498050594 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67bc0106a0 con 0x7f67c41075f0 2024-05-09T09:42:04.963 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.961+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f67c419ff00 con 0x7f67c41075f0 2024-05-09T09:42:04.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.963+0000 7f67cee39740 1 Processor -- start 2024-05-09T09:42:04.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67baffd640 1 -- 172.21.15.69:0/1498050594 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f67bc003970 con 0x7f67c41075f0 2024-05-09T09:42:04.966 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 -- start start 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f231a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f236b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56037f191340 con 0x56037f2371c0 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56037f139290 con 0x56037f231660 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67cee39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56037f18d300 con 0x56037f232050 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35144/0 (socket says 172.21.15.69:35144) 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f236b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67c95e8640 1 -- 172.21.15.69:0/773956330 learned_addr learned my addr 172.21.15.69:0/773956330 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.964+0000 7f67c95e8640 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f231a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67baffd640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 0x7f67b00a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 -- 172.21.15.69:0/773956330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 msgr2=0x56037f231a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:04.967 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f231a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 -- 172.21.15.69:0/773956330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 msgr2=0x56037f239650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 -- 172.21.15.69:0/773956330 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56037f15d8e0 con 0x56037f232050 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c95e8640 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67baffd640 1 -- 172.21.15.69:0/1498050594 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67bc0cf7a0 con 0x7f67c41075f0 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67ca5ea640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 0x7f67b00a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67c9de9640 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f236b50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009c80 tx=0x7f67c0011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a16648bf7ecdb89a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67affff640 1 -- 172.21.15.69:0/773956330 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0004230 con 0x56037f232050 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67ca5ea640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 0x7f67b00a50f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0ccf20 tx=0x7f67bc00bf10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67affff640 1 -- 172.21.15.69:0/773956330 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c00043d0 con 0x56037f232050 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.965+0000 7f67affff640 1 -- 172.21.15.69:0/773956330 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0004cd0 con 0x56037f232050 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 -- 172.21.15.69:0/773956330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 msgr2=0x56037f236b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f236b50 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f67c0009c80 tx=0x7f67c0011810 comp rx=0 tx=0).stop 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 -- 172.21.15.69:0/773956330 shutdown_connections 2024-05-09T09:42:04.968 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f2371c0 0x56037f239650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f236b50 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 --2- 172.21.15.69:0/773956330 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f231a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 -- 172.21.15.69:0/773956330 >> 172.21.15.69:0/773956330 conn(0x56037f2288c0 msgr2=0x56037f228cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.966+0000 7f67cee39740 1 -- 172.21.15.69:0/773956330 shutdown_connections 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.967+0000 7f67cee39740 1 -- 172.21.15.69:0/773956330 wait complete. 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.967+0000 7f67cee39740 1 -- start start 2024-05-09T09:42:04.969 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.967+0000 7f67cee39740 1 Processor -- start 2024-05-09T09:42:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.967+0000 7f67cee39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23b830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.972 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f242db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x56037f23a0e0 con 0x56037f232050 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x56037f14c790 con 0x56037f231660 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x56037f23a260 con 0x56037f2371c0 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67cee39740 10 client.0 _ll_register_callbacks cb 0x56037f187a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35146/0 (socket says 172.21.15.69:35146) 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 -- 172.21.15.69:0/3400164167 learned_addr learned my addr 172.21.15.69:0/3400164167 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 -- 172.21.15.69:0/3400164167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 msgr2=0x56037f242db0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f242db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 -- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 msgr2=0x56037f23b830 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23b830 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.973 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.968+0000 7f67c9de9640 1 -- 172.21.15.69:0/3400164167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56037eea5210 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67c9de9640 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f67c0011d10 tx=0x7f67c0019ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ed7e17edacb538e5 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c0028020 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67c0024620 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67c00043d0 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67c0033050 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67cee39740 10 client.17841 fetch_fsmap learned FSMap version 47 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.969+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56037ef71620 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67c00043d0 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67c0018070 con 0x56037f232050 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 10 client.17841 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 10 client.17841 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 20 client.17841 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:04.974 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f246eb0 0x56037f2492a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 4 client.17841 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x56037f19ea70 con 0x56037f246eb0 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.970+0000 7f67c95e8640 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f246eb0 0x56037f2492a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.971+0000 7f67ad7fa640 10 client.17841 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.971+0000 7f67c95e8640 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f246eb0 0x56037f2492a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=668cb07622f9158b server_cookie=72d6a3b407d22c12 in_seq=0 out_seq=0 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67ad7fa640 1 -- 172.21.15.69:0/3400164167 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x56037f19ea70 con 0x56037f246eb0 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67ad7fa640 10 client.17841 handle_command_reply: tid=0 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 client.17841 shutdown 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 msgr2=0x56037f23bd70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 secure :-1 s=READY pgs=1101 cs=0 l=1 rev1=1 crypto rx=0x7f67c0011d10 tx=0x7f67c0019ad0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 shutdown_connections 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f246eb0 0x56037f2492a0 crc :-1 s=READY pgs=264 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.975 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f242db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f23bd70 unknown :-1 s=CLOSED pgs=1101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 --2- 172.21.15.69:0/3400164167 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23b830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 >> 172.21.15.69:0/3400164167 conn(0x56037f2288c0 msgr2=0x56037f227a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67ad7fa640 0 client.17841 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.972+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 shutdown_connections 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.973+0000 7f67cee39740 1 -- 172.21.15.69:0/3400164167 wait complete. 2024-05-09T09:42:04.976 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.973+0000 7f67cee39740 20 client.17841 trim_cache size 0 max 16384 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 Processor -- start 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 -- start start 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f242d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f2371c0 0x56037f2276b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41aab00 con 0x56037f231660 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a83d0 con 0x56037f2371c0 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67cee39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a6220 con 0x56037f232050 2024-05-09T09:42:04.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35154/0 (socket says 172.21.15.69:35154) 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67c95e8640 1 -- 172.21.15.69:0/1787472208 learned_addr learned my addr 172.21.15.69:0/1787472208 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.998+0000 7f67c9de9640 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f242d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 -- 172.21.15.69:0/1787472208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 msgr2=0x56037f242d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f242d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 -- 172.21.15.69:0/1787472208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f2371c0 msgr2=0x56037f2276b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f2371c0 0x56037f2276b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 -- 172.21.15.69:0/1787472208 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x56037f232430 con 0x56037f231660 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f67b4005ab0 tx=0x7f67b4009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9377bf99e7ab9353 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67af7fe640 1 -- 172.21.15.69:0/1787472208 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4000e60 con 0x56037f231660 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67af7fe640 1 -- 172.21.15.69:0/1787472208 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4004480 con 0x56037f231660 2024-05-09T09:42:05.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:04.999+0000 7f67af7fe640 1 -- 172.21.15.69:0/1787472208 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67b4035040 con 0x56037f231660 2024-05-09T09:42:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 -- 172.21.15.69:0/1787472208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 msgr2=0x56037f227170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 crypto rx=0x7f67b4005ab0 tx=0x7f67b4009680 comp rx=0 tx=0).stop 2024-05-09T09:42:05.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 -- 172.21.15.69:0/1787472208 shutdown_connections 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f2371c0 0x56037f2276b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f232050 0x56037f242d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 --2- 172.21.15.69:0/1787472208 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f231660 0x56037f227170 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 -- 172.21.15.69:0/1787472208 >> 172.21.15.69:0/1787472208 conn(0x56037f2288c0 msgr2=0x56037f233230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 -- 172.21.15.69:0/1787472208 shutdown_connections 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.000+0000 7f67cee39740 1 -- 172.21.15.69:0/1787472208 wait complete. 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 -- start start 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 Processor -- start 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f2ab460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a6890 con 0x56037f232050 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a7d00 con 0x56037f231660 2024-05-09T09:42:05.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f67c41a6f60 con 0x56037f2371c0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67cee39740 10 client.0 _ll_register_callbacks cb 0x7f67b4009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67c9de9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:35160/0 (socket says 172.21.15.69:35160) 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67c95e8640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.001+0000 7f67c9de9640 1 -- 172.21.15.69:0/1900312850 learned_addr learned my addr 172.21.15.69:0/1900312850 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f2ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 -- 172.21.15.69:0/1900312850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 msgr2=0x56037f2ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f2ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 -- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 msgr2=0x56037f23fb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23fb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 -- 172.21.15.69:0/1900312850 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x56037f2ab9a0 con 0x56037f232050 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67c9de9640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x56037f23cd10 tx=0x7f67c00051d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e0cb23d0ee2134a server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67ad7fa640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:05.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f67c003c060 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f67b4004480 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f67c0004a90 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=47) ==== 24+0+0 (secure 0 0 0) 0x7f67c0043050 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67cee39740 10 client.17847 fetch_fsmap learned FSMap version 47 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.002+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=47,monmap=4+}) -- 0x56037f2acb10 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f67b4004480 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 47) ==== 8346+0+0 (secure 0 0 0) 0x7f67c0018070 con 0x56037f232050 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 10 client.17847 fetch_fsmap finished waiting for FSMap version 47 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 10 client.17847 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 20 client.17847 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f2ad4f0 0x56037f2af910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 4 client.17847 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:42:05.007 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x56037f198dc0 con 0x56037f2ad4f0 2024-05-09T09:42:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f2ad4f0 0x56037f2af910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67ad7fa640 10 client.17847 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:05.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.003+0000 7f67c95e8640 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f2ad4f0 0x56037f2af910 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed922e44f9f274c5 server_cookie=5c2aa86a7884936c in_seq=0 out_seq=0 2024-05-09T09:42:05.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:05 smithi163 ceph-mon[34354]: pgmap v1065: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.4 MiB/s wr, 711 op/s 2024-05-09T09:42:05.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:05 smithi103 ceph-mon[34421]: pgmap v1065: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.4 MiB/s wr, 711 op/s 2024-05-09T09:42:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:05 smithi069 ceph-mon[30123]: pgmap v1065: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 8.7 MiB/s rd, 8.4 MiB/s wr, 711 op/s 2024-05-09T09:42:05.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309335 27.01 MB/sec execute 483 sec latency 612.395 ms 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.875+0000 7f67ad7fa640 1 -- 172.21.15.69:0/1900312850 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x56037f198dc0 con 0x56037f2ad4f0 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.875+0000 7f67ad7fa640 10 client.17847 handle_command_reply: tid=0 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.875+0000 7f67cee39740 1 client.17847 shutdown 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 msgr2=0x56037f2400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 crypto rx=0x56037f23cd10 tx=0x7f67c00051d0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 shutdown_connections 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x56037f2ad4f0 0x56037f2af910 crc :-1 s=READY pgs=265 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x56037f2371c0 0x56037f2ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67ad7fa640 0 client.17847 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x56037f232050 0x56037f2400a0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.876 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 --2- 172.21.15.69:0/1900312850 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x56037f231660 0x56037f23fb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 >> 172.21.15.69:0/1900312850 conn(0x56037f2288c0 msgr2=0x56037f22b640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 shutdown_connections 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 1 -- 172.21.15.69:0/1900312850 wait complete. 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cee39740 20 client.17847 trim_cache size 0 max 16384 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:42:05.877 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:42:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.876+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 msgr2=0x7f67b00a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 0x7f67b00a50f0 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f67bc0ccf20 tx=0x7f67bc00bf10 comp rx=0 tx=0).stop 2024-05-09T09:42:05.878 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 msgr2=0x7f67c419f0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 0x7f67c419f0d0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 crypto rx=0x7f67bc00da10 tx=0x7f67bc00ded0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 shutdown_connections 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f67b00a2c30 0x7f67b00a50f0 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f67c41075f0 0x7f67c419f0d0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f67c4104cc0 0x7f67c4198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 --2- 172.21.15.69:0/1498050594 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f67c4102390 0x7f67c4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 >> 172.21.15.69:0/1498050594 conn(0x7f67c40fbd20 msgr2=0x7f67c4102c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 shutdown_connections 2024-05-09T09:42:05.879 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:05.877+0000 7f67cc04d640 1 -- 172.21.15.69:0/1498050594 wait complete. 2024-05-09T09:42:06.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309767 26.99 MB/sec execute 484 sec latency 213.698 ms 2024-05-09T09:42:07.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:07 smithi163 ceph-mon[34354]: pgmap v1066: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.6 MiB/s wr, 500 op/s 2024-05-09T09:42:07.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:07 smithi103 ceph-mon[34421]: pgmap v1066: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.6 MiB/s wr, 500 op/s 2024-05-09T09:42:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:07 smithi069 ceph-mon[30123]: pgmap v1066: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 6.6 MiB/s wr, 500 op/s 2024-05-09T09:42:07.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309772 26.94 MB/sec execute 485 sec latency 1213.740 ms 2024-05-09T09:42:08.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309781 26.88 MB/sec execute 486 sec latency 2174.127 ms 2024-05-09T09:42:09.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:09 smithi163 ceph-mon[34354]: pgmap v1067: 129 pgs: 129 active+clean; 680 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.1 MiB/s wr, 577 op/s 2024-05-09T09:42:09.305 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:09 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:09.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:09 smithi103 ceph-mon[34421]: pgmap v1067: 129 pgs: 129 active+clean; 680 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.1 MiB/s wr, 577 op/s 2024-05-09T09:42:09.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:09 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:09 smithi069 ceph-mon[30123]: pgmap v1067: 129 pgs: 129 active+clean; 680 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 7.1 MiB/s wr, 577 op/s 2024-05-09T09:42:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:09 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:09.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 309906 26.84 MB/sec execute 487 sec latency 2145.552 ms 2024-05-09T09:42:10.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 310103 26.81 MB/sec execute 488 sec latency 152.257 ms 2024-05-09T09:42:11.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:11 smithi103 ceph-mon[34421]: pgmap v1068: 129 pgs: 129 active+clean; 728 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 634 op/s 2024-05-09T09:42:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:11 smithi069 ceph-mon[30123]: pgmap v1068: 129 pgs: 129 active+clean; 728 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 634 op/s 2024-05-09T09:42:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:11 smithi163 ceph-mon[34354]: pgmap v1068: 129 pgs: 129 active+clean; 728 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 11 MiB/s wr, 634 op/s 2024-05-09T09:42:11.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 310363 26.78 MB/sec execute 489 sec latency 144.945 ms 2024-05-09T09:42:12.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 310546 26.75 MB/sec execute 490 sec latency 157.799 ms 2024-05-09T09:42:13.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:13 smithi103 ceph-mon[34421]: pgmap v1069: 129 pgs: 129 active+clean; 748 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-05-09T09:42:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:13 smithi069 ceph-mon[30123]: pgmap v1069: 129 pgs: 129 active+clean; 748 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-05-09T09:42:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:13 smithi163 ceph-mon[34354]: pgmap v1069: 129 pgs: 129 active+clean; 748 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 12 MiB/s wr, 664 op/s 2024-05-09T09:42:13.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 310769 26.72 MB/sec execute 491 sec latency 244.707 ms 2024-05-09T09:42:14.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:14 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:14.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:14 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:14.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:14 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:42:14.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 311175 26.70 MB/sec execute 492 sec latency 139.388 ms 2024-05-09T09:42:15.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:15 smithi103 ceph-mon[34421]: pgmap v1070: 129 pgs: 129 active+clean; 734 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 15 MiB/s wr, 814 op/s 2024-05-09T09:42:15.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:15 smithi069 ceph-mon[30123]: pgmap v1070: 129 pgs: 129 active+clean; 734 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 15 MiB/s wr, 814 op/s 2024-05-09T09:42:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:15 smithi163 ceph-mon[34354]: pgmap v1070: 129 pgs: 129 active+clean; 734 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.6 MiB/s rd, 15 MiB/s wr, 814 op/s 2024-05-09T09:42:15.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 311714 26.71 MB/sec execute 493 sec latency 115.561 ms 2024-05-09T09:42:16.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 312463 26.75 MB/sec execute 494 sec latency 80.904 ms 2024-05-09T09:42:17.357 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:17 smithi163 ceph-mon[34354]: pgmap v1071: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 MiB/s wr, 753 op/s 2024-05-09T09:42:17.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:17 smithi103 ceph-mon[34421]: pgmap v1071: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 MiB/s wr, 753 op/s 2024-05-09T09:42:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:17 smithi069 ceph-mon[30123]: pgmap v1071: 129 pgs: 129 active+clean; 695 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 3.6 MiB/s rd, 14 MiB/s wr, 753 op/s 2024-05-09T09:42:17.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 312987 26.76 MB/sec execute 495 sec latency 57.720 ms 2024-05-09T09:42:18.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 313397 26.74 MB/sec execute 496 sec latency 86.593 ms 2024-05-09T09:42:18.988 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- 172.21.15.69:0/3107412033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 msgr2=0x7f1e58104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/3107412033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58104700 secure :-1 s=READY pgs=1104 cs=0 l=1 rev1=1 crypto rx=0x7f1e48009900 tx=0x7f1e4802a410 comp rx=0 tx=0).stop 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- 172.21.15.69:0/3107412033 shutdown_connections 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/3107412033 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58107570 0x7f1e581099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/3107412033 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58104c40 0x7f1e58107030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/3107412033 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58104700 unknown :-1 s=CLOSED pgs=1104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- 172.21.15.69:0/3107412033 >> 172.21.15.69:0/3107412033 conn(0x7f1e580fbd00 msgr2=0x7f1e580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- 172.21.15.69:0/3107412033 shutdown_connections 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- 172.21.15.69:0/3107412033 wait complete. 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 Processor -- start 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- start start 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.989 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58104c40 0x7f1e58198060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58107570 0x7f1e5819f0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810d580 con 0x7f1e58102310 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810d400 con 0x7f1e58107570 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e5e34f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1e5810d700 con 0x7f1e58104c40 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.987+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60028/0 (socket says 172.21.15.69:60028) 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 -- 172.21.15.69:0/2858802937 learned_addr learned my addr 172.21.15.69:0/2858802937 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e577fe640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58104c40 0x7f1e58198060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58107570 0x7f1e5819f0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 -- 172.21.15.69:0/2858802937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58104c40 msgr2=0x7f1e58198060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58104c40 0x7f1e58198060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 -- 172.21.15.69:0/2858802937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58107570 msgr2=0x7f1e5819f0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:18.990 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58107570 0x7f1e5819f0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 -- 172.21.15.69:0/2858802937 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1e5819f740 con 0x7f1e58102310 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e57fff640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1e48031440 tx=0x7f1e48032ce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e557fa640 1 -- 172.21.15.69:0/2858802937 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4803f030 con 0x7f1e58102310 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1e5819f9d0 con 0x7f1e58102310 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e557fa640 1 -- 172.21.15.69:0/2858802937 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e48003ce0 con 0x7f1e58102310 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e557fa640 1 -- 172.21.15.69:0/2858802937 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48035070 con 0x7f1e58102310 2024-05-09T09:42:18.991 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.988+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1e5819fec0 con 0x7f1e58102310 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.991+0000 7f1e557fa640 1 -- 172.21.15.69:0/2858802937 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1e48042030 con 0x7f1e58102310 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.991+0000 7f1e61057740 1 Processor -- start 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 -- start start 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57d1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d2050 0x55fcd57d6b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.997 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fcd5731340 con 0x55fcd57d2050 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcd56d9290 con 0x55fcd57d1660 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e61057740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fcd572d300 con 0x55fcd57d71c0 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e5c8ec640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57d1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:35970/0 (socket says 172.21.15.69:35970) 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d2050 0x55fcd57d6b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e5c8ec640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57d1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37324/0 (socket says 172.21.15.69:37324) 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 -- 172.21.15.69:0/3053740467 learned_addr learned my addr 172.21.15.69:0/3053740467 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:18.998 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 -- 172.21.15.69:0/3053740467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 msgr2=0x55fcd57d1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57d1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 -- 172.21.15.69:0/3053740467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d2050 msgr2=0x55fcd57d6b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d2050 0x55fcd57d6b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e577fe640 1 -- 172.21.15.69:0/3053740467 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd56fd8e0 con 0x55fcd57d71c0 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.992+0000 7f1e557fa640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 0x7f1e400a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e577fe640 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009c60 tx=0x7f1e4c011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fc15e846143e47f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 0x7f1e400a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e557fa640 1 -- 172.21.15.69:0/2858802937 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1e480ef850 con 0x7f1e58102310 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e3e7fc640 1 -- 172.21.15.69:0/3053740467 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c004230 con 0x55fcd57d71c0 2024-05-09T09:42:18.999 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e3e7fc640 1 -- 172.21.15.69:0/3053740467 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e4c0043d0 con 0x55fcd57d71c0 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.993+0000 7f1e3e7fc640 1 -- 172.21.15.69:0/3053740467 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c004d60 con 0x55fcd57d71c0 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 -- 172.21.15.69:0/3053740467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 msgr2=0x55fcd57d9650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 0x7f1e400a50f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x55fcd57d28b0 tx=0x7f1e50009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c009c60 tx=0x7f1e4c011810 comp rx=0 tx=0).stop 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 -- 172.21.15.69:0/3053740467 shutdown_connections 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d71c0 0x55fcd57d9650 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.000 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d2050 0x55fcd57d6b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 --2- 172.21.15.69:0/3053740467 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57d1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 -- 172.21.15.69:0/3053740467 >> 172.21.15.69:0/3053740467 conn(0x55fcd57c88c0 msgr2=0x55fcd57c8cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 -- 172.21.15.69:0/3053740467 shutdown_connections 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.994+0000 7f1e61057740 1 -- 172.21.15.69:0/3053740467 wait complete. 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e61057740 1 -- start start 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e61057740 1 Processor -- start 2024-05-09T09:42:19.001 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e61057740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57db7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e61057740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57dbce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e61057740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57db7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.995+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57db7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37340/0 (socket says 172.21.15.69:37340) 2024-05-09T09:42:19.002 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e57fff640 1 -- 172.21.15.69:0/334766275 learned_addr learned my addr 172.21.15.69:0/334766275 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55fcd57da170 con 0x55fcd57d71c0 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55fcd56ec790 con 0x55fcd57d1660 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55fcd57da2f0 con 0x55fcd57d2050 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57dbce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e61057740 10 client.0 _ll_register_callbacks cb 0x55fcd5727a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:19.003 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 -- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 msgr2=0x55fcd57db7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57db7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 -- 172.21.15.69:0/334766275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 msgr2=0x55fcd57e2d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e577fe640 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd5445210 con 0x55fcd57d2050 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e4c0043d0 con 0x55fcd57d2050 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.996+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:42:19.004 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e577fe640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57dbce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c011d10 tx=0x7f1e4c0051c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca40fbe737b365b9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e377fe640 10 client.27034 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e377fe640 1 -- 172.21.15.69:0/334766275 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c028020 con 0x55fcd57d2050 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e377fe640 1 -- 172.21.15.69:0/334766275 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1e4c024420 con 0x55fcd57d2050 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e377fe640 1 -- 172.21.15.69:0/334766275 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1e4c024660 con 0x55fcd57d2050 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e61057740 10 client.27034 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.997+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({fsmap=49}) -- 0x55fcd5511620 con 0x55fcd57d2050 2024-05-09T09:42:19.005 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e377fe640 1 -- 172.21.15.69:0/334766275 <== mon.1 v2:172.21.15.163:3300/0 4 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f1e4c018070 con 0x55fcd57d2050 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 10 client.27034 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 10 client.27034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 20 client.27034 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd57e7190 0x55fcd57e9580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 4 client.27034 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55fcd573ea70 con 0x55fcd57e7190 2024-05-09T09:42:19.006 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e57fff640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd57e7190 0x55fcd57e9580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e57fff640 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd57e7190 0x55fcd57e9580 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9715c82d51bd8c2c server_cookie=648569b9d3ac2b79 in_seq=0 out_seq=0 2024-05-09T09:42:19.008 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.998+0000 7f1e377fe640 10 client.27034 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.999+0000 7f1e377fe640 1 -- 172.21.15.69:0/334766275 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55fcd573ea70 con 0x55fcd57e7190 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.999+0000 7f1e377fe640 10 client.27034 handle_command_reply: tid=0 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:18.999+0000 7f1e61057740 1 client.27034 shutdown 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 msgr2=0x55fcd57dbce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57dbce0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c011d10 tx=0x7f1e4c0051c0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 shutdown_connections 2024-05-09T09:42:19.009 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd57e7190 0x55fcd57e9580 crc :-1 s=READY pgs=266 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57dbce0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 --2- 172.21.15.69:0/334766275 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57db7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 >> 172.21.15.69:0/334766275 conn(0x55fcd57c88c0 msgr2=0x55fcd57c7a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e377fe640 0 client.27034 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:19.010 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 shutdown_connections 2024-05-09T09:42:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 1 -- 172.21.15.69:0/334766275 wait complete. 2024-05-09T09:42:19.011 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.000+0000 7f1e61057740 20 client.27034 trim_cache size 0 max 16384 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 Processor -- start 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 -- start start 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57ef0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581aaac0 con 0x55fcd57d1660 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581a8330 con 0x55fcd57d71c0 2024-05-09T09:42:19.021 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e61057740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581a6180 con 0x55fcd57d2050 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57ef0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57ef0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:35988/0 (socket says 172.21.15.69:35988) 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.020+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60066/0 (socket says 172.21.15.69:60066) 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e577fe640 1 -- 172.21.15.69:0/1080037359 learned_addr learned my addr 172.21.15.69:0/1080037359 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 -- 172.21.15.69:0/1080037359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 msgr2=0x55fcd57ef0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57ef0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 -- 172.21.15.69:0/1080037359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d71c0 msgr2=0x55fcd57e2ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 -- 172.21.15.69:0/1080037359 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55fcd57d2430 con 0x55fcd57d1660 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e57fff640 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ecfd0 tx=0x7f1e480f5550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3b8fb4f3c156b16 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:19.022 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e3dffb640 1 -- 172.21.15.69:0/1080037359 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e48049040 con 0x55fcd57d1660 2024-05-09T09:42:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e3dffb640 1 -- 172.21.15.69:0/1080037359 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e480bbde0 con 0x55fcd57d1660 2024-05-09T09:42:19.023 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.021+0000 7f1e3dffb640 1 -- 172.21.15.69:0/1080037359 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e480ba0e0 con 0x55fcd57d1660 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.022+0000 7f1e61057740 1 -- 172.21.15.69:0/1080037359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 msgr2=0x55fcd57eeb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.022+0000 7f1e61057740 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 crypto rx=0x7f1e480ecfd0 tx=0x7f1e480f5550 comp rx=0 tx=0).stop 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.022+0000 7f1e61057740 1 -- 172.21.15.69:0/1080037359 shutdown_connections 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.022+0000 7f1e61057740 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d71c0 0x55fcd57e2ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57ef0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 --2- 172.21.15.69:0/1080037359 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d1660 0x55fcd57eeb80 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- 172.21.15.69:0/1080037359 >> 172.21.15.69:0/1080037359 conn(0x55fcd57c88c0 msgr2=0x55fcd57db7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- 172.21.15.69:0/1080037359 shutdown_connections 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- 172.21.15.69:0/1080037359 wait complete. 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- start start 2024-05-09T09:42:19.027 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 Processor -- start 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57e07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581a67f0 con 0x55fcd57d71c0 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581a7c60 con 0x55fcd57d1660 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1e581a6ec0 con 0x55fcd57d2050 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e61057740 10 client.0 _ll_register_callbacks cb 0x7f1e480f5420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e5c8ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e577fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57e07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e5c8ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:60074/0 (socket says 172.21.15.69:60074) 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e5c8ec640 1 -- 172.21.15.69:0/254814242 learned_addr learned my addr 172.21.15.69:0/254814242 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:19.028 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.023+0000 7f1e57fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:37354/0 (socket says 172.21.15.69:37354) 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 -- 172.21.15.69:0/254814242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 msgr2=0x55fcd57e07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57e07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 -- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 msgr2=0x55fcd57e02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 -- 172.21.15.69:0/254814242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55fcd57e1de0 con 0x55fcd57d71c0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e57fff640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e377fe640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e5c8ec640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f1e500041b0 tx=0x7f1e50004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=60b80d0ca3e2d691 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1e50004860 con 0x55fcd57d71c0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f1e480bbde0 con 0x55fcd57d71c0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1e50004b20 con 0x55fcd57d71c0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1e50034050 con 0x55fcd57d71c0 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e61057740 10 client.17865 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:19.029 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.024+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55fcd584d3f0 con 0x55fcd57d71c0 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1e480bbde0 con 0x55fcd57d71c0 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f1e50028070 con 0x55fcd57d71c0 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 10 client.17865 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 10 client.17865 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 20 client.17865 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd584dd00 0x55fcd5850120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 4 client.17865 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55fcd5738dc0 con 0x55fcd584dd00 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e57fff640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd584dd00 0x55fcd5850120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e377fe640 10 client.17865 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:19.030 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.025+0000 7f1e57fff640 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd584dd00 0x55fcd5850120 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed5cd3c81d38556c server_cookie=39793e22a7d89d5f in_seq=0 out_seq=0 2024-05-09T09:42:19.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:19 smithi103 ceph-mon[34421]: pgmap v1072: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 817 op/s 2024-05-09T09:42:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:19 smithi069 ceph-mon[30123]: pgmap v1072: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 817 op/s 2024-05-09T09:42:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:19 smithi163 ceph-mon[34354]: pgmap v1072: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 817 op/s 2024-05-09T09:42:19.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.557+0000 7f1e377fe640 1 -- 172.21.15.69:0/254814242 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55fcd5738dc0 con 0x55fcd584dd00 2024-05-09T09:42:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.557+0000 7f1e377fe640 10 client.17865 handle_command_reply: tid=0 2024-05-09T09:42:19.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.557+0000 7f1e61057740 1 client.17865 shutdown 2024-05-09T09:42:19.558 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 msgr2=0x55fcd57e17f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 crypto rx=0x7f1e500041b0 tx=0x7f1e50004ab0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 shutdown_connections 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55fcd57d71c0 0x55fcd57e17f0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55fcd584dd00 0x55fcd5850120 crc :-1 s=READY pgs=267 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e377fe640 0 client.17865 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:19.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55fcd57d2050 0x55fcd57e07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 --2- 172.21.15.69:0/254814242 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55fcd57d1660 0x55fcd57e02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 >> 172.21.15.69:0/254814242 conn(0x55fcd57c88c0 msgr2=0x55fcd57cb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 shutdown_connections 2024-05-09T09:42:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.558+0000 7f1e61057740 1 -- 172.21.15.69:0/254814242 wait complete. 2024-05-09T09:42:19.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.559+0000 7f1e61057740 20 client.17865 trim_cache size 0 max 16384 2024-05-09T09:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:42:19.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.559+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 msgr2=0x7f1e400a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.559+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 0x7f1e400a50f0 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x55fcd57d28b0 tx=0x7f1e50009500 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.559+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 msgr2=0x7f1e58197b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.559+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 crypto rx=0x7f1e48031440 tx=0x7f1e48032ce0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 shutdown_connections 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1e400a2c30 0x7f1e400a50f0 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1e58107570 0x7f1e5819f0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1e58104c40 0x7f1e58198060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 --2- 172.21.15.69:0/2858802937 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1e58102310 0x7f1e58197b20 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:19.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 >> 172.21.15.69:0/2858802937 conn(0x7f1e580fbd00 msgr2=0x7f1e58102c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 shutdown_connections 2024-05-09T09:42:19.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:19.560+0000 7f1e5e34f640 1 -- 172.21.15.69:0/2858802937 wait complete. 2024-05-09T09:42:19.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 313671 26.72 MB/sec execute 497 sec latency 423.033 ms 2024-05-09T09:42:20.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 314278 26.71 MB/sec execute 498 sec latency 83.306 ms 2024-05-09T09:42:21.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:21 smithi103 ceph-mon[34421]: pgmap v1073: 129 pgs: 129 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 823 op/s 2024-05-09T09:42:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:21 smithi069 ceph-mon[30123]: pgmap v1073: 129 pgs: 129 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 823 op/s 2024-05-09T09:42:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:21 smithi163 ceph-mon[34354]: pgmap v1073: 129 pgs: 129 active+clean; 664 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.0 MiB/s rd, 13 MiB/s wr, 823 op/s 2024-05-09T09:42:21.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 315009 26.73 MB/sec execute 499 sec latency 62.801 ms 2024-05-09T09:42:22.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 315679 26.77 MB/sec execute 500 sec latency 127.292 ms 2024-05-09T09:42:23.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:23 smithi103 ceph-mon[34421]: pgmap v1074: 129 pgs: 129 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.2 MiB/s wr, 629 op/s 2024-05-09T09:42:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:23 smithi069 ceph-mon[30123]: pgmap v1074: 129 pgs: 129 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.2 MiB/s wr, 629 op/s 2024-05-09T09:42:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:23 smithi163 ceph-mon[34354]: pgmap v1074: 129 pgs: 129 active+clean; 665 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.6 MiB/s rd, 8.2 MiB/s wr, 629 op/s 2024-05-09T09:42:23.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 316390 26.80 MB/sec execute 501 sec latency 41.254 ms 2024-05-09T09:42:24.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 316781 26.79 MB/sec execute 502 sec latency 100.013 ms 2024-05-09T09:42:25.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:25 smithi103 ceph-mon[34421]: pgmap v1075: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 717 op/s 2024-05-09T09:42:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:25 smithi069 ceph-mon[30123]: pgmap v1075: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 717 op/s 2024-05-09T09:42:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:25 smithi163 ceph-mon[34354]: pgmap v1075: 129 pgs: 129 active+clean; 670 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 6.7 MiB/s wr, 717 op/s 2024-05-09T09:42:25.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 317280 26.78 MB/sec execute 503 sec latency 84.406 ms 2024-05-09T09:42:26.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 317830 26.78 MB/sec execute 504 sec latency 110.737 ms 2024-05-09T09:42:27.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:27 smithi103 ceph-mon[34421]: pgmap v1076: 129 pgs: 129 active+clean; 672 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 560 op/s 2024-05-09T09:42:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:27 smithi069 ceph-mon[30123]: pgmap v1076: 129 pgs: 129 active+clean; 672 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 560 op/s 2024-05-09T09:42:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:27 smithi163 ceph-mon[34354]: pgmap v1076: 129 pgs: 129 active+clean; 672 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.8 MiB/s rd, 3.7 MiB/s wr, 560 op/s 2024-05-09T09:42:27.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 318493 26.79 MB/sec execute 505 sec latency 80.547 ms 2024-05-09T09:42:28.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 319133 26.81 MB/sec execute 506 sec latency 65.142 ms 2024-05-09T09:42:29.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:29 smithi103 ceph-mon[34421]: pgmap v1077: 129 pgs: 129 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 529 op/s 2024-05-09T09:42:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:29 smithi069 ceph-mon[30123]: pgmap v1077: 129 pgs: 129 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 529 op/s 2024-05-09T09:42:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:29 smithi163 ceph-mon[34354]: pgmap v1077: 129 pgs: 129 active+clean; 677 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 529 op/s 2024-05-09T09:42:29.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 319808 26.84 MB/sec execute 507 sec latency 49.548 ms 2024-05-09T09:42:30.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 320308 26.84 MB/sec execute 508 sec latency 52.456 ms 2024-05-09T09:42:31.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:31 smithi103 ceph-mon[34421]: pgmap v1078: 129 pgs: 129 active+clean; 685 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.6 MiB/s wr, 533 op/s 2024-05-09T09:42:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:31 smithi069 ceph-mon[30123]: pgmap v1078: 129 pgs: 129 active+clean; 685 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.6 MiB/s wr, 533 op/s 2024-05-09T09:42:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:31 smithi163 ceph-mon[34354]: pgmap v1078: 129 pgs: 129 active+clean; 685 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 3.6 MiB/s wr, 533 op/s 2024-05-09T09:42:31.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 320850 26.85 MB/sec execute 509 sec latency 62.651 ms 2024-05-09T09:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.700+0000 7fa9b087d640 1 -- 172.21.15.69:0/2067185845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac1018a0 msgr2=0x7fa9ac101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.700+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2067185845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac101cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4009900 tx=0x7fa9a402a410 comp rx=0 tx=0).stop 2024-05-09T09:42:32.701 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.700+0000 7fa9b087d640 1 -- 172.21.15.69:0/2067185845 shutdown_connections 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.700+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2067185845 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac103450 0x7fa9ac109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.700+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2067185845 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac102a90 0x7fa9ac102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2067185845 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac101cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 -- 172.21.15.69:0/2067185845 >> 172.21.15.69:0/2067185845 conn(0x7fa9ac0fce10 msgr2=0x7fa9ac0ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 -- 172.21.15.69:0/2067185845 shutdown_connections 2024-05-09T09:42:32.702 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 -- 172.21.15.69:0/2067185845 wait complete. 2024-05-09T09:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 Processor -- start 2024-05-09T09:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 -- start start 2024-05-09T09:42:32.704 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.701+0000 7fa9b087d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac1978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9b087d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 0x7fa9ac197e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9b087d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9b087d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac10d520 con 0x7fa9ac103450 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9b087d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac10d3a0 con 0x7fa9ac1018a0 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9b087d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac10d6a0 con 0x7fa9ac102a90 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9a9d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 0x7fa9ac197e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac1978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40622/0 (socket says 172.21.15.69:40622) 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac1978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58472/0 (socket says 172.21.15.69:58472) 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 -- 172.21.15.69:0/2488933041 learned_addr learned my addr 172.21.15.69:0/2488933041 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9a9d9b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 0x7fa9ac197e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:59840/0 (socket says 172.21.15.69:59840) 2024-05-09T09:42:32.705 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 -- 172.21.15.69:0/2488933041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 msgr2=0x7fa9ac197e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 0x7fa9ac197e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 -- 172.21.15.69:0/2488933041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 msgr2=0x7fa9ac1978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac1978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 -- 172.21.15.69:0/2488933041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fa9ac19f510 con 0x7fa9ac103450 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00eb10 tx=0x7fa99c00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.702+0000 7fa99b7fe640 1 -- 172.21.15.69:0/2488933041 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c004b20 con 0x7fa9ac103450 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.703+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fa9ac19f800 con 0x7fa9ac103450 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.703+0000 7fa99b7fe640 1 -- 172.21.15.69:0/2488933041 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c004cc0 con 0x7fa9ac103450 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.703+0000 7fa99b7fe640 1 -- 172.21.15.69:0/2488933041 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c010760 con 0x7fa9ac103450 2024-05-09T09:42:32.706 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.703+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fa9ac19fd50 con 0x7fa9ac103450 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.705+0000 7fa99b7fe640 1 -- 172.21.15.69:0/2488933041 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fa99c010900 con 0x7fa9ac103450 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.706+0000 7fa99b7fe640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 0x7fa9940a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.706+0000 7fa99b7fe640 1 -- 172.21.15.69:0/2488933041 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa99c0cf9b0 con 0x7fa9ac103450 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.706+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 0x7fa9940a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.706+0000 7fa9b358d740 1 Processor -- start 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.706+0000 7fa9b358d740 1 -- start start 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164150 0x555d5d164570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d168ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.712 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0bf340 con 0x555d5d169610 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d5d067290 con 0x555d5d164150 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9b358d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555d5d0bb300 con 0x555d5d164b40 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 0x7fa9940a50f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4004770 tx=0x7fa9a4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164150 0x555d5d164570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40636/0 (socket says 172.21.15.69:40636) 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 -- 172.21.15.69:0/4119835686 learned_addr learned my addr 172.21.15.69:0/4119835686 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d168ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 -- 172.21.15.69:0/4119835686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 msgr2=0x555d5d168ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d168ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 -- 172.21.15.69:0/4119835686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164150 msgr2=0x555d5d164570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164150 0x555d5d164570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.713 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 -- 172.21.15.69:0/4119835686 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5d08b8e0 con 0x555d5d169610 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0982f0 tx=0x7fa99c09bd20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3293b5ce34bbaeb9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.707+0000 7fa998ff9640 1 -- 172.21.15.69:0/4119835686 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c09be90 con 0x555d5d169610 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa998ff9640 1 -- 172.21.15.69:0/4119835686 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c0a0240 con 0x555d5d169610 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa998ff9640 1 -- 172.21.15.69:0/4119835686 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c0a48d0 con 0x555d5d169610 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 -- 172.21.15.69:0/4119835686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 msgr2=0x555d5d16baa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0982f0 tx=0x7fa99c09bd20 comp rx=0 tx=0).stop 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 -- 172.21.15.69:0/4119835686 shutdown_connections 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d16baa0 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d168ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 --2- 172.21.15.69:0/4119835686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164150 0x555d5d164570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 -- 172.21.15.69:0/4119835686 >> 172.21.15.69:0/4119835686 conn(0x555d5d156840 msgr2=0x555d5d156c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:32.714 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 -- 172.21.15.69:0/4119835686 shutdown_connections 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.708+0000 7fa9b358d740 1 -- 172.21.15.69:0/4119835686 wait complete. 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 -- start start 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 Processor -- start 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 0x555d5d16e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x555d5d16c5c0 con 0x555d5d169610 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x555d5d07a790 con 0x555d5d164b40 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x555d5d16c740 con 0x555d5d164150 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9b358d740 10 client.0 _ll_register_callbacks cb 0x555d5d0b5a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.715 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:40640/0 (socket says 172.21.15.69:40640) 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:59864/0 (socket says 172.21.15.69:59864) 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9aad9d640 1 -- 172.21.15.69:0/3057989528 learned_addr learned my addr 172.21.15.69:0/3057989528 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.709+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 0x555d5d16e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 -- 172.21.15.69:0/3057989528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 msgr2=0x555d5d16dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 -- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 msgr2=0x555d5d16e1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 0x555d5d16e1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 -- 172.21.15.69:0/3057989528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5cdd3210 con 0x555d5d169610 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 0x555d5d16e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0a45d0 tx=0x7fa99c004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fe2554d491375b66 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c09b4e0 con 0x555d5d169610 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa99c097be0 con 0x555d5d169610 2024-05-09T09:42:32.716 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa99c0a0240 con 0x555d5d169610 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa99c09be90 con 0x555d5d169610 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 10 client.17877 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x555d5ce9f620 con 0x555d5d169610 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c0a0240 con 0x555d5d169610 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fa99c082030 con 0x555d5d169610 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 10 client.17877 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 10 client.17877 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 20 client.17877 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d17bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 4 client.17877 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.710+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x555d5d0cca70 con 0x555d5d1798f0 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.711+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d17bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.711+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d17bd10 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3fb59f63db11038e server_cookie=44af44100a7f29ad in_seq=0 out_seq=0 2024-05-09T09:42:32.717 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.711+0000 7fa991ffb640 10 client.17877 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.712+0000 7fa991ffb640 1 -- 172.21.15.69:0/3057989528 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x555d5d0cca70 con 0x555d5d1798f0 2024-05-09T09:42:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.712+0000 7fa991ffb640 10 client.17877 handle_command_reply: tid=0 2024-05-09T09:42:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.712+0000 7fa9b358d740 1 client.17877 shutdown 2024-05-09T09:42:32.719 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.712+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 msgr2=0x555d5d1751e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.712+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 crypto rx=0x7fa99c0a45d0 tx=0x7fa99c004a40 comp rx=0 tx=0).stop 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 shutdown_connections 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d169610 0x555d5d1751e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d164b40 0x555d5d16e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d17bd10 crc :-1 s=READY pgs=268 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 --2- 172.21.15.69:0/3057989528 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa991ffb640 0 client.17877 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 >> 172.21.15.69:0/3057989528 conn(0x555d5d156840 msgr2=0x555d5d1637e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 shutdown_connections 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 1 -- 172.21.15.69:0/3057989528 wait complete. 2024-05-09T09:42:32.720 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.713+0000 7fa9b358d740 20 client.17877 trim_cache size 0 max 16384 2024-05-09T09:42:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.731+0000 7fa9b358d740 1 Processor -- start 2024-05-09T09:42:32.731 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.731+0000 7fa9b358d740 1 -- start start 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d1751a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1aa950 con 0x555d5d164150 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1a81c0 con 0x555d5d169610 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9b358d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1a6010 con 0x555d5d164b40 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.733 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9aad9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:58484/0 (socket says 172.21.15.69:58484) 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9aad9d640 1 -- 172.21.15.69:0/1038936341 learned_addr learned my addr 172.21.15.69:0/1038936341 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.732+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d1751a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 -- 172.21.15.69:0/1038936341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 msgr2=0x555d5d1751a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d1751a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 -- 172.21.15.69:0/1038936341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 msgr2=0x555d5d16dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 -- 172.21.15.69:0/1038936341 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x555d5d1651a0 con 0x555d5d169610 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:32.734 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa99c097fb0 tx=0x7fa99c009200 comp rx=0 tx=0).ready entity=mon.2 client_cookie=72312265a6cf9f37 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa993fff640 1 -- 172.21.15.69:0/1038936341 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c09b7f0 con 0x555d5d169610 2024-05-09T09:42:32.738 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa993fff640 1 -- 172.21.15.69:0/1038936341 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c0b8070 con 0x555d5d169610 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.733+0000 7fa993fff640 1 -- 172.21.15.69:0/1038936341 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa99c09ae70 con 0x555d5d169610 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 -- 172.21.15.69:0/1038936341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 msgr2=0x555d5d16e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7fa99c097fb0 tx=0x7fa99c009200 comp rx=0 tx=0).stop 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 -- 172.21.15.69:0/1038936341 shutdown_connections 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d16e1a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164b40 0x555d5d1751a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 --2- 172.21.15.69:0/1038936341 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164150 0x555d5d16dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.734+0000 7fa9b358d740 1 -- 172.21.15.69:0/1038936341 >> 172.21.15.69:0/1038936341 conn(0x555d5d156840 msgr2=0x555d5d156c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 -- 172.21.15.69:0/1038936341 shutdown_connections 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 -- 172.21.15.69:0/1038936341 wait complete. 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 -- start start 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 Processor -- start 2024-05-09T09:42:32.739 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 0x555d5d172850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.735+0000 7fa9b358d740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d173860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9b358d740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1a6680 con 0x555d5d164b40 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9b358d740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1a7af0 con 0x555d5d169610 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9b358d740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fa9ac1a6d50 con 0x555d5d164150 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9b358d740 10 client.0 _ll_register_callbacks cb 0x7fa9a4040070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9aa59c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:59894/0 (socket says 172.21.15.69:59894) 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9aa59c640 1 -- 172.21.15.69:0/2684717142 learned_addr learned my addr 172.21.15.69:0/2684717142 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 0x555d5d172850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9aad9d640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d173860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 -- 172.21.15.69:0/2684717142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 msgr2=0x555d5d172310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.740 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 -- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 msgr2=0x555d5d173860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d173860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 -- 172.21.15.69:0/2684717142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x555d5d173e50 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9a9d9b640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 0x555d5d172850 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x555d5d16f140 tx=0x7fa9a000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13a37a3a54cd9a35 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa991ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fa9a0023060 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fa99c0b8070 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.736+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fa9a0004050 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fa9a002a050 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 10 client.17883 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x555d5d175260 con 0x555d5d164b40 2024-05-09T09:42:32.741 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fa99c0b8070 con 0x555d5d164b40 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fa9a0010430 con 0x555d5d164b40 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 10 client.17883 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 10 client.17883 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 20 client.17883 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d1e2490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 4 client.17883 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x555d5d0c6dc0 con 0x555d5d1798f0 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.737+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d1e2490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.738+0000 7fa9aa59c640 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d1e2490 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4b2a006ab85bf1fb server_cookie=d710cd9c4415f1d5 in_seq=0 out_seq=0 2024-05-09T09:42:32.742 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:32.738+0000 7fa991ffb640 10 client.17883 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:32.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 321368 26.84 MB/sec execute 510 sec latency 100.610 ms 2024-05-09T09:42:33.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:33 smithi103 ceph-mon[34421]: pgmap v1079: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-05-09T09:42:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:33 smithi069 ceph-mon[30123]: pgmap v1079: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-05-09T09:42:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:33 smithi163 ceph-mon[34354]: pgmap v1079: 129 pgs: 129 active+clean; 683 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.9 MiB/s wr, 430 op/s 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.752+0000 7fa991ffb640 1 -- 172.21.15.69:0/2684717142 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x555d5d0c6dc0 con 0x555d5d1798f0 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.752+0000 7fa991ffb640 10 client.17883 handle_command_reply: tid=0 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.752+0000 7fa9b358d740 1 client.17883 shutdown 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 msgr2=0x555d5d172850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 0x555d5d172850 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 crypto rx=0x555d5d16f140 tx=0x7fa9a000f790 comp rx=0 tx=0).stop 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 shutdown_connections 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x555d5d169610 0x555d5d173860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x555d5d164b40 0x555d5d172850 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.754 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x555d5d1798f0 0x555d5d1e2490 crc :-1 s=READY pgs=269 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 --2- 172.21.15.69:0/2684717142 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x555d5d164150 0x555d5d172310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 >> 172.21.15.69:0/2684717142 conn(0x555d5d156840 msgr2=0x555d5d16dc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa991ffb640 0 client.17883 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 shutdown_connections 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 1 -- 172.21.15.69:0/2684717142 wait complete. 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.753+0000 7fa9b358d740 20 client.17883 trim_cache size 0 max 16384 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:42:33.755 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 msgr2=0x7fa9940a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 0x7fa9940a50f0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fa9a4004770 tx=0x7fa9a4000bc0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 msgr2=0x7fa9ac19eea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 secure :-1 s=READY pgs=1108 cs=0 l=1 rev1=1 crypto rx=0x7fa99c00eb10 tx=0x7fa99c00c2d0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 shutdown_connections 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fa9940a2c30 0x7fa9940a50f0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fa9ac103450 0x7fa9ac19eea0 unknown :-1 s=CLOSED pgs=1108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fa9ac102a90 0x7fa9ac197e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 --2- 172.21.15.69:0/2488933041 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fa9ac1018a0 0x7fa9ac1978f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.754+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 >> 172.21.15.69:0/2488933041 conn(0x7fa9ac0fce10 msgr2=0x7fa9ac105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.755+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 shutdown_connections 2024-05-09T09:42:33.758 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:33.755+0000 7fa9b087d640 1 -- 172.21.15.69:0/2488933041 wait complete. 2024-05-09T09:42:33.843 INFO:tasks.workunit.client.0.smithi069.stdout: 10 321493 26.81 MB/sec execute 511 sec latency 796.674 ms 2024-05-09T09:42:34.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:34 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:42:34.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:34 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:42:34.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:34 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:42:34.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 322085 26.80 MB/sec execute 512 sec latency 89.027 ms 2024-05-09T09:42:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:35 smithi069 ceph-mon[30123]: pgmap v1080: 129 pgs: 129 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.5 MiB/s wr, 494 op/s 2024-05-09T09:42:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:42:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:42:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:42:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:35 smithi163 ceph-mon[34354]: pgmap v1080: 129 pgs: 129 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.5 MiB/s wr, 494 op/s 2024-05-09T09:42:35.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:42:35.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:42:35.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:42:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:35 smithi103 ceph-mon[34421]: pgmap v1080: 129 pgs: 129 active+clean; 669 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.5 MiB/s wr, 494 op/s 2024-05-09T09:42:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:42:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:42:35.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:42:35.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 322586 26.82 MB/sec execute 513 sec latency 86.718 ms 2024-05-09T09:42:36.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323162 26.83 MB/sec execute 514 sec latency 65.066 ms 2024-05-09T09:42:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:37 smithi069 ceph-mon[30123]: pgmap v1081: 129 pgs: 129 active+clean; 674 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 422 op/s 2024-05-09T09:42:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:37 smithi163 ceph-mon[34354]: pgmap v1081: 129 pgs: 129 active+clean; 674 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 422 op/s 2024-05-09T09:42:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:37 smithi103 ceph-mon[34421]: pgmap v1081: 129 pgs: 129 active+clean; 674 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 3.0 MiB/s wr, 422 op/s 2024-05-09T09:42:37.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323446 26.80 MB/sec execute 515 sec latency 515.591 ms 2024-05-09T09:42:38.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323455 26.75 MB/sec execute 516 sec latency 1491.794 ms 2024-05-09T09:42:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:39 smithi069 ceph-mon[30123]: pgmap v1082: 129 pgs: 129 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.9 MiB/s wr, 437 op/s 2024-05-09T09:42:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:39 smithi163 ceph-mon[34354]: pgmap v1082: 129 pgs: 129 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.9 MiB/s wr, 437 op/s 2024-05-09T09:42:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:39 smithi103 ceph-mon[34421]: pgmap v1082: 129 pgs: 129 active+clean; 675 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 2.9 MiB/s wr, 437 op/s 2024-05-09T09:42:39.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323497 26.71 MB/sec execute 517 sec latency 2396.041 ms 2024-05-09T09:42:40.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323611 26.66 MB/sec execute 518 sec latency 3396.081 ms 2024-05-09T09:42:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:41 smithi069 ceph-mon[30123]: pgmap v1083: 129 pgs: 129 active+clean; 700 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.6 MiB/s wr, 493 op/s 2024-05-09T09:42:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:41 smithi163 ceph-mon[34354]: pgmap v1083: 129 pgs: 129 active+clean; 700 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.6 MiB/s wr, 493 op/s 2024-05-09T09:42:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:41 smithi103 ceph-mon[34421]: pgmap v1083: 129 pgs: 129 active+clean; 700 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.6 MiB/s wr, 493 op/s 2024-05-09T09:42:41.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323797 26.63 MB/sec execute 519 sec latency 3542.269 ms 2024-05-09T09:42:42.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 323986 26.60 MB/sec execute 520 sec latency 354.082 ms 2024-05-09T09:42:43.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:43 smithi069 ceph-mon[30123]: pgmap v1084: 129 pgs: 129 active+clean; 745 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.5 MiB/s wr, 553 op/s 2024-05-09T09:42:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:43 smithi163 ceph-mon[34354]: pgmap v1084: 129 pgs: 129 active+clean; 745 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.5 MiB/s wr, 553 op/s 2024-05-09T09:42:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:43 smithi103 ceph-mon[34421]: pgmap v1084: 129 pgs: 129 active+clean; 745 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.1 MiB/s rd, 9.5 MiB/s wr, 553 op/s 2024-05-09T09:42:43.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 324161 26.56 MB/sec execute 521 sec latency 183.630 ms 2024-05-09T09:42:44.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 324475 26.55 MB/sec execute 522 sec latency 134.248 ms 2024-05-09T09:42:45.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:45 smithi069 ceph-mon[30123]: pgmap v1085: 129 pgs: 129 active+clean; 747 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-05-09T09:42:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:45 smithi163 ceph-mon[34354]: pgmap v1085: 129 pgs: 129 active+clean; 747 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-05-09T09:42:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:45 smithi103 ceph-mon[34421]: pgmap v1085: 129 pgs: 129 active+clean; 747 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 10 MiB/s wr, 567 op/s 2024-05-09T09:42:45.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 325017 26.54 MB/sec execute 523 sec latency 95.261 ms 2024-05-09T09:42:46.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 325583 26.55 MB/sec execute 524 sec latency 83.227 ms 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 -- 172.21.15.69:0/2106979921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10db50 msgr2=0x7f251c10dfd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 --2- 172.21.15.69:0/2106979921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10db50 0x7f251c10dfd0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f2510009900 tx=0x7f251002a420 comp rx=0 tx=0).stop 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 -- 172.21.15.69:0/2106979921 shutdown_connections 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 --2- 172.21.15.69:0/2106979921 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10db50 0x7f251c10dfd0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 --2- 172.21.15.69:0/2106979921 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f251c10b1a0 0x7f251c10d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 --2- 172.21.15.69:0/2106979921 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c10ac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.890 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 -- 172.21.15.69:0/2106979921 >> 172.21.15.69:0/2106979921 conn(0x7f251c06e8a0 msgr2=0x7f251c070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 -- 172.21.15.69:0/2106979921 shutdown_connections 2024-05-09T09:42:46.891 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.890+0000 7f25218cd640 1 -- 172.21.15.69:0/2106979921 wait complete. 2024-05-09T09:42:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.891+0000 7f25218cd640 1 Processor -- start 2024-05-09T09:42:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.891+0000 7f25218cd640 1 -- start start 2024-05-09T09:42:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c1a69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.892 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f251c10db50 0x7f251c1adf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f251c11c430 con 0x7f251c10db50 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f251c11c2b0 con 0x7f251c108830 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f25218cd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f251c11c5b0 con 0x7f251c10b1a0 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53674/0 (socket says 172.21.15.69:53674) 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 -- 172.21.15.69:0/2004978651 learned_addr learned my addr 172.21.15.69:0/2004978651 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:46.893 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251affd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c1a69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 -- 172.21.15.69:0/2004978651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 msgr2=0x7f251c1a69a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c1a69a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.894 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 -- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f251c10db50 msgr2=0x7f251c1adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f251c10db50 0x7f251c1adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251a7fc640 1 -- 172.21.15.69:0/2004978651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f251c1ae5c0 con 0x7f251c10b1a0 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.892+0000 7f251affd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c1a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.893+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f251400ba50 tx=0x7f251400bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.895+0000 7f25208cb640 1 -- 172.21.15.69:0/2004978651 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514019070 con 0x7f251c10b1a0 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.895+0000 7f25208cb640 1 -- 172.21.15.69:0/2004978651 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f251400cbe0 con 0x7f251c10b1a0 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.895+0000 7f25208cb640 1 -- 172.21.15.69:0/2004978651 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514007770 con 0x7f251c10b1a0 2024-05-09T09:42:46.895 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.895+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f251c1ae8b0 con 0x7f251c10b1a0 2024-05-09T09:42:46.896 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.895+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f251c1aee30 con 0x7f251c10b1a0 2024-05-09T09:42:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.899+0000 7f25245db740 1 Processor -- start 2024-05-09T09:42:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.899+0000 7f25245db740 1 -- start start 2024-05-09T09:42:46.900 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.899+0000 7f25208cb640 1 -- 172.21.15.69:0/2004978651 <== mon.1 v2:172.21.15.163:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f251401c070 con 0x7f251c10b1a0 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.899+0000 7f25245db740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25245db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595c40 0x55daa559a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25245db740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa559adb0 0x55daa559d240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25245db740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55daa54f9340 con 0x55daa5595250 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25245db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55daa54a1290 con 0x55daa5595c40 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25245db740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55daa54f5300 con 0x55daa559adb0 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43170/0 (socket says 172.21.15.69:43170) 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 -- 172.21.15.69:0/239636854 learned_addr learned my addr 172.21.15.69:0/239636854 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251affd640 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595c40 0x55daa559a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251a7fc640 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa559adb0 0x55daa559d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 -- 172.21.15.69:0/239636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa559adb0 msgr2=0x55daa559d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa559adb0 0x55daa559d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.901 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 -- 172.21.15.69:0/239636854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595c40 msgr2=0x55daa559a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595c40 0x55daa559a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 -- 172.21.15.69:0/239636854 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa54c58e0 con 0x55daa5595250 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f251b7fe640 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f251002a940 tx=0x7f25100040f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fcbe184955647722 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.900+0000 7f25208cb640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 0x7f25040a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.901+0000 7f2501ffb640 1 -- 172.21.15.69:0/239636854 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f251003e030 con 0x55daa5595250 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.901+0000 7f251affd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 0x7f25040a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.901+0000 7f25208cb640 1 -- 172.21.15.69:0/2004978651 <== mon.1 v2:172.21.15.163:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f25140ce8d0 con 0x7f251c10b1a0 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.901+0000 7f2501ffb640 1 -- 172.21.15.69:0/239636854 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25100042d0 con 0x55daa5595250 2024-05-09T09:42:46.902 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.901+0000 7f2501ffb640 1 -- 172.21.15.69:0/239636854 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2510035040 con 0x55daa5595250 2024-05-09T09:42:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- 172.21.15.69:0/239636854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 msgr2=0x55daa5595670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 secure :-1 s=READY pgs=1112 cs=0 l=1 rev1=1 crypto rx=0x7f251002a940 tx=0x7f25100040f0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.906 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f251affd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 0x7f25040a5260 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55daa5596040 tx=0x7f250c0090e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- 172.21.15.69:0/239636854 shutdown_connections 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa559adb0 0x55daa559d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595c40 0x55daa559a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 --2- 172.21.15.69:0/239636854 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa5595670 unknown :-1 s=CLOSED pgs=1112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- 172.21.15.69:0/239636854 >> 172.21.15.69:0/239636854 conn(0x55daa55908c0 msgr2=0x55daa5590cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- 172.21.15.69:0/239636854 shutdown_connections 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- 172.21.15.69:0/239636854 wait complete. 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 -- start start 2024-05-09T09:42:46.907 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.902+0000 7f25245db740 1 Processor -- start 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa55a6a00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55daa559dd50 con 0x55daa5595c40 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55daa54b4790 con 0x55daa559adb0 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55daa559ded0 con 0x55daa5595250 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f25245db740 10 client.0 _ll_register_callbacks cb 0x55daa54efa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251b7fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa55a6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251affd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53694/0 (socket says 172.21.15.69:53694) 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251affd640 1 -- 172.21.15.69:0/1595241499 learned_addr learned my addr 172.21.15.69:0/1595241499 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:46.908 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43180/0 (socket says 172.21.15.69:43180) 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 -- 172.21.15.69:0/1595241499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 msgr2=0x55daa559f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 -- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 msgr2=0x55daa55a6a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa55a6a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 -- 172.21.15.69:0/1595241499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa520d210 con 0x55daa5595c40 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251affd640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251b7fe640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa55a6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f251a7fc640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f25140cc0a0 tx=0x7f2514004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dcca477cd5111419 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.903+0000 7f24faffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514014a40 con 0x55daa5595c40 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f25140a3590 con 0x55daa5595c40 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25100042d0 con 0x55daa5595c40 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f25140a37b0 con 0x55daa5595c40 2024-05-09T09:42:46.909 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 10 client.17889 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55daa52d9620 con 0x55daa5595c40 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25100042d0 con 0x55daa5595c40 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f2514099eb0 con 0x55daa5595c40 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 10 client.17889 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 10 client.17889 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 20 client.17889 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa55ab020 0x55daa55ad460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 4 client.17889 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55daa5506a70 con 0x55daa55ab020 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.904+0000 7f251affd640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa55ab020 0x55daa55ad460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.905+0000 7f24faffd640 10 client.17889 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:46.910 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.905+0000 7f251affd640 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa55ab020 0x55daa55ad460 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8f5c94ec50fa7076 server_cookie=885a868cbd072185 in_seq=0 out_seq=0 2024-05-09T09:42:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f24faffd640 1 -- 172.21.15.69:0/1595241499 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55daa5506a70 con 0x55daa55ab020 2024-05-09T09:42:46.911 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f24faffd640 10 client.17889 handle_command_reply: tid=0 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 client.17889 shutdown 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 msgr2=0x55daa559f9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 secure :-1 s=READY pgs=1113 cs=0 l=1 rev1=1 crypto rx=0x7f25140cc0a0 tx=0x7f2514004a40 comp rx=0 tx=0).stop 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 shutdown_connections 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa55a6a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595c40 0x55daa559f9c0 unknown :-1 s=CLOSED pgs=1113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa55ab020 0x55daa55ad460 crc :-1 s=READY pgs=270 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f25245db740 1 --2- 172.21.15.69:0/1595241499 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595250 0x55daa559f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.908+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 >> 172.21.15.69:0/1595241499 conn(0x55daa55908c0 msgr2=0x55daa558fa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.907+0000 7f24faffd640 0 client.17889 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.908+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 shutdown_connections 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.908+0000 7f25245db740 1 -- 172.21.15.69:0/1595241499 wait complete. 2024-05-09T09:42:46.912 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.908+0000 7f25245db740 20 client.17889 trim_cache size 0 max 16384 2024-05-09T09:42:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f25245db740 1 Processor -- start 2024-05-09T09:42:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f25245db740 1 -- start start 2024-05-09T09:42:46.931 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f25245db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 0x55daa559d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f25245db740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f25245db740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa559adb0 0x55daa5596e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f251affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 0x55daa559d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.932+0000 7f251affd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 0x55daa559d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:59584/0 (socket says 172.21.15.69:59584) 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53706/0 (socket says 172.21.15.69:53706) 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f251affd640 1 -- 172.21.15.69:0/2689311530 learned_addr learned my addr 172.21.15.69:0/2689311530 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1bba10 con 0x55daa559adb0 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1b9280 con 0x55daa5595250 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1b70d0 con 0x55daa5595c40 2024-05-09T09:42:46.933 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.933+0000 7f251b7fe640 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa559adb0 0x55daa5596e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 -- 172.21.15.69:0/2689311530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 msgr2=0x55daa559d1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 0x55daa559d1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 -- 172.21.15.69:0/2689311530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa559adb0 msgr2=0x55daa5596e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa559adb0 0x55daa5596e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 -- 172.21.15.69:0/2689311530 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55daa55adae0 con 0x55daa5595c40 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.934+0000 7f251a7fc640 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25140977e0 tx=0x7f25140cec60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9450a40d2344766f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.934 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.935+0000 7f25017fa640 1 -- 172.21.15.69:0/2689311530 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f25140a8070 con 0x55daa5595c40 2024-05-09T09:42:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.935+0000 7f25017fa640 1 -- 172.21.15.69:0/2689311530 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25140a0390 con 0x55daa5595c40 2024-05-09T09:42:46.935 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.935+0000 7f25017fa640 1 -- 172.21.15.69:0/2689311530 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2514014b10 con 0x55daa5595c40 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 msgr2=0x55daa55a69c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f25140977e0 tx=0x7f25140cec60 comp rx=0 tx=0).stop 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 shutdown_connections 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa559adb0 0x55daa5596e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a69c0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 --2- 172.21.15.69:0/2689311530 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa5595250 0x55daa559d1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 >> 172.21.15.69:0/2689311530 conn(0x55daa55908c0 msgr2=0x55daa5590ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 shutdown_connections 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.936+0000 7f25245db740 1 -- 172.21.15.69:0/2689311530 wait complete. 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 -- start start 2024-05-09T09:42:46.937 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 Processor -- start 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa560f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1b7740 con 0x55daa5595250 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1b8bb0 con 0x55daa559adb0 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f251c1b7e10 con 0x55daa5595c40 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.937+0000 7f25245db740 10 client.0 _ll_register_callbacks cb 0x7f251409aa90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43194/0 (socket says 172.21.15.69:43194) 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251a7fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:53722/0 (socket says 172.21.15.69:53722) 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 -- 172.21.15.69:0/3333784855 learned_addr learned my addr 172.21.15.69:0/3333784855 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:42:46.938 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251b7fe640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa560f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 -- 172.21.15.69:0/3333784855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 msgr2=0x55daa55a3ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.940 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 -- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 msgr2=0x55daa560f5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa560f5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 -- 172.21.15.69:0/3333784855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55daa559ab00 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251a7fc640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251b7fe640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa560f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f251affd640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f250c009ce0 tx=0x7f250c018000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4b01ae538ab2722c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.938+0000 7f24faffd640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f250c02ae20 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f250c023920 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f25140a0390 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f250c03c050 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 10 client.17895 fetch_fsmap learned FSMap version 49 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55daa5610940 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f25140a0390 con 0x55daa5595250 2024-05-09T09:42:46.941 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f250c020040 con 0x55daa5595250 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 10 client.17895 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 10 client.17895 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 20 client.17895 populate_metadata read hostname 'smithi069' 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa56113d0 0x55daa56137f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 4 client.17895 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.939+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55daa5500dc0 con 0x55daa56113d0 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.941+0000 7f251a7fc640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa56113d0 0x55daa56137f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.942+0000 7f251a7fc640 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa56113d0 0x55daa56137f0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5192cf0e34655e51 server_cookie=ff1c5e824293f10a in_seq=0 out_seq=0 2024-05-09T09:42:46.942 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:46.942+0000 7f24faffd640 10 client.17895 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:47.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f24faffd640 1 -- 172.21.15.69:0/3333784855 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55daa5500dc0 con 0x55daa56113d0 2024-05-09T09:42:47.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f24faffd640 10 client.17895 handle_command_reply: tid=0 2024-05-09T09:42:47.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 client.17895 shutdown 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 msgr2=0x55daa55a3580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 secure :-1 s=READY pgs=1114 cs=0 l=1 rev1=1 crypto rx=0x7f250c009ce0 tx=0x7f250c018000 comp rx=0 tx=0).stop 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 shutdown_connections 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55daa559adb0 0x55daa560f5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55daa56113d0 0x55daa56137f0 crc :-1 s=READY pgs=271 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55daa5595c40 0x55daa55a3ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 --2- 172.21.15.69:0/3333784855 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55daa5595250 0x55daa55a3580 unknown :-1 s=CLOSED pgs=1114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 >> 172.21.15.69:0/3333784855 conn(0x55daa55908c0 msgr2=0x55daa5596e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.455+0000 7f24faffd640 0 client.17895 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 shutdown_connections 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25245db740 1 -- 172.21.15.69:0/3333784855 wait complete. 2024-05-09T09:42:47.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25245db740 20 client.17895 trim_cache size 0 max 16384 2024-05-09T09:42:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:42:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:42:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:42:47.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:42:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 msgr2=0x7f25040a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 0x7f25040a5260 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x55daa5596040 tx=0x7f250c0090e0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 msgr2=0x7f251c1a6ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:42:47.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f251400ba50 tx=0x7f251400bf10 comp rx=0 tx=0).stop 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 shutdown_connections 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f25040a2da0 0x7f25040a5260 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f251c10db50 0x7f251c1adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f251c10b1a0 0x7f251c1a6ee0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 --2- 172.21.15.69:0/2004978651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f251c108830 0x7f251c1a69a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:42:47.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 >> 172.21.15.69:0/2004978651 conn(0x7f251c06e8a0 msgr2=0x7f251c1095f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:42:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 shutdown_connections 2024-05-09T09:42:47.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:42:47.456+0000 7f25218cd640 1 -- 172.21.15.69:0/2004978651 wait complete. 2024-05-09T09:42:47.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:47 smithi069 ceph-mon[30123]: pgmap v1086: 129 pgs: 129 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-05-09T09:42:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:47 smithi163 ceph-mon[34354]: pgmap v1086: 129 pgs: 129 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-05-09T09:42:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:47 smithi103 ceph-mon[34421]: pgmap v1086: 129 pgs: 129 active+clean; 694 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 MiB/s rd, 12 MiB/s wr, 669 op/s 2024-05-09T09:42:47.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 326042 26.56 MB/sec execute 525 sec latency 389.284 ms 2024-05-09T09:42:48.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 326584 26.57 MB/sec execute 526 sec latency 107.191 ms 2024-05-09T09:42:49.418 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:49 smithi103 ceph-mon[34421]: pgmap v1087: 129 pgs: 129 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 637 op/s 2024-05-09T09:42:49.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:49 smithi069 ceph-mon[30123]: pgmap v1087: 129 pgs: 129 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 637 op/s 2024-05-09T09:42:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:49 smithi163 ceph-mon[34354]: pgmap v1087: 129 pgs: 129 active+clean; 681 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 4.2 MiB/s rd, 12 MiB/s wr, 637 op/s 2024-05-09T09:42:49.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 327056 26.57 MB/sec execute 527 sec latency 97.116 ms 2024-05-09T09:42:50.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 327468 26.56 MB/sec execute 528 sec latency 101.193 ms 2024-05-09T09:42:51.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:51 smithi069 ceph-mon[30123]: pgmap v1088: 129 pgs: 129 active+clean; 667 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 12 MiB/s wr, 708 op/s 2024-05-09T09:42:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:51 smithi163 ceph-mon[34354]: pgmap v1088: 129 pgs: 129 active+clean; 667 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 12 MiB/s wr, 708 op/s 2024-05-09T09:42:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:51 smithi103 ceph-mon[34421]: pgmap v1088: 129 pgs: 129 active+clean; 667 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 12 MiB/s wr, 708 op/s 2024-05-09T09:42:51.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 327955 26.54 MB/sec execute 529 sec latency 79.622 ms 2024-05-09T09:42:52.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 328575 26.56 MB/sec execute 530 sec latency 76.937 ms 2024-05-09T09:42:53.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:53 smithi069 ceph-mon[30123]: pgmap v1089: 129 pgs: 129 active+clean; 663 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-05-09T09:42:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:53 smithi163 ceph-mon[34354]: pgmap v1089: 129 pgs: 129 active+clean; 663 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-05-09T09:42:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:53 smithi103 ceph-mon[34421]: pgmap v1089: 129 pgs: 129 active+clean; 663 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 9.4 MiB/s wr, 697 op/s 2024-05-09T09:42:53.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 329202 26.57 MB/sec execute 531 sec latency 141.028 ms 2024-05-09T09:42:54.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 329953 26.60 MB/sec execute 532 sec latency 46.414 ms 2024-05-09T09:42:55.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:55 smithi069 ceph-mon[30123]: pgmap v1090: 129 pgs: 129 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.0 MiB/s wr, 592 op/s 2024-05-09T09:42:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:55 smithi163 ceph-mon[34354]: pgmap v1090: 129 pgs: 129 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.0 MiB/s wr, 592 op/s 2024-05-09T09:42:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:55 smithi103 ceph-mon[34421]: pgmap v1090: 129 pgs: 129 active+clean; 658 MiB data, 11 GiB used, 1.0 TiB / 1.0 TiB avail; 6.1 MiB/s rd, 5.0 MiB/s wr, 592 op/s 2024-05-09T09:42:55.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 330442 26.61 MB/sec execute 533 sec latency 72.292 ms 2024-05-09T09:42:56.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 330995 26.62 MB/sec execute 534 sec latency 86.324 ms 2024-05-09T09:42:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:57 smithi069 ceph-mon[30123]: pgmap v1091: 129 pgs: 129 active+clean; 652 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-05-09T09:42:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:57 smithi163 ceph-mon[34354]: pgmap v1091: 129 pgs: 129 active+clean; 652 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-05-09T09:42:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:57 smithi103 ceph-mon[34421]: pgmap v1091: 129 pgs: 129 active+clean; 652 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 4.9 MiB/s wr, 641 op/s 2024-05-09T09:42:57.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 331410 26.59 MB/sec execute 535 sec latency 92.742 ms 2024-05-09T09:42:58.844 INFO:tasks.workunit.client.0.smithi069.stdout: 10 331915 26.59 MB/sec execute 536 sec latency 150.710 ms 2024-05-09T09:42:59.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:42:59 smithi069 ceph-mon[30123]: pgmap v1092: 129 pgs: 129 active+clean; 655 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 473 op/s 2024-05-09T09:42:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:42:59 smithi163 ceph-mon[34354]: pgmap v1092: 129 pgs: 129 active+clean; 655 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 473 op/s 2024-05-09T09:42:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:42:59 smithi103 ceph-mon[34421]: pgmap v1092: 129 pgs: 129 active+clean; 655 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.2 MiB/s rd, 2.7 MiB/s wr, 473 op/s 2024-05-09T09:42:59.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 332685 26.62 MB/sec execute 537 sec latency 72.217 ms 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 -- 172.21.15.69:0/842132064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4105460 msgr2=0x7fcba41058e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 --2- 172.21.15.69:0/842132064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4105460 0x7fcba41058e0 secure :-1 s=READY pgs=1115 cs=0 l=1 rev1=1 crypto rx=0x7fcb9c009ca0 tx=0x7fcb9c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 -- 172.21.15.69:0/842132064 shutdown_connections 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 --2- 172.21.15.69:0/842132064 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4105460 0x7fcba41058e0 unknown :-1 s=CLOSED pgs=1115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 --2- 172.21.15.69:0/842132064 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba4104f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 --2- 172.21.15.69:0/842132064 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4100140 0x7fcba4102570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.556 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 -- 172.21.15.69:0/842132064 >> 172.21.15.69:0/842132064 conn(0x7fcba40fbd40 msgr2=0x7fcba40fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 -- 172.21.15.69:0/842132064 shutdown_connections 2024-05-09T09:43:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 -- 172.21.15.69:0/842132064 wait complete. 2024-05-09T09:43:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.556+0000 7fcbaa310640 1 Processor -- start 2024-05-09T09:43:00.557 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 -- start start 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba419e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4105460 0x7fcba41a55f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4113b40 con 0x7fcba4100140 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41139c0 con 0x7fcba4102ab0 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcbaa310640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcba4113cc0 con 0x7fcba4105460 2024-05-09T09:43:00.559 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37048/0 (socket says 172.21.15.69:37048) 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 -- 172.21.15.69:0/3756035752 learned_addr learned my addr 172.21.15.69:0/3756035752 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba37fe640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba419e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba88ad640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4105460 0x7fcba41a55f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 -- 172.21.15.69:0/3756035752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4105460 msgr2=0x7fcba41a55f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4105460 0x7fcba41a55f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 -- 172.21.15.69:0/3756035752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 msgr2=0x7fcba419e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba419e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 -- 172.21.15.69:0/3756035752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fcba41a5c60 con 0x7fcba4100140 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba37fe640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.557+0000 7fcba3fff640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400e9f0 tx=0x7fcb9400edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.558+0000 7fcba17fa640 1 -- 172.21.15.69:0/3756035752 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400cd80 con 0x7fcba4100140 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.558+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fcba41a5f50 con 0x7fcba4100140 2024-05-09T09:43:00.560 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.558+0000 7fcba17fa640 1 -- 172.21.15.69:0/3756035752 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb94005500 con 0x7fcba4100140 2024-05-09T09:43:00.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.558+0000 7fcba17fa640 1 -- 172.21.15.69:0/3756035752 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb94010690 con 0x7fcba4100140 2024-05-09T09:43:00.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.558+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fcba41a63b0 con 0x7fcba4100140 2024-05-09T09:43:00.561 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.560+0000 7fcba17fa640 1 -- 172.21.15.69:0/3756035752 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fcb940046c0 con 0x7fcba4100140 2024-05-09T09:43:00.569 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.560+0000 7fcbad01e740 1 Processor -- start 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.560+0000 7fcbad01e740 1 -- start start 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073604740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d073563340 con 0x55d0735ff250 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d07350b290 con 0x55d073604db0 2024-05-09T09:43:00.570 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcbad01e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d07355f300 con 0x55d0735ffc40 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba17fa640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 0x7fcb8c0a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba88ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba88ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37050/0 (socket says 172.21.15.69:37050) 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40568/0 (socket says 172.21.15.69:40568) 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba88ad640 1 -- 172.21.15.69:0/3306768188 learned_addr learned my addr 172.21.15.69:0/3306768188 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba3fff640 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073604740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba17fa640 1 -- 172.21.15.69:0/3756035752 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcb940cf7a0 con 0x7fcba4100140 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba88ad640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 0x7fcb8c0a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 -- 172.21.15.69:0/3306768188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 msgr2=0x55d073604740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073604740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 -- 172.21.15.69:0/3306768188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 msgr2=0x55d0735ff670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.571 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba37fe640 1 -- 172.21.15.69:0/3306768188 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d07352f8e0 con 0x55d073604db0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.561+0000 7fcba88ad640 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcba37fe640 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009fa0 tx=0x7fcb980118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eb8757c2570ba974 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcba88ad640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 0x7fcb8c0a50f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55d0736004a0 tx=0x7fcb9c02a8c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcb8a7fc640 1 -- 172.21.15.69:0/3306768188 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98004030 con 0x55d073604db0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcb8a7fc640 1 -- 172.21.15.69:0/3306768188 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb98005500 con 0x55d073604db0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcb8a7fc640 1 -- 172.21.15.69:0/3306768188 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98004d50 con 0x55d073604db0 2024-05-09T09:43:00.572 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcbad01e740 1 -- 172.21.15.69:0/3306768188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 msgr2=0x55d073607240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.562+0000 7fcbad01e740 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009fa0 tx=0x7fcb980118d0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 -- 172.21.15.69:0/3306768188 shutdown_connections 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d073607240 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073604740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 --2- 172.21.15.69:0/3306768188 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0735ff670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 -- 172.21.15.69:0/3306768188 >> 172.21.15.69:0/3306768188 conn(0x55d0735fa8c0 msgr2=0x55d0735facc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 -- 172.21.15.69:0/3306768188 shutdown_connections 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 -- 172.21.15.69:0/3306768188 wait complete. 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 -- start start 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 Processor -- start 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.563+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073609970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.573 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d0736109b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55d073607d60 con 0x55d073604db0 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55d07351e790 con 0x55d0735ff250 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55d073607ee0 con 0x55d0735ffc40 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcbad01e740 10 client.0 _ll_register_callbacks cb 0x55d073559a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:40572/0 (socket says 172.21.15.69:40572) 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba88ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d0736109b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 -- 172.21.15.69:0/626412527 learned_addr learned my addr 172.21.15.69:0/626412527 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba37fe640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073609970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 -- 172.21.15.69:0/626412527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 msgr2=0x55d073609970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073609970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 -- 172.21.15.69:0/626412527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 msgr2=0x55d0736109b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.574 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d0736109b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 -- 172.21.15.69:0/626412527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d073277210 con 0x55d0735ff250 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba88ad640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d0736109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcba3fff640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409bc10 tx=0x7fcb9400a440 comp rx=0 tx=0).ready entity=mon.2 client_cookie=de820adcdd29af9c server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.564+0000 7fcb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9400a250 con 0x55d0735ff250 2024-05-09T09:43:00.575 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb98005500 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcb9409bee0 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fcb9409bc80 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcbad01e740 10 client.27155 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.565+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55d073343620 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb98005500 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fcb940b4070 con 0x55d0735ff250 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 10 client.27155 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 10 client.27155 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 20 client.27155 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d073614f70 0x55d073617400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 4 client.27155 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55d073570a70 con 0x55d073614f70 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.566+0000 7fcba37fe640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d073614f70 0x55d073617400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.567+0000 7fcb837fe640 10 client.27155 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:00.576 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.567+0000 7fcba37fe640 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d073614f70 0x55d073617400 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=24bf1c5deac2b484 server_cookie=13ed61e23bb58911 in_seq=0 out_seq=0 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcb837fe640 1 -- 172.21.15.69:0/626412527 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55d073570a70 con 0x55d073614f70 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcb837fe640 10 client.27155 handle_command_reply: tid=0 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 client.27155 shutdown 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 msgr2=0x55d073609430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fcb9409bc10 tx=0x7fcb9400a440 comp rx=0 tx=0).stop 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 shutdown_connections 2024-05-09T09:43:00.578 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d0736109b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d073614f70 0x55d073617400 crc :-1 s=READY pgs=272 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073609970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 --2- 172.21.15.69:0/626412527 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d073609430 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 >> 172.21.15.69:0/626412527 conn(0x55d0735fa8c0 msgr2=0x55d0735f9a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcb837fe640 0 client.27155 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 shutdown_connections 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 1 -- 172.21.15.69:0/626412527 wait complete. 2024-05-09T09:43:00.579 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.573+0000 7fcbad01e740 20 client.27155 trim_cache size 0 max 16384 2024-05-09T09:43:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.598+0000 7fcbad01e740 1 Processor -- start 2024-05-09T09:43:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.598+0000 7fcbad01e740 1 -- start start 2024-05-09T09:43:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073610970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.599 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d07361cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b3120 con 0x55d0735ff250 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b0990 con 0x55d073604db0 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcbad01e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41ae7e0 con 0x55d0735ffc40 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37064/0 (socket says 172.21.15.69:37064) 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073610970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 -- 172.21.15.69:0/1101258264 learned_addr learned my addr 172.21.15.69:0/1101258264 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba88ad640 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d07361cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 -- 172.21.15.69:0/1101258264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 msgr2=0x55d073610970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073610970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.600 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 -- 172.21.15.69:0/1101258264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 msgr2=0x55d07361cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d07361cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 -- 172.21.15.69:0/1101258264 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55d073600020 con 0x55d0735ff250 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.599+0000 7fcba3fff640 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcb940980e0 tx=0x7fcb94004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ce15159e5c353c3 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.600+0000 7fcb89ffb640 1 -- 172.21.15.69:0/1101258264 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb940b9070 con 0x55d0735ff250 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.600+0000 7fcb89ffb640 1 -- 172.21.15.69:0/1101258264 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb9400a250 con 0x55d0735ff250 2024-05-09T09:43:00.601 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.600+0000 7fcb89ffb640 1 -- 172.21.15.69:0/1101258264 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb9409bee0 con 0x55d0735ff250 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.600+0000 7fcbad01e740 1 -- 172.21.15.69:0/1101258264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 msgr2=0x55d0736071a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.600+0000 7fcbad01e740 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 secure :-1 s=READY pgs=1117 cs=0 l=1 rev1=1 crypto rx=0x7fcb940980e0 tx=0x7fcb94004a40 comp rx=0 tx=0).stop 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- 172.21.15.69:0/1101258264 shutdown_connections 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d073604db0 0x55d07361cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d073610970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- 172.21.15.69:0/1101258264 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d0735ff250 0x55d0736071a0 unknown :-1 s=CLOSED pgs=1117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- 172.21.15.69:0/1101258264 >> 172.21.15.69:0/1101258264 conn(0x55d0735fa8c0 msgr2=0x55d073604460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- 172.21.15.69:0/1101258264 shutdown_connections 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- 172.21.15.69:0/1101258264 wait complete. 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- start start 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 Processor -- start 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d07360d690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.605 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d07360dbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d073679570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41aee50 con 0x55d073604db0 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41b02c0 con 0x55d0735ff250 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.601+0000 7fcbad01e740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fcba41af520 con 0x55d0735ffc40 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcbad01e740 10 client.0 _ll_register_callbacks cb 0x7fcb9409bb20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba88ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d073679570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d07360dbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba88ad640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d073679570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:37078/0 (socket says 172.21.15.69:37078) 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba88ad640 1 -- 172.21.15.69:0/263811913 learned_addr learned my addr 172.21.15.69:0/263811913 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba3fff640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d07360d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 -- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 msgr2=0x55d07360d690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d07360d690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.606 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 -- 172.21.15.69:0/263811913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 msgr2=0x55d073679570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d073679570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 -- 172.21.15.69:0/263811913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55d07360e240 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcba37fe640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d07360dbd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009730 tx=0x7fcb98009b50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b9e37da6a5119324 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcb837fe640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fcb98032060 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7fcb9400a250 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.602+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fcb98004180 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7fcb98039050 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 10 client.27094 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55d07367ac10 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fcb9400a250 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7fcb980035f0 con 0x55d0735ffc40 2024-05-09T09:43:00.607 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 10 client.27094 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 10 client.27094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 20 client.27094 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d07367b450 0x55d07367d870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 4 client.27094 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55d07356adc0 con 0x55d07367b450 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.603+0000 7fcba3fff640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d07367b450 0x55d07367d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.604+0000 7fcb837fe640 10 client.27094 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:00.608 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:00.604+0000 7fcba3fff640 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d07367b450 0x55d07367d870 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=98c5997dc6ac40ac server_cookie=4a501f1d7b1669da in_seq=0 out_seq=0 2024-05-09T09:43:00.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 333086 26.61 MB/sec execute 538 sec latency 175.599 ms 2024-05-09T09:43:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcb837fe640 1 -- 172.21.15.69:0/263811913 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55d07356adc0 con 0x55d07367b450 2024-05-09T09:43:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcb837fe640 10 client.27094 handle_command_reply: tid=0 2024-05-09T09:43:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 client.27094 shutdown 2024-05-09T09:43:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 msgr2=0x55d07360dbd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:01.395 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d07360dbd0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fcb98009730 tx=0x7fcb98009b50 comp rx=0 tx=0).stop 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 shutdown_connections 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55d073604db0 0x55d073679570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55d07367b450 0x55d07367d870 crc :-1 s=READY pgs=273 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcb837fe640 0 client.27094 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55d0735ffc40 0x55d07360dbd0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.396 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 --2- 172.21.15.69:0/263811913 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55d0735ff250 0x55d07360d690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.393+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 >> 172.21.15.69:0/263811913 conn(0x55d0735fa8c0 msgr2=0x55d073609310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.394+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 shutdown_connections 2024-05-09T09:43:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.394+0000 7fcbad01e740 1 -- 172.21.15.69:0/263811913 wait complete. 2024-05-09T09:43:01.397 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.394+0000 7fcbad01e740 20 client.27094 trim_cache size 0 max 16384 2024-05-09T09:43:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:43:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:43:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:43:01.398 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:43:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 msgr2=0x7fcb8c0a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 0x7fcb8c0a50f0 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x55d0736004a0 tx=0x7fcb9c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 msgr2=0x7fcba419e040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 secure :-1 s=READY pgs=1116 cs=0 l=1 rev1=1 crypto rx=0x7fcb9400e9f0 tx=0x7fcb9400edf0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.400 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 shutdown_connections 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fcb8c0a2c30 0x7fcb8c0a50f0 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fcba4105460 0x7fcba41a55f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fcba4102ab0 0x7fcba419e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 --2- 172.21.15.69:0/3756035752 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fcba4100140 0x7fcba419e040 unknown :-1 s=CLOSED pgs=1116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 >> 172.21.15.69:0/3756035752 conn(0x7fcba40fbd40 msgr2=0x7fcba4100df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:01.401 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 shutdown_connections 2024-05-09T09:43:01.402 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:01.395+0000 7fcbaa310640 1 -- 172.21.15.69:0/3756035752 wait complete. 2024-05-09T09:43:01.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:01 smithi069 ceph-mon[30123]: pgmap v1093: 129 pgs: 129 active+clean; 661 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.1 MiB/s wr, 478 op/s 2024-05-09T09:43:01.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:01 smithi163 ceph-mon[34354]: pgmap v1093: 129 pgs: 129 active+clean; 661 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.1 MiB/s wr, 478 op/s 2024-05-09T09:43:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:01 smithi103 ceph-mon[34421]: pgmap v1093: 129 pgs: 129 active+clean; 661 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 3.1 MiB/s wr, 478 op/s 2024-05-09T09:43:01.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 333494 26.61 MB/sec execute 539 sec latency 557.703 ms 2024-05-09T09:43:02.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334024 26.63 MB/sec execute 540 sec latency 68.673 ms 2024-05-09T09:43:03.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:03 smithi069 ceph-mon[30123]: pgmap v1094: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 466 op/s 2024-05-09T09:43:03.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:03 smithi163 ceph-mon[34354]: pgmap v1094: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 466 op/s 2024-05-09T09:43:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:03 smithi103 ceph-mon[34421]: pgmap v1094: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 3.4 MiB/s wr, 466 op/s 2024-05-09T09:43:03.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334110 26.58 MB/sec execute 541 sec latency 883.180 ms 2024-05-09T09:43:04.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334151 26.54 MB/sec execute 542 sec latency 1584.038 ms 2024-05-09T09:43:05.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:05 smithi069 ceph-mon[30123]: pgmap v1095: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 370 op/s 2024-05-09T09:43:05.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:05 smithi163 ceph-mon[34354]: pgmap v1095: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 370 op/s 2024-05-09T09:43:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:05 smithi103 ceph-mon[34421]: pgmap v1095: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.7 MiB/s rd, 2.8 MiB/s wr, 370 op/s 2024-05-09T09:43:05.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334381 26.50 MB/sec execute 543 sec latency 1493.283 ms 2024-05-09T09:43:06.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334555 26.47 MB/sec execute 544 sec latency 165.462 ms 2024-05-09T09:43:07.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:07 smithi069 ceph-mon[30123]: pgmap v1096: 129 pgs: 129 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.6 MiB/s wr, 571 op/s 2024-05-09T09:43:07.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:07 smithi163 ceph-mon[34354]: pgmap v1096: 129 pgs: 129 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.6 MiB/s wr, 571 op/s 2024-05-09T09:43:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:07 smithi103 ceph-mon[34421]: pgmap v1096: 129 pgs: 129 active+clean; 714 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.9 MiB/s rd, 7.6 MiB/s wr, 571 op/s 2024-05-09T09:43:07.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 334807 26.44 MB/sec execute 545 sec latency 187.606 ms 2024-05-09T09:43:08.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 335011 26.41 MB/sec execute 546 sec latency 216.051 ms 2024-05-09T09:43:09.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:09 smithi069 ceph-mon[30123]: pgmap v1097: 129 pgs: 129 active+clean; 725 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.5 MiB/s wr, 570 op/s 2024-05-09T09:43:09.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:09 smithi163 ceph-mon[34354]: pgmap v1097: 129 pgs: 129 active+clean; 725 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.5 MiB/s wr, 570 op/s 2024-05-09T09:43:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:09 smithi103 ceph-mon[34421]: pgmap v1097: 129 pgs: 129 active+clean; 725 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.9 MiB/s rd, 8.5 MiB/s wr, 570 op/s 2024-05-09T09:43:09.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 335618 26.44 MB/sec execute 547 sec latency 150.399 ms 2024-05-09T09:43:10.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 336328 26.44 MB/sec execute 548 sec latency 89.228 ms 2024-05-09T09:43:11.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:11 smithi069 ceph-mon[30123]: pgmap v1098: 129 pgs: 129 active+clean; 702 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 657 op/s 2024-05-09T09:43:11.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:11 smithi163 ceph-mon[34354]: pgmap v1098: 129 pgs: 129 active+clean; 702 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 657 op/s 2024-05-09T09:43:11.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:11 smithi103 ceph-mon[34421]: pgmap v1098: 129 pgs: 129 active+clean; 702 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.6 MiB/s rd, 10 MiB/s wr, 657 op/s 2024-05-09T09:43:11.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 336976 26.48 MB/sec execute 549 sec latency 69.992 ms 2024-05-09T09:43:12.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 337392 26.47 MB/sec execute 550 sec latency 125.543 ms 2024-05-09T09:43:13.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:13 smithi069 ceph-mon[30123]: pgmap v1099: 129 pgs: 129 active+clean; 667 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 731 op/s 2024-05-09T09:43:13.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:13 smithi163 ceph-mon[34354]: pgmap v1099: 129 pgs: 129 active+clean; 667 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 731 op/s 2024-05-09T09:43:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:13 smithi103 ceph-mon[34421]: pgmap v1099: 129 pgs: 129 active+clean; 667 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.0 MiB/s rd, 10 MiB/s wr, 731 op/s 2024-05-09T09:43:13.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 337811 26.46 MB/sec execute 551 sec latency 99.069 ms 2024-05-09T09:43:14.527 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 -- 172.21.15.69:0/905294146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98104c40 msgr2=0x7ffa98107030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/905294146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98104c40 0x7ffa98107030 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c009900 tx=0x7ffa8c02a410 comp rx=0 tx=0).stop 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 -- 172.21.15.69:0/905294146 shutdown_connections 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/905294146 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98107570 0x7ffa981099c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/905294146 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98104c40 0x7ffa98107030 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/905294146 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98102310 0x7ffa98104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 -- 172.21.15.69:0/905294146 >> 172.21.15.69:0/905294146 conn(0x7ffa980fbce0 msgr2=0x7ffa980fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 -- 172.21.15.69:0/905294146 shutdown_connections 2024-05-09T09:43:14.528 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.527+0000 7ffa9d08c640 1 -- 172.21.15.69:0/905294146 wait complete. 2024-05-09T09:43:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 Processor -- start 2024-05-09T09:43:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 -- start start 2024-05-09T09:43:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98102310 0x7ffa98197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98104c40 0x7ffa98198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.530 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9810d500 con 0x7ffa98107570 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9810d380 con 0x7ffa98102310 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9d08c640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa9810d680 con 0x7ffa98104c40 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98104c40 0x7ffa98198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43158/0 (socket says 172.21.15.69:43158) 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 -- 172.21.15.69:0/673352588 learned_addr learned my addr 172.21.15.69:0/673352588 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98102310 0x7ffa98197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 -- 172.21.15.69:0/673352588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98104c40 msgr2=0x7ffa98198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98104c40 0x7ffa98198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 -- 172.21.15.69:0/673352588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98102310 msgr2=0x7ffa98197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98102310 0x7ffa98197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.531 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.528+0000 7ffa9759e640 1 -- 172.21.15.69:0/673352588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7ffa9819f710 con 0x7ffa98107570 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa9759e640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000eb10 tx=0x7ffa9000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa7ffff640 1 -- 172.21.15.69:0/673352588 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90004b20 con 0x7ffa98107570 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7ffa9819fa00 con 0x7ffa98107570 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa7ffff640 1 -- 172.21.15.69:0/673352588 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa90004cc0 con 0x7ffa98107570 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa7ffff640 1 -- 172.21.15.69:0/673352588 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa90010760 con 0x7ffa98107570 2024-05-09T09:43:14.532 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.529+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7ffa9819fed0 con 0x7ffa98107570 2024-05-09T09:43:14.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.531+0000 7ffa7ffff640 1 -- 172.21.15.69:0/673352588 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7ffa90005020 con 0x7ffa98107570 2024-05-09T09:43:14.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 Processor -- start 2024-05-09T09:43:14.539 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 -- start start 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x556841427a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x55684142f650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556841387340 con 0x556841428050 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55684132f290 con 0x55684142d1c0 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556841383300 con 0x556841427660 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x55684142f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43166/0 (socket says 172.21.15.69:43166) 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 -- 172.21.15.69:0/979675090 learned_addr learned my addr 172.21.15.69:0/979675090 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa7ffff640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 0x7ffa800a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9659c640 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x556841427a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9759e640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 0x7ffa800a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.540 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 -- 172.21.15.69:0/979675090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 msgr2=0x556841427a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x556841427a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 -- 172.21.15.69:0/979675090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 msgr2=0x55684142f650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x55684142f650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa96d9d640 1 -- 172.21.15.69:0/979675090 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5568413538e0 con 0x556841428050 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa9659c640 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x55684142f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.532+0000 7ffa7ffff640 1 -- 172.21.15.69:0/673352588 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ffa900cf970 con 0x7ffa98107570 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009cf0 tx=0x7ffa88011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13589728ecf60259 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa7d7fa640 1 -- 172.21.15.69:0/979675090 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88004230 con 0x556841428050 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9759e640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 0x7ffa800a50f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa90098820 tx=0x7ffa9006ef10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa7d7fa640 1 -- 172.21.15.69:0/979675090 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa880043d0 con 0x556841428050 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa7d7fa640 1 -- 172.21.15.69:0/979675090 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa88019800 con 0x556841428050 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 -- 172.21.15.69:0/979675090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 msgr2=0x55684142cb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 crypto rx=0x7ffa88009cf0 tx=0x7ffa88011810 comp rx=0 tx=0).stop 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 -- 172.21.15.69:0/979675090 shutdown_connections 2024-05-09T09:43:14.541 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x55684142f650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841428050 0x55684142cb50 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.533+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/979675090 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x556841427a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- 172.21.15.69:0/979675090 >> 172.21.15.69:0/979675090 conn(0x55684141e8c0 msgr2=0x55684141ecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- 172.21.15.69:0/979675090 shutdown_connections 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- 172.21.15.69:0/979675090 wait complete. 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- start start 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 Processor -- start 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x5568414317c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841431d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x556841430080 con 0x55684142d1c0 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x556841342790 con 0x556841428050 2024-05-09T09:43:14.542 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x556841430200 con 0x556841427660 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9fd97740 10 client.0 _ll_register_callbacks cb 0x55684137da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43182/0 (socket says 172.21.15.69:43182) 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9659c640 1 -- 172.21.15.69:0/137900286 learned_addr learned my addr 172.21.15.69:0/137900286 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa9659c640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x5568414317c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.534+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841431d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 -- 172.21.15.69:0/137900286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 msgr2=0x5568414317c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x5568414317c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 -- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 msgr2=0x556841431d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841431d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 -- 172.21.15.69:0/137900286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55684109b210 con 0x55684142d1c0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9659c640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0314e0 tx=0x7ffa8c02a410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2312803b9af82af9 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.543 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c035070 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ffa8c004480 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa880043d0 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ffa8c0035f0 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9fd97740 10 client.17925 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556841167620 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.535+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa880043d0 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ffa8c03d7f0 con 0x55684142d1c0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 10 client.17925 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 10 client.17925 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 20 client.17925 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55684143d290 0x55684143f700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 4 client.17925 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x556841394a70 con 0x55684143d290 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.536+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55684143d290 0x55684143f700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.544 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.537+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55684143d290 0x55684143f700 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8578143987119992 server_cookie=9a7ec5ed07efe786 in_seq=0 out_seq=0 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.537+0000 7ffa7aff5640 10 client.17925 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.538+0000 7ffa7aff5640 1 -- 172.21.15.69:0/137900286 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x556841394a70 con 0x55684143d290 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.538+0000 7ffa7aff5640 10 client.17925 handle_command_reply: tid=0 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 client.17925 shutdown 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 msgr2=0x556841438d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 secure :-1 s=READY pgs=1121 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0314e0 tx=0x7ffa8c02a410 comp rx=0 tx=0).stop 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 shutdown_connections 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55684142d1c0 0x556841438d40 unknown :-1 s=CLOSED pgs=1121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841431d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55684143d290 0x55684143f700 crc :-1 s=READY pgs=274 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/137900286 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841427660 0x5568414317c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 >> 172.21.15.69:0/137900286 conn(0x55684141e8c0 msgr2=0x55684141da00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa7aff5640 0 client.17925 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 shutdown_connections 2024-05-09T09:43:14.545 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 1 -- 172.21.15.69:0/137900286 wait complete. 2024-05-09T09:43:14.546 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.539+0000 7ffa9fd97740 20 client.17925 trim_cache size 0 max 16384 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 Processor -- start 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 -- start start 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841438d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55684142d1c0 0x556841421f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981aaa30 con 0x556841427660 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981a82a0 con 0x556841428050 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.557+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981a60f0 con 0x55684142d1c0 2024-05-09T09:43:14.562 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa96d9d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841438d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43196/0 (socket says 172.21.15.69:43196) 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 -- 172.21.15.69:0/1122289101 learned_addr learned my addr 172.21.15.69:0/1122289101 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55684142d1c0 0x556841421f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 -- 172.21.15.69:0/1122289101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55684142d1c0 msgr2=0x556841421f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55684142d1c0 0x556841421f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 -- 172.21.15.69:0/1122289101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 msgr2=0x556841438d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841438d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 -- 172.21.15.69:0/1122289101 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x556841428d20 con 0x556841427660 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa9659c640 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0318a0 tx=0x7ffa8c031b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=13cb0c85a01a450b server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa7cff9640 1 -- 172.21.15.69:0/1122289101 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c035040 con 0x556841427660 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa7cff9640 1 -- 172.21.15.69:0/1122289101 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8c042070 con 0x556841427660 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.558+0000 7ffa7cff9640 1 -- 172.21.15.69:0/1122289101 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c004b90 con 0x556841427660 2024-05-09T09:43:14.563 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- 172.21.15.69:0/1122289101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 msgr2=0x55684142c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0318a0 tx=0x7ffa8c031b60 comp rx=0 tx=0).stop 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- 172.21.15.69:0/1122289101 shutdown_connections 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55684142d1c0 0x556841421f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x556841428050 0x556841438d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/1122289101 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x55684142c8a0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- 172.21.15.69:0/1122289101 >> 172.21.15.69:0/1122289101 conn(0x55684141e8c0 msgr2=0x556841421870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- 172.21.15.69:0/1122289101 shutdown_connections 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- 172.21.15.69:0/1122289101 wait complete. 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 -- start start 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.559+0000 7ffa9fd97740 1 Processor -- start 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 0x556841436840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x556841437800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981a6760 con 0x556841427660 2024-05-09T09:43:14.564 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981a7bd0 con 0x55684142d1c0 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7ffa981a6e30 con 0x556841428050 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 10 client.0 _ll_register_callbacks cb 0x7ffa8c032da0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:43206/0 (socket says 172.21.15.69:43206) 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 -- 172.21.15.69:0/544861223 learned_addr learned my addr 172.21.15.69:0/544861223 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 0x556841436840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x556841437800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 -- 172.21.15.69:0/544861223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 msgr2=0x556841436840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 0x556841436840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 -- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 msgr2=0x556841437800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x556841437800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 -- 172.21.15.69:0/544861223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x556841437df0 con 0x556841427660 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 0x556841436840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:14.565 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9659c640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0314e0 tx=0x7ffa8c005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c29cfebe97c076ae server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa7aff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7ffa8c035030 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7ffa8c003dd0 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.560+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7ffa8c042070 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7ffa8c03d950 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 10 client.17931 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x556841439100 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7ffa8c042070 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7ffa8c005320 con 0x556841427660 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 10 client.17931 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 10 client.17931 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 20 client.17931 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5568414a3f90 0x5568414a63b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:14.566 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 4 client.17931 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:43:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55684138edc0 con 0x5568414a3f90 2024-05-09T09:43:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.561+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5568414a3f90 0x5568414a63b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.562+0000 7ffa96d9d640 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5568414a3f90 0x5568414a63b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=68bdc3b4e201e501 server_cookie=2b374ace4943b92c in_seq=0 out_seq=0 2024-05-09T09:43:14.567 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:14.562+0000 7ffa7aff5640 10 client.17931 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:14.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 338210 26.44 MB/sec execute 552 sec latency 260.451 ms 2024-05-09T09:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.102+0000 7ffa7aff5640 1 -- 172.21.15.69:0/544861223 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55684138edc0 con 0x5568414a3f90 2024-05-09T09:43:15.104 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.102+0000 7ffa7aff5640 10 client.17931 handle_command_reply: tid=0 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.102+0000 7ffa9fd97740 1 client.17931 shutdown 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 msgr2=0x556841436300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 crypto rx=0x7ffa8c0314e0 tx=0x7ffa8c005c90 comp rx=0 tx=0).stop 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 shutdown_connections 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55684142d1c0 0x556841437800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5568414a3f90 0x5568414a63b0 crc :-1 s=READY pgs=275 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa7aff5640 0 client.17931 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x556841428050 0x556841436840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 --2- 172.21.15.69:0/544861223 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x556841427660 0x556841436300 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 >> 172.21.15.69:0/544861223 conn(0x55684141e8c0 msgr2=0x556841421870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.103+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 shutdown_connections 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.104+0000 7ffa9fd97740 1 -- 172.21.15.69:0/544861223 wait complete. 2024-05-09T09:43:15.105 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.104+0000 7ffa9fd97740 20 client.17931 trim_cache size 0 max 16384 2024-05-09T09:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:43:15.106 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.105+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 msgr2=0x7ffa800a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.105+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 0x7ffa800a50f0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7ffa90098820 tx=0x7ffa9006ef10 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.105+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 msgr2=0x7ffa9819f0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.105+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 crypto rx=0x7ffa9000eb10 tx=0x7ffa9000c2d0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 shutdown_connections 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7ffa800a2c30 0x7ffa800a50f0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7ffa98107570 0x7ffa9819f0a0 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7ffa98104c40 0x7ffa98198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 --2- 172.21.15.69:0/673352588 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7ffa98102310 0x7ffa98197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 >> 172.21.15.69:0/673352588 conn(0x7ffa980fbce0 msgr2=0x7ffa98102d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 shutdown_connections 2024-05-09T09:43:15.107 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:15.106+0000 7ffa9d08c640 1 -- 172.21.15.69:0/673352588 wait complete. 2024-05-09T09:43:15.511 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:15 smithi069 ceph-mon[30123]: pgmap v1100: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-05-09T09:43:15.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:15 smithi163 ceph-mon[34354]: pgmap v1100: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-05-09T09:43:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:15 smithi103 ceph-mon[34421]: pgmap v1100: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 MiB/s rd, 10 MiB/s wr, 686 op/s 2024-05-09T09:43:15.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 338622 26.44 MB/sec execute 553 sec latency 372.749 ms 2024-05-09T09:43:16.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 339352 26.44 MB/sec execute 554 sec latency 70.787 ms 2024-05-09T09:43:17.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:17 smithi069 ceph-mon[30123]: pgmap v1101: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-05-09T09:43:17.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:17 smithi163 ceph-mon[34354]: pgmap v1101: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-05-09T09:43:17.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:17 smithi103 ceph-mon[34421]: pgmap v1101: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 10 MiB/s wr, 716 op/s 2024-05-09T09:43:17.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 340052 26.48 MB/sec execute 555 sec latency 44.501 ms 2024-05-09T09:43:18.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 340667 26.51 MB/sec execute 556 sec latency 81.940 ms 2024-05-09T09:43:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:19 smithi069 ceph-mon[30123]: pgmap v1102: 129 pgs: 129 active+clean; 673 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2024-05-09T09:43:19.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:19 smithi163 ceph-mon[34354]: pgmap v1102: 129 pgs: 129 active+clean; 673 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2024-05-09T09:43:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:19 smithi103 ceph-mon[34421]: pgmap v1102: 129 pgs: 129 active+clean; 673 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.2 MiB/s rd, 5.6 MiB/s wr, 543 op/s 2024-05-09T09:43:19.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 341119 26.50 MB/sec execute 557 sec latency 72.944 ms 2024-05-09T09:43:20.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 341484 26.47 MB/sec execute 558 sec latency 113.621 ms 2024-05-09T09:43:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:21 smithi069 ceph-mon[30123]: pgmap v1103: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 599 op/s 2024-05-09T09:43:21.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:21 smithi163 ceph-mon[34354]: pgmap v1103: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 599 op/s 2024-05-09T09:43:21.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:21 smithi103 ceph-mon[34421]: pgmap v1103: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 4.7 MiB/s wr, 599 op/s 2024-05-09T09:43:21.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 342047 26.49 MB/sec execute 559 sec latency 67.472 ms 2024-05-09T09:43:22.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 342787 26.50 MB/sec execute 560 sec latency 56.441 ms 2024-05-09T09:43:23.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:23 smithi069 ceph-mon[30123]: pgmap v1104: 129 pgs: 129 active+clean; 654 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 533 op/s 2024-05-09T09:43:23.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:23 smithi163 ceph-mon[34354]: pgmap v1104: 129 pgs: 129 active+clean; 654 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 533 op/s 2024-05-09T09:43:23.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:23 smithi103 ceph-mon[34421]: pgmap v1104: 129 pgs: 129 active+clean; 654 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 3.3 MiB/s wr, 533 op/s 2024-05-09T09:43:23.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 343326 26.51 MB/sec execute 561 sec latency 88.749 ms 2024-05-09T09:43:24.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 343966 26.53 MB/sec execute 562 sec latency 181.587 ms 2024-05-09T09:43:25.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:25 smithi069 ceph-mon[30123]: pgmap v1105: 129 pgs: 129 active+clean; 658 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 441 op/s 2024-05-09T09:43:25.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:25 smithi163 ceph-mon[34354]: pgmap v1105: 129 pgs: 129 active+clean; 658 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 441 op/s 2024-05-09T09:43:25.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:25 smithi103 ceph-mon[34421]: pgmap v1105: 129 pgs: 129 active+clean; 658 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 2.9 MiB/s wr, 441 op/s 2024-05-09T09:43:25.845 INFO:tasks.workunit.client.0.smithi069.stdout: 10 344390 26.53 MB/sec execute 563 sec latency 72.753 ms 2024-05-09T09:43:26.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 344925 26.52 MB/sec execute 564 sec latency 77.095 ms 2024-05-09T09:43:27.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:27 smithi069 ceph-mon[30123]: pgmap v1106: 129 pgs: 129 active+clean; 664 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 448 op/s 2024-05-09T09:43:27.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:27 smithi163 ceph-mon[34354]: pgmap v1106: 129 pgs: 129 active+clean; 664 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 448 op/s 2024-05-09T09:43:27.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:27 smithi103 ceph-mon[34421]: pgmap v1106: 129 pgs: 129 active+clean; 664 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.4 MiB/s rd, 3.1 MiB/s wr, 448 op/s 2024-05-09T09:43:27.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 345445 26.53 MB/sec execute 565 sec latency 101.009 ms 2024-05-09T09:43:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 -- 172.21.15.69:0/1250249968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15841018b0 msgr2=0x7f1584101cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 --2- 172.21.15.69:0/1250249968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15841018b0 0x7f1584101cd0 secure :-1 s=READY pgs=1124 cs=0 l=1 rev1=1 crypto rx=0x7f1574009900 tx=0x7f157402a410 comp rx=0 tx=0).stop 2024-05-09T09:43:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 -- 172.21.15.69:0/1250249968 shutdown_connections 2024-05-09T09:43:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 --2- 172.21.15.69:0/1250249968 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584103460 0x7f1584109cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 --2- 172.21.15.69:0/1250249968 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1584102aa0 0x7f1584102f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 --2- 172.21.15.69:0/1250249968 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f15841018b0 0x7f1584101cd0 unknown :-1 s=CLOSED pgs=1124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 -- 172.21.15.69:0/1250249968 >> 172.21.15.69:0/1250249968 conn(0x7f15840fce20 msgr2=0x7f15840ff260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 -- 172.21.15.69:0/1250249968 shutdown_connections 2024-05-09T09:43:28.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.232+0000 7f1589721640 1 -- 172.21.15.69:0/1250249968 wait complete. 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.233+0000 7f1589721640 1 Processor -- start 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.233+0000 7f1589721640 1 -- start start 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.233+0000 7f1589721640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.233+0000 7f1589721640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584102aa0 0x7f1584073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.233+0000 7f1589721640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1584103460 0x7f1584074160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f15827fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584102aa0 0x7f1584073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54840/0 (socket says 172.21.15.69:54840) 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 -- 172.21.15.69:0/3496434134 learned_addr learned my addr 172.21.15.69:0/3496434134 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1589721640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f158410d580 con 0x7f1584103460 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f15837fe640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1584103460 0x7f1584074160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f158410d400 con 0x7f15841018b0 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f158410d700 con 0x7f1584102aa0 2024-05-09T09:43:28.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 -- 172.21.15.69:0/3496434134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584102aa0 msgr2=0x7f1584073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584102aa0 0x7f1584073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 -- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1584103460 msgr2=0x7f1584074160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1584103460 0x7f1584074160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 -- 172.21.15.69:0/3496434134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f15840746a0 con 0x7f15841018b0 2024-05-09T09:43:28.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.234+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1574031440 tx=0x7f1574032ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.235+0000 7f156bfff640 1 -- 172.21.15.69:0/3496434134 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157403f030 con 0x7f15841018b0 2024-05-09T09:43:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.235+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1584074930 con 0x7f15841018b0 2024-05-09T09:43:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.235+0000 7f156bfff640 1 -- 172.21.15.69:0/3496434134 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1574003ce0 con 0x7f15841018b0 2024-05-09T09:43:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.235+0000 7f156bfff640 1 -- 172.21.15.69:0/3496434134 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574035070 con 0x7f15841018b0 2024-05-09T09:43:28.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.235+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f15841a6540 con 0x7f15841018b0 2024-05-09T09:43:28.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.237+0000 7f156bfff640 1 -- 172.21.15.69:0/3496434134 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1574042030 con 0x7f15841018b0 2024-05-09T09:43:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f156bfff640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 0x7f156c0a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 Processor -- start 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 -- start start 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 0x55dcf81a1740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf81a1db0 0x55dcf81a4240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dcf8100340 con 0x55dcf81a1db0 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dcf80a8290 con 0x55dcf819c250 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f158c42a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dcf80fc300 con 0x55dcf819cc40 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f156bfff640 1 -- 172.21.15.69:0/3496434134 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f15740ef830 con 0x7f15841018b0 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f15827fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 0x55dcf81a1740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f15827fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 0x55dcf81a1740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:58662/0 (socket says 172.21.15.69:58662) 2024-05-09T09:43:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54842/0 (socket says 172.21.15.69:54842) 2024-05-09T09:43:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f15827fc640 1 -- 172.21.15.69:0/3286170303 learned_addr learned my addr 172.21.15.69:0/3286170303 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:28.241 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f15837fe640 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf81a1db0 0x55dcf81a4240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f1582ffd640 1 -- 172.21.15.69:0/3286170303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 msgr2=0x55dcf81a1740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 0x55dcf81a1740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.239+0000 7f1582ffd640 1 -- 172.21.15.69:0/3286170303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf81a1db0 msgr2=0x55dcf81a4240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf81a1db0 0x55dcf81a4240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f1582ffd640 1 -- 172.21.15.69:0/3286170303 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf80cc8e0 con 0x55dcf819c250 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f15740ed000 tx=0x7f15740bb7e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d860687fe16d16e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f15697fa640 1 -- 172.21.15.69:0/3286170303 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1574049040 con 0x55dcf819c250 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f15697fa640 1 -- 172.21.15.69:0/3286170303 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15740c4980 con 0x55dcf819c250 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.240+0000 7f15697fa640 1 -- 172.21.15.69:0/3286170303 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157403b660 con 0x55dcf819c250 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f15827fc640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 0x7f156c0a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 -- 172.21.15.69:0/3286170303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 msgr2=0x55dcf819c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f15740ed000 tx=0x7f15740bb7e0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 -- 172.21.15.69:0/3286170303 shutdown_connections 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf81a1db0 0x55dcf81a4240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819cc40 0x55dcf81a1740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 --2- 172.21.15.69:0/3286170303 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf819c250 0x55dcf819c670 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 -- 172.21.15.69:0/3286170303 >> 172.21.15.69:0/3286170303 conn(0x55dcf81978c0 msgr2=0x55dcf8197cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 -- 172.21.15.69:0/3286170303 shutdown_connections 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f158c42a740 1 -- 172.21.15.69:0/3286170303 wait complete. 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.241+0000 7f15827fc640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 0x7f156c0a5170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dcf819d040 tx=0x7f1578013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 -- start start 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 Processor -- start 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81a61b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81a66f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55dcf81a4b20 con 0x55dcf819cc40 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55dcf80bb790 con 0x55dcf81a1db0 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f158c42a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55dcf81a4ca0 con 0x55dcf819c250 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f15827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81a66f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f15827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81a66f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44764/0 (socket says 172.21.15.69:44764) 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f15827fc640 1 -- 172.21.15.69:0/3213451187 learned_addr learned my addr 172.21.15.69:0/3213451187 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.242+0000 7f15837fe640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81a61b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f158c42a740 10 client.0 _ll_register_callbacks cb 0x55dcf80f6a20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 -- 172.21.15.69:0/3213451187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 msgr2=0x55dcf81a61b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81a61b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 -- 172.21.15.69:0/3213451187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 msgr2=0x55dcf81a66f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81a66f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 -- 172.21.15.69:0/3213451187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcf7e14210 con 0x55dcf81a1db0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f15837fe640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f157c005fd0 tx=0x7f157c013e80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fd5f77fd180bf5e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.243+0000 7f1566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.103:3300/0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c003970 con 0x55dcf81a1db0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mon.2 v2:172.21.15.103:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f157c003b10 con 0x55dcf81a1db0 2024-05-09T09:43:28.247 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f15740c4980 con 0x55dcf81a1db0 2024-05-09T09:43:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f157c003d10 con 0x55dcf81a1db0 2024-05-09T09:43:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f158c42a740 10 client.27191 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.244+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55dcf7ee0620 con 0x55dcf81a1db0 2024-05-09T09:43:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mon.2 v2:172.21.15.103:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f15740c4980 con 0x55dcf81a1db0 2024-05-09T09:43:28.248 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mon.2 v2:172.21.15.103:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f157c01a070 con 0x55dcf81a1db0 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 10 client.27191 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 10 client.27191 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 20 client.27191 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf81b1cb0 0x55dcf81b4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 4 client.27191 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.245+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55dcf810da70 con 0x55dcf81b1cb0 2024-05-09T09:43:28.249 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.246+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf81b1cb0 0x55dcf81b4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.250+0000 7f1582ffd640 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf81b1cb0 0x55dcf81b4150 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e7347fe8b2b1012 server_cookie=1d331efa99a1c529 in_seq=0 out_seq=0 2024-05-09T09:43:28.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.250+0000 7f1566ff5640 10 client.27191 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:28.251 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.251+0000 7f1566ff5640 1 -- 172.21.15.69:0/3213451187 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55dcf810da70 con 0x55dcf81b1cb0 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.251+0000 7f1566ff5640 10 client.27191 handle_command_reply: tid=0 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.251+0000 7f158c42a740 1 client.27191 shutdown 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 msgr2=0x55dcf81ad730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad730 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f157c005fd0 tx=0x7f157c013e80 comp rx=0 tx=0).stop 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 shutdown_connections 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad730 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.252 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81a66f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf81b1cb0 0x55dcf81b4150 crc :-1 s=READY pgs=276 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 --2- 172.21.15.69:0/3213451187 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81a61b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f1566ff5640 0 client.27191 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:28.253 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 >> 172.21.15.69:0/3213451187 conn(0x55dcf81978c0 msgr2=0x55dcf8196a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.252+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 shutdown_connections 2024-05-09T09:43:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.253+0000 7f158c42a740 1 -- 172.21.15.69:0/3213451187 wait complete. 2024-05-09T09:43:28.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.253+0000 7f158c42a740 20 client.27191 trim_cache size 0 max 16384 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 Processor -- start 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 -- start start 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81b9a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81b9f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15841b33e0 con 0x55dcf819cc40 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15841b0c50 con 0x55dcf81a1db0 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.277+0000 7f158c42a740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15841aeaa0 con 0x55dcf819c250 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f1582ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81b9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.279 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:54868/0 (socket says 172.21.15.69:54868) 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 -- 172.21.15.69:0/2527732433 learned_addr learned my addr 172.21.15.69:0/2527732433 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15827fc640 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81b9f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 -- 172.21.15.69:0/2527732433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 msgr2=0x55dcf81b9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81b9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 -- 172.21.15.69:0/2527732433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 msgr2=0x55dcf81b9f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81b9f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 -- 172.21.15.69:0/2527732433 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55dcf8198630 con 0x55dcf81a1db0 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f15837fe640 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f157c011f90 tx=0x7f157c013980 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d8dd75cb034e5c9e server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.278+0000 7f1568ff9640 1 -- 172.21.15.69:0/2527732433 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c02d020 con 0x55dcf81a1db0 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.279+0000 7f1568ff9640 1 -- 172.21.15.69:0/2527732433 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f157c01bcb0 con 0x55dcf81a1db0 2024-05-09T09:43:28.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.279+0000 7f1568ff9640 1 -- 172.21.15.69:0/2527732433 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157c029440 con 0x55dcf81a1db0 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.279+0000 7f158c42a740 1 -- 172.21.15.69:0/2527732433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 msgr2=0x55dcf81ad6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.279+0000 7f158c42a740 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f157c011f90 tx=0x7f157c013980 comp rx=0 tx=0).stop 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 -- 172.21.15.69:0/2527732433 shutdown_connections 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ad6f0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81b9f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.283 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 --2- 172.21.15.69:0/2527732433 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81b9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 -- 172.21.15.69:0/2527732433 >> 172.21.15.69:0/2527732433 conn(0x55dcf81978c0 msgr2=0x55dcf8197ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 -- 172.21.15.69:0/2527732433 shutdown_connections 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 -- 172.21.15.69:0/2527732433 wait complete. 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 -- start start 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.280+0000 7f158c42a740 1 Processor -- start 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81aabb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ac100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f15827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.284 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f15827fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:44782/0 (socket says 172.21.15.69:44782) 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f15827fc640 1 -- 172.21.15.69:0/4136414943 learned_addr learned my addr 172.21.15.69:0/4136414943 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f1582ffd640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81aabb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f15841af110 con 0x55dcf819cc40 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f15841b0580 con 0x55dcf81a1db0 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f15841af7e0 con 0x55dcf819c250 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.281+0000 7f158c42a740 10 client.0 _ll_register_callbacks cb 0x7f157c009b90 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 -- 172.21.15.69:0/4136414943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 msgr2=0x55dcf81aabb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81aabb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 -- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 msgr2=0x55dcf81ac100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15837fe640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ac100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ac100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:28.285 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55dcf81ac6f0 con 0x55dcf819cc40 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15837fe640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f15827fc640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1578004450 tx=0x7f15780046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4cc2f742f4e417b4 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f1566ff5640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.282+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f157802c060 con 0x55dcf819cc40 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1578027850 con 0x55dcf819cc40 2024-05-09T09:43:28.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f157c01bcb0 con 0x55dcf819cc40 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f1578033050 con 0x55dcf819cc40 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f158c42a740 10 client.17949 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55dcf81ada60 con 0x55dcf819cc40 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f157c01bcb0 con 0x55dcf819cc40 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.283+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f157801bba0 con 0x55dcf819cc40 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 10 client.17949 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 10 client.17949 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 20 client.17949 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:28.288 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf8218a60 0x55dcf821ae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 4 client.17949 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:43:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55dcf8107dc0 con 0x55dcf8218a60 2024-05-09T09:43:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f1582ffd640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf8218a60 0x55dcf821ae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f1566ff5640 10 client.17949 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:28.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:28.284+0000 7f1582ffd640 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf8218a60 0x55dcf821ae80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=330ccfe3d0167b92 server_cookie=aacd84160a022cc0 in_seq=0 out_seq=0 2024-05-09T09:43:28.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 345838 26.51 MB/sec execute 566 sec latency 507.844 ms 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.080+0000 7f1566ff5640 1 -- 172.21.15.69:0/4136414943 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55dcf8107dc0 con 0x55dcf8218a60 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.080+0000 7f1566ff5640 10 client.17949 handle_command_reply: tid=0 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.080+0000 7f158c42a740 1 client.17949 shutdown 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 msgr2=0x55dcf81ab0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 crypto rx=0x7f1578004450 tx=0x7f15780046c0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.082 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 shutdown_connections 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55dcf81a1db0 0x55dcf81ac100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55dcf819cc40 0x55dcf81ab0f0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55dcf8218a60 0x55dcf821ae80 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f1566ff5640 0 client.17949 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 --2- 172.21.15.69:0/4136414943 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55dcf819c250 0x55dcf81aabb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.081+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 >> 172.21.15.69:0/4136414943 conn(0x55dcf81978c0 msgr2=0x55dcf81a6170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.082+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 shutdown_connections 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.082+0000 7f158c42a740 1 -- 172.21.15.69:0/4136414943 wait complete. 2024-05-09T09:43:29.083 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.082+0000 7f158c42a740 20 client.17949 trim_cache size 0 max 16384 2024-05-09T09:43:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:43:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:43:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:43:29.084 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:43:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 msgr2=0x7f156c0a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 0x7f156c0a5170 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x55dcf819d040 tx=0x7f1578013040 comp rx=0 tx=0).stop 2024-05-09T09:43:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 msgr2=0x7f15840755e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:29.085 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1574031440 tx=0x7f1574032ce0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 shutdown_connections 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f156c0a2cb0 0x7f156c0a5170 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1584103460 0x7f1584074160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1584102aa0 0x7f1584073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 --2- 172.21.15.69:0/3496434134 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f15841018b0 0x7f15840755e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 >> 172.21.15.69:0/3496434134 conn(0x7f15840fce20 msgr2=0x7f1584105d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.083+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 shutdown_connections 2024-05-09T09:43:29.086 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:29.084+0000 7f1589721640 1 -- 172.21.15.69:0/3496434134 wait complete. 2024-05-09T09:43:29.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:29 smithi069 ceph-mon[30123]: pgmap v1107: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 451 op/s 2024-05-09T09:43:29.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:29 smithi163 ceph-mon[34354]: pgmap v1107: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 451 op/s 2024-05-09T09:43:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:29 smithi103 ceph-mon[34421]: pgmap v1107: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 3.2 MiB/s wr, 451 op/s 2024-05-09T09:43:29.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 346349 26.53 MB/sec execute 567 sec latency 735.971 ms 2024-05-09T09:43:30.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 347021 26.54 MB/sec execute 568 sec latency 67.521 ms 2024-05-09T09:43:31.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:31 smithi069 ceph-mon[30123]: pgmap v1108: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:43:31.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:31 smithi163 ceph-mon[34354]: pgmap v1108: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:43:31.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:31 smithi103 ceph-mon[34421]: pgmap v1108: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.0 MiB/s rd, 3.4 MiB/s wr, 501 op/s 2024-05-09T09:43:31.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 347474 26.53 MB/sec execute 569 sec latency 91.332 ms 2024-05-09T09:43:32.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 347967 26.54 MB/sec execute 570 sec latency 80.151 ms 2024-05-09T09:43:33.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:33 smithi069 ceph-mon[30123]: pgmap v1109: 129 pgs: 129 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 521 op/s 2024-05-09T09:43:33.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:33 smithi163 ceph-mon[34354]: pgmap v1109: 129 pgs: 129 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 521 op/s 2024-05-09T09:43:33.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:33 smithi103 ceph-mon[34421]: pgmap v1109: 129 pgs: 129 active+clean; 681 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.1 MiB/s rd, 3.4 MiB/s wr, 521 op/s 2024-05-09T09:43:33.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 348498 26.54 MB/sec execute 571 sec latency 105.992 ms 2024-05-09T09:43:34.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 349110 26.53 MB/sec execute 572 sec latency 92.170 ms 2024-05-09T09:43:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:35 smithi069 ceph-mon[30123]: pgmap v1110: 129 pgs: 129 active+clean; 676 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 505 op/s 2024-05-09T09:43:35.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:35 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:43:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:35 smithi163 ceph-mon[34354]: pgmap v1110: 129 pgs: 129 active+clean; 676 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 505 op/s 2024-05-09T09:43:35.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:35 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:43:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:35 smithi103 ceph-mon[34421]: pgmap v1110: 129 pgs: 129 active+clean; 676 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.2 MiB/s rd, 3.2 MiB/s wr, 505 op/s 2024-05-09T09:43:35.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:35 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:43:35.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 349714 26.57 MB/sec execute 573 sec latency 67.519 ms 2024-05-09T09:43:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:43:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:43:36.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:43:36.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:43:36.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:43:36.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:43:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:43:36.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:43:36.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:43:36.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 350455 26.58 MB/sec execute 574 sec latency 51.176 ms 2024-05-09T09:43:37.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:37 smithi069 ceph-mon[30123]: pgmap v1111: 129 pgs: 129 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:43:37.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:37 smithi163 ceph-mon[34354]: pgmap v1111: 129 pgs: 129 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:43:37.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:37 smithi103 ceph-mon[34421]: pgmap v1111: 129 pgs: 129 active+clean; 683 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 8.5 MiB/s rd, 3.4 MiB/s wr, 516 op/s 2024-05-09T09:43:37.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 350910 26.58 MB/sec execute 575 sec latency 93.375 ms 2024-05-09T09:43:38.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351368 26.58 MB/sec execute 576 sec latency 91.100 ms 2024-05-09T09:43:39.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:39 smithi069 ceph-mon[30123]: pgmap v1112: 129 pgs: 129 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.0 MiB/s wr, 469 op/s 2024-05-09T09:43:39.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:39 smithi163 ceph-mon[34354]: pgmap v1112: 129 pgs: 129 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.0 MiB/s wr, 469 op/s 2024-05-09T09:43:39.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:39 smithi103 ceph-mon[34421]: pgmap v1112: 129 pgs: 129 active+clean; 684 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.0 MiB/s rd, 3.0 MiB/s wr, 469 op/s 2024-05-09T09:43:39.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351397 26.54 MB/sec execute 577 sec latency 1052.708 ms 2024-05-09T09:43:40.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351398 26.49 MB/sec execute 578 sec latency 2052.750 ms 2024-05-09T09:43:41.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:41 smithi069 ceph-mon[30123]: pgmap v1113: 129 pgs: 129 active+clean; 693 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 525 op/s 2024-05-09T09:43:41.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:41 smithi163 ceph-mon[34354]: pgmap v1113: 129 pgs: 129 active+clean; 693 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 525 op/s 2024-05-09T09:43:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:41 smithi103 ceph-mon[34421]: pgmap v1113: 129 pgs: 129 active+clean; 693 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.6 MiB/s rd, 3.5 MiB/s wr, 525 op/s 2024-05-09T09:43:41.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351488 26.45 MB/sec execute 579 sec latency 3052.793 ms 2024-05-09T09:43:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 -- 172.21.15.69:0/3878272000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc107610 msgr2=0x7f60bc109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 --2- 172.21.15.69:0/3878272000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc107610 0x7f60bc109a00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009ca0 tx=0x7f60b402a8d0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 -- 172.21.15.69:0/3878272000 shutdown_connections 2024-05-09T09:43:42.228 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 --2- 172.21.15.69:0/3878272000 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc107610 0x7f60bc109a00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 --2- 172.21.15.69:0/3878272000 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc104ce0 0x7f60bc1070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 --2- 172.21.15.69:0/3878272000 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc1023b0 0x7f60bc1047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 -- 172.21.15.69:0/3878272000 >> 172.21.15.69:0/3878272000 conn(0x7f60bc0fbd00 msgr2=0x7f60bc0fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 -- 172.21.15.69:0/3878272000 shutdown_connections 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.228+0000 7f60c4742640 1 -- 172.21.15.69:0/3878272000 wait complete. 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 Processor -- start 2024-05-09T09:43:42.229 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 -- start start 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc104ce0 0x7f60bc197ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc107610 0x7f60bc19ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10d5a0 con 0x7f60bc1023b0 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10d420 con 0x7f60bc104ce0 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.229+0000 7f60c4742640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc10d720 con 0x7f60bc107610 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47090/0 (socket says 172.21.15.69:47090) 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 -- 172.21.15.69:0/1735848314 learned_addr learned my addr 172.21.15.69:0/1735848314 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc104ce0 0x7f60bc197ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.231 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c2cdf640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc107610 0x7f60bc19ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 -- 172.21.15.69:0/1735848314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc107610 msgr2=0x7f60bc19ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc107610 0x7f60bc19ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 -- 172.21.15.69:0/1735848314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc104ce0 msgr2=0x7f60bc197ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc104ce0 0x7f60bc197ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 -- 172.21.15.69:0/1735848314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f60bc19f5c0 con 0x7f60bc1023b0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60c24de640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f60b800b450 tx=0x7f60b800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60b37fe640 1 -- 172.21.15.69:0/1735848314 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b8015020 con 0x7f60bc1023b0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60b37fe640 1 -- 172.21.15.69:0/1735848314 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60b8011d70 con 0x7f60bc1023b0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.230+0000 7f60b37fe640 1 -- 172.21.15.69:0/1735848314 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b800f340 con 0x7f60bc1023b0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.231+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f60bc19f8b0 con 0x7f60bc1023b0 2024-05-09T09:43:42.232 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.231+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f60bc19fe30 con 0x7f60bc1023b0 2024-05-09T09:43:42.233 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.234+0000 7f60b37fe640 1 -- 172.21.15.69:0/1735848314 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f60b8018020 con 0x7f60bc1023b0 2024-05-09T09:43:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60b37fe640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 0x7f60a80a5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 Processor -- start 2024-05-09T09:43:42.234 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60b37fe640 1 -- 172.21.15.69:0/1735848314 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f60b80cedc0 con 0x7f60bc1023b0 2024-05-09T09:43:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 0x7f60a80a5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 -- start start 2024-05-09T09:43:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbc3670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.235 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbcb240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.235+0000 7f60c7450740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5617abb27340 con 0x5617abbc3c40 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c7450740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5617abacf290 con 0x5617abbc8db0 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c7450740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5617abb23300 con 0x5617abbc3250 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 0x7f60a80a5170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f60bc198eb0 tx=0x7f60ac005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c24de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbc3670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47098/0 (socket says 172.21.15.69:47098) 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c24de640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbc3670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:52550/0 (socket says 172.21.15.69:52550) 2024-05-09T09:43:42.236 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.236+0000 7f60c1cdd640 1 -- 172.21.15.69:0/1621083320 learned_addr learned my addr 172.21.15.69:0/1621083320 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c2cdf640 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbcb240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 -- 172.21.15.69:0/1621083320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 msgr2=0x5617abbc3670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbc3670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.237 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 -- 172.21.15.69:0/1621083320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 msgr2=0x5617abbcb240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbcb240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 -- 172.21.15.69:0/1621083320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617abaf38e0 con 0x5617abbc3c40 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00e630 tx=0x7f60ac014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f071943e9fb49e56 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60b0ff9640 1 -- 172.21.15.69:0/1621083320 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac004020 con 0x5617abbc3c40 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60b0ff9640 1 -- 172.21.15.69:0/1621083320 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac005500 con 0x5617abbc3c40 2024-05-09T09:43:42.238 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.237+0000 7f60b0ff9640 1 -- 172.21.15.69:0/1621083320 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0035f0 con 0x5617abbc3c40 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.238+0000 7f60c7450740 1 -- 172.21.15.69:0/1621083320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 msgr2=0x5617abbc8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.238+0000 7f60c7450740 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00e630 tx=0x7f60ac014000 comp rx=0 tx=0).stop 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.239+0000 7f60c7450740 1 -- 172.21.15.69:0/1621083320 shutdown_connections 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.239+0000 7f60c7450740 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbcb240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.239+0000 7f60c7450740 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbc8740 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.239+0000 7f60c7450740 1 --2- 172.21.15.69:0/1621083320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbc3670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.239 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.239+0000 7f60c7450740 1 -- 172.21.15.69:0/1621083320 >> 172.21.15.69:0/1621083320 conn(0x5617abbbe8c0 msgr2=0x5617abbbecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:42.240 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.240+0000 7f60c7450740 1 -- 172.21.15.69:0/1621083320 shutdown_connections 2024-05-09T09:43:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.242+0000 7f60c7450740 1 -- 172.21.15.69:0/1621083320 wait complete. 2024-05-09T09:43:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.243+0000 7f60c7450740 1 -- start start 2024-05-09T09:43:42.242 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.243+0000 7f60c7450740 1 Processor -- start 2024-05-09T09:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.243+0000 7f60c7450740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbd1920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.243+0000 7f60c7450740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c7450740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbd8e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.243 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c7450740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x5617abbcbd60 con 0x5617abbc3c40 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47110/0 (socket says 172.21.15.69:47110) 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c1cdd640 1 -- 172.21.15.69:0/3870589118 learned_addr learned my addr 172.21.15.69:0/3870589118 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c7450740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x5617abae2790 con 0x5617abbc8db0 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x5617abbcbee0 con 0x5617abbc3250 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.244+0000 7f60c7450740 10 client.0 _ll_register_callbacks cb 0x5617abb1da20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:42.244 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.245+0000 7f60c1cdd640 1 -- 172.21.15.69:0/3870589118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 msgr2=0x5617abbd1920 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-05-09T09:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.245+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbd1920 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.245+0000 7f60c1cdd640 1 -- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 msgr2=0x5617abbd8e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.245+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbd8e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.245 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.245+0000 7f60c1cdd640 1 -- 172.21.15.69:0/3870589118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617ab83b210 con 0x5617abbc3c40 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.246+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00e600 tx=0x7f60ac004820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaf60d0853ce9f75 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.246+0000 7f60a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.246+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac004f20 con 0x5617abbc3c40 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.246+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f60ac0050c0 con 0x5617abbc3c40 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.246+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60ac005500 con 0x5617abbc3c40 2024-05-09T09:43:42.246 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.247+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f60ac036050 con 0x5617abbc3c40 2024-05-09T09:43:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.251+0000 7f60c7450740 10 client.17961 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:42.250 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.251+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5617ab907620 con 0x5617abbc3c40 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.253+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac0052e0 con 0x5617abbc3c40 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.253+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f60ac039020 con 0x5617abbc3c40 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.254+0000 7f60c7450740 10 client.17961 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.254+0000 7f60c7450740 10 client.17961 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.254+0000 7f60c7450740 20 client.17961 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.255+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abbdcfe0 0x5617abbdf3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.255+0000 7f60c7450740 4 client.17961 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:43:42.254 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.255+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x7f60ac005500 con 0x5617abbdcfe0 2024-05-09T09:43:42.258 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.259+0000 7f60c24de640 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abbdcfe0 0x5617abbdf3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.272 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.272+0000 7f60c24de640 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abbdcfe0 0x5617abbdf3f0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c1d9713ecdd10ee5 server_cookie=31c530eccb0a21dc in_seq=0 out_seq=0 2024-05-09T09:43:42.273 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.273+0000 7f60a5ffb640 10 client.17961 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.277+0000 7f60a5ffb640 1 -- 172.21.15.69:0/3870589118 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x7f60ac005500 con 0x5617abbdcfe0 2024-05-09T09:43:42.276 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.277+0000 7f60a5ffb640 10 client.17961 handle_command_reply: tid=0 2024-05-09T09:43:42.277 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.277+0000 7f60c7450740 1 client.17961 shutdown 2024-05-09T09:43:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.280+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 msgr2=0x5617abbd1e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.280 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.280+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00e600 tx=0x7f60ac004820 comp rx=0 tx=0).stop 2024-05-09T09:43:42.281 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 shutdown_connections 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc8db0 0x5617abbd8e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3c40 0x5617abbd1e60 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abbdcfe0 0x5617abbdf3f0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 --2- 172.21.15.69:0/3870589118 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbd1920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.282+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 >> 172.21.15.69:0/3870589118 conn(0x5617abbbe8c0 msgr2=0x5617abbbda00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:42.282 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.283+0000 7f60a5ffb640 0 client.17961 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:42.286 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.287+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 shutdown_connections 2024-05-09T09:43:42.287 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.288+0000 7f60c7450740 1 -- 172.21.15.69:0/3870589118 wait complete. 2024-05-09T09:43:42.289 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.290+0000 7f60c7450740 20 client.17961 trim_cache size 0 max 16384 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.323+0000 7f60c7450740 1 Processor -- start 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.323+0000 7f60c7450740 1 -- start start 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbe4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 0x5617abbd8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1aab50 con 0x5617abbc8db0 2024-05-09T09:43:42.324 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c1cdd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:60058/0 (socket says 172.21.15.69:60058) 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c1cdd640 1 -- 172.21.15.69:0/599837149 learned_addr learned my addr 172.21.15.69:0/599837149 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c2cdf640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 0x5617abbd8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1a83c0 con 0x5617abbc3c40 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.324+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1a6210 con 0x5617abbc3250 2024-05-09T09:43:42.325 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c24de640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbe4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c1cdd640 1 -- 172.21.15.69:0/599837149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 msgr2=0x5617abbe4cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbe4cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c1cdd640 1 -- 172.21.15.69:0/599837149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 msgr2=0x5617abbd8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 0x5617abbd8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c1cdd640 1 -- 172.21.15.69:0/599837149 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x5617abbbf630 con 0x5617abbc3c40 2024-05-09T09:43:42.326 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.325+0000 7f60c2cdf640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 0x5617abbd8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:42.328 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.328+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0058c0 tx=0x7f60ac004680 comp rx=0 tx=0).ready entity=mon.2 client_cookie=131b198963e7137f server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.330+0000 7f60a7fff640 1 -- 172.21.15.69:0/599837149 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac03f070 con 0x5617abbc3c40 2024-05-09T09:43:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.330+0000 7f60a7fff640 1 -- 172.21.15.69:0/599837149 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac009240 con 0x5617abbc3c40 2024-05-09T09:43:42.329 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.330+0000 7f60a7fff640 1 -- 172.21.15.69:0/599837149 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60ac0035f0 con 0x5617abbc3c40 2024-05-09T09:43:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 msgr2=0x5617abbe5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:42.330 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f60ac0058c0 tx=0x7f60ac004680 comp rx=0 tx=0).stop 2024-05-09T09:43:42.331 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 shutdown_connections 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc8db0 0x5617abbd8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbe5200 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 --2- 172.21.15.69:0/599837149 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc3250 0x5617abbe4cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.331+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 >> 172.21.15.69:0/599837149 conn(0x5617abbbe8c0 msgr2=0x5617abbc1870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.332+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 shutdown_connections 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.332+0000 7f60c7450740 1 -- 172.21.15.69:0/599837149 wait complete. 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.332+0000 7f60c7450740 1 -- start start 2024-05-09T09:43:42.332 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.332+0000 7f60c7450740 1 Processor -- start 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbd6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc8db0 0x5617abbd7b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1a6880 con 0x5617abbc3250 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1a7cf0 con 0x5617abbc3c40 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f60bc1a6f50 con 0x5617abbc8db0 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.333+0000 7f60c7450740 10 client.0 _ll_register_callbacks cb 0x7f60b80092c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:42.335 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.334+0000 7f60c24de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.334+0000 7f60c24de640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:47134/0 (socket says 172.21.15.69:47134) 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.334+0000 7f60c24de640 1 -- 172.21.15.69:0/509285381 learned_addr learned my addr 172.21.15.69:0/509285381 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.334+0000 7f60c2cdf640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc8db0 0x5617abbd7b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.334+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbd6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 -- 172.21.15.69:0/509285381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc8db0 msgr2=0x5617abbd7b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc8db0 0x5617abbd7b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 -- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 msgr2=0x5617abbd6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbd6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 -- 172.21.15.69:0/509285381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x5617abbd80f0 con 0x5617abbc3250 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbd6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60c24de640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f60b80cc8b0 tx=0x7f60b80bb040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=65ffc9e560423c10 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:42.336 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60a5ffb640 10 client.0 ms_handle_connect on v2:172.21.15.69:3300/0 2024-05-09T09:43:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.335+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f60b80a3f10 con 0x5617abbc3250 2024-05-09T09:43:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.336+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mon.0 v2:172.21.15.69:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f60b8010550 con 0x5617abbc3250 2024-05-09T09:43:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.336+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f60ac009240 con 0x5617abbc3250 2024-05-09T09:43:42.337 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.336+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f60b80995a0 con 0x5617abbc3250 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.336+0000 7f60c7450740 10 client.17967 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.336+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x5617abc43660 con 0x5617abbc3250 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mon.0 v2:172.21.15.69:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f60ac009240 con 0x5617abbc3250 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mon.0 v2:172.21.15.69:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f60b809f120 con 0x5617abbc3250 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 10 client.17967 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:42.338 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 10 client.17967 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 20 client.17967 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abc44070 0x5617abc46490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 4 client.17967 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:43:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.337+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x5617abbd1710 con 0x5617abc44070 2024-05-09T09:43:42.339 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.339+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abc44070 0x5617abc46490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:42.343 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.344+0000 7f60c1cdd640 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abc44070 0x5617abc46490 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f6566b4bede6b6c9 server_cookie=4cc3b18f546f1752 in_seq=0 out_seq=0 2024-05-09T09:43:42.346 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:42.345+0000 7f60a5ffb640 10 client.17967 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:42.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351611 26.42 MB/sec execute 580 sec latency 3160.973 ms 2024-05-09T09:43:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60a5ffb640 1 -- 172.21.15.69:0/509285381 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x5617abbd1710 con 0x5617abc44070 2024-05-09T09:43:43.317 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60a5ffb640 10 client.17967 handle_command_reply: tid=0 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60c7450740 1 client.17967 shutdown 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 msgr2=0x5617abbd6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 secure :-1 s=READY pgs=1129 cs=0 l=1 rev1=1 crypto rx=0x7f60b80cc8b0 tx=0x7f60b80bb040 comp rx=0 tx=0).stop 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 shutdown_connections 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.317+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x5617abc44070 0x5617abc46490 crc :-1 s=READY pgs=279 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x5617abbc8db0 0x5617abbd7b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60a5ffb640 0 client.17967 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x5617abbc3c40 0x5617abbd6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 --2- 172.21.15.69:0/509285381 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x5617abbc3250 0x5617abbd6650 unknown :-1 s=CLOSED pgs=1129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 >> 172.21.15.69:0/509285381 conn(0x5617abbbe8c0 msgr2=0x5617abbbd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 shutdown_connections 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 1 -- 172.21.15.69:0/509285381 wait complete. 2024-05-09T09:43:43.318 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.318+0000 7f60c7450740 20 client.17967 trim_cache size 0 max 16384 2024-05-09T09:43:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:43:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:43:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:43:43.319 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 msgr2=0x7f60a80a5170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 0x7f60a80a5170 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f60bc198eb0 tx=0x7f60ac005f90 comp rx=0 tx=0).stop 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 msgr2=0x7f60bc1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 secure :-1 s=READY pgs=1126 cs=0 l=1 rev1=1 crypto rx=0x7f60b800b450 tx=0x7f60b800b910 comp rx=0 tx=0).stop 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 shutdown_connections 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f60a80a2cb0 0x7f60a80a5170 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.320 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f60bc107610 0x7f60bc19ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f60bc104ce0 0x7f60bc197ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 --2- 172.21.15.69:0/1735848314 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f60bc1023b0 0x7f60bc1979a0 unknown :-1 s=CLOSED pgs=1126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 >> 172.21.15.69:0/1735848314 conn(0x7f60bc0fbd00 msgr2=0x7f60bc105620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 shutdown_connections 2024-05-09T09:43:43.321 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:43.319+0000 7f60c4742640 1 -- 172.21.15.69:0/1735848314 wait complete. 2024-05-09T09:43:43.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:43 smithi163 ceph-mon[34354]: pgmap v1114: 129 pgs: 129 active+clean; 741 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 533 op/s 2024-05-09T09:43:43.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:43 smithi103 ceph-mon[34421]: pgmap v1114: 129 pgs: 129 active+clean; 741 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 533 op/s 2024-05-09T09:43:43.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:43 smithi069 ceph-mon[30123]: pgmap v1114: 129 pgs: 129 active+clean; 741 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.3 MiB/s rd, 8.4 MiB/s wr, 533 op/s 2024-05-09T09:43:43.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 351815 26.39 MB/sec execute 581 sec latency 746.755 ms 2024-05-09T09:43:44.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 352054 26.37 MB/sec execute 582 sec latency 154.193 ms 2024-05-09T09:43:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:45 smithi163 ceph-mon[34354]: pgmap v1115: 129 pgs: 129 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 463 op/s 2024-05-09T09:43:45.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:45 smithi103 ceph-mon[34421]: pgmap v1115: 129 pgs: 129 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 463 op/s 2024-05-09T09:43:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:45 smithi069 ceph-mon[30123]: pgmap v1115: 129 pgs: 129 active+clean; 756 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 10 MiB/s wr, 463 op/s 2024-05-09T09:43:45.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 352251 26.34 MB/sec execute 583 sec latency 162.310 ms 2024-05-09T09:43:46.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 352512 26.32 MB/sec execute 584 sec latency 143.774 ms 2024-05-09T09:43:47.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:47 smithi163 ceph-mon[34354]: pgmap v1116: 129 pgs: 129 active+clean; 764 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 636 op/s 2024-05-09T09:43:47.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:47 smithi103 ceph-mon[34421]: pgmap v1116: 129 pgs: 129 active+clean; 764 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 636 op/s 2024-05-09T09:43:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:47 smithi069 ceph-mon[30123]: pgmap v1116: 129 pgs: 129 active+clean; 764 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 15 MiB/s wr, 636 op/s 2024-05-09T09:43:47.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 352751 26.29 MB/sec execute 585 sec latency 157.647 ms 2024-05-09T09:43:48.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 353021 26.27 MB/sec execute 586 sec latency 137.260 ms 2024-05-09T09:43:49.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:49 smithi163 ceph-mon[34354]: pgmap v1117: 129 pgs: 129 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 646 op/s 2024-05-09T09:43:49.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:49 smithi103 ceph-mon[34421]: pgmap v1117: 129 pgs: 129 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 646 op/s 2024-05-09T09:43:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:49 smithi069 ceph-mon[30123]: pgmap v1117: 129 pgs: 129 active+clean; 755 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.1 MiB/s rd, 14 MiB/s wr, 646 op/s 2024-05-09T09:43:49.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 353664 26.29 MB/sec execute 587 sec latency 101.212 ms 2024-05-09T09:43:50.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 354238 26.28 MB/sec execute 588 sec latency 67.131 ms 2024-05-09T09:43:51.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:51 smithi163 ceph-mon[34354]: pgmap v1118: 129 pgs: 129 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 MiB/s wr, 745 op/s 2024-05-09T09:43:51.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:51 smithi103 ceph-mon[34421]: pgmap v1118: 129 pgs: 129 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 MiB/s wr, 745 op/s 2024-05-09T09:43:51.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:51 smithi069 ceph-mon[30123]: pgmap v1118: 129 pgs: 129 active+clean; 711 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.3 MiB/s rd, 15 MiB/s wr, 745 op/s 2024-05-09T09:43:51.846 INFO:tasks.workunit.client.0.smithi069.stdout: 10 354974 26.32 MB/sec execute 589 sec latency 70.156 ms 2024-05-09T09:43:52.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 355559 26.34 MB/sec execute 590 sec latency 70.683 ms 2024-05-09T09:43:53.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:53 smithi163 ceph-mon[34354]: pgmap v1119: 129 pgs: 129 active+clean; 662 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 14 MiB/s wr, 800 op/s 2024-05-09T09:43:53.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:53 smithi103 ceph-mon[34421]: pgmap v1119: 129 pgs: 129 active+clean; 662 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 14 MiB/s wr, 800 op/s 2024-05-09T09:43:53.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:53 smithi069 ceph-mon[30123]: pgmap v1119: 129 pgs: 129 active+clean; 662 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.8 MiB/s rd, 14 MiB/s wr, 800 op/s 2024-05-09T09:43:53.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 356141 26.33 MB/sec execute 591 sec latency 100.305 ms 2024-05-09T09:43:54.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 356583 26.33 MB/sec execute 592 sec latency 69.028 ms 2024-05-09T09:43:55.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:55 smithi163 ceph-mon[34354]: pgmap v1120: 129 pgs: 129 active+clean; 659 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 702 op/s 2024-05-09T09:43:55.562 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:55 smithi069 ceph-mon[30123]: pgmap v1120: 129 pgs: 129 active+clean; 659 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 702 op/s 2024-05-09T09:43:55.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:55 smithi103 ceph-mon[34421]: pgmap v1120: 129 pgs: 129 active+clean; 659 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.5 MiB/s rd, 8.8 MiB/s wr, 702 op/s 2024-05-09T09:43:55.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 357219 26.35 MB/sec execute 593 sec latency 51.871 ms 2024-05-09T09:43:56.438 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.435+0000 7f5c693f9640 1 -- 172.21.15.69:0/1800417193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64102340 msgr2=0x7f5c64108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.435+0000 7f5c693f9640 1 --2- 172.21.15.69:0/1800417193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64102340 0x7f5c64108bd0 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 crypto rx=0x7f5c5c009ca0 tx=0x7f5c5c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- 172.21.15.69:0/1800417193 shutdown_connections 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- 172.21.15.69:0/1800417193 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64102340 0x7f5c64108bd0 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- 172.21.15.69:0/1800417193 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64101980 0x7f5c64101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- 172.21.15.69:0/1800417193 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64100790 0x7f5c64100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- 172.21.15.69:0/1800417193 >> 172.21.15.69:0/1800417193 conn(0x7f5c640fbd00 msgr2=0x7f5c640fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- 172.21.15.69:0/1800417193 shutdown_connections 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- 172.21.15.69:0/1800417193 wait complete. 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 Processor -- start 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- start start 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.439 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64101980 0x7f5c64197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64102340 0x7f5c6419f020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6410c3f0 con 0x7f5c64100790 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6410c270 con 0x7f5c64101980 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c693f9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c6410c570 con 0x7f5c64102340 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33852/0 (socket says 172.21.15.69:33852) 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c62ffd640 1 -- 172.21.15.69:0/3010330612 learned_addr learned my addr 172.21.15.69:0/3010330612 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.436+0000 7f5c627fc640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64101980 0x7f5c64197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c637fe640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64102340 0x7f5c6419f020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 -- 172.21.15.69:0/3010330612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64102340 msgr2=0x7f5c6419f020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64102340 0x7f5c6419f020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.440 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 -- 172.21.15.69:0/3010330612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64101980 msgr2=0x7f5c64197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64101980 0x7f5c64197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 -- 172.21.15.69:0/3010330612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5c6419f690 con 0x7f5c64100790 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e9c0 tx=0x7f5c5400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c5bfff640 1 -- 172.21.15.69:0/3010330612 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5400cd80 con 0x7f5c64100790 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c5bfff640 1 -- 172.21.15.69:0/3010330612 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c540035f0 con 0x7f5c64100790 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5c6419f980 con 0x7f5c64100790 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c5bfff640 1 -- 172.21.15.69:0/3010330612 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c54010690 con 0x7f5c64100790 2024-05-09T09:43:56.441 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.437+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5c6419fed0 con 0x7f5c64100790 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.439+0000 7f5c5bfff640 1 -- 172.21.15.69:0/3010330612 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5c540037a0 con 0x7f5c64100790 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 Processor -- start 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 -- start start 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3c5670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3ca740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3cd240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a54f329340 con 0x55a54f3c5250 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a54f2d1290 con 0x55a54f3cadb0 2024-05-09T09:43:56.451 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a54f325300 con 0x55a54f3c5c40 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c5bfff640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 0x7f5c500a50f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c637fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3c5670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c637fe640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3c5670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33862/0 (socket says 172.21.15.69:33862) 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3cd240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c637fe640 1 -- 172.21.15.69:0/497335884 learned_addr learned my addr 172.21.15.69:0/497335884 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3ca740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c637fe640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 0x7f5c500a50f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.452 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 -- 172.21.15.69:0/497335884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 msgr2=0x55a54f3ca740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3ca740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c5bfff640 1 -- 172.21.15.69:0/3010330612 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5c540cf7a0 con 0x7f5c64100790 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 -- 172.21.15.69:0/497335884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 msgr2=0x55a54f3c5670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3c5670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.440+0000 7f5c627fc640 1 -- 172.21.15.69:0/497335884 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a54f2f58e0 con 0x55a54f3cadb0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.441+0000 7f5c627fc640 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3cd240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009bf0 tx=0x7f5c4c011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c730d1498f5f318 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.441+0000 7f5c637fe640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 0x7f5c500a50f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55a54f3c64a0 tx=0x7f5c5c005d80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.441+0000 7f5c597fa640 1 -- 172.21.15.69:0/497335884 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c004050 con 0x55a54f3cadb0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.441+0000 7f5c597fa640 1 -- 172.21.15.69:0/497335884 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c005500 con 0x55a54f3cadb0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.441+0000 7f5c597fa640 1 -- 172.21.15.69:0/497335884 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c004e40 con 0x55a54f3cadb0 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/497335884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 msgr2=0x55a54f3cd240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3cd240 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c009bf0 tx=0x7f5c4c011810 comp rx=0 tx=0).stop 2024-05-09T09:43:56.453 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/497335884 shutdown_connections 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3cd240 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3ca740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/497335884 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3c5670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/497335884 >> 172.21.15.69:0/497335884 conn(0x55a54f3c08c0 msgr2=0x55a54f3c0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/497335884 shutdown_connections 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/497335884 wait complete. 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.443+0000 7f5c6c1e2740 1 -- start start 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 Processor -- start 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3cf400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.454 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x55a54f3cdd60 con 0x55a54f3c5250 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x55a54f2e4790 con 0x55a54f3cadb0 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x55a54f3cdee0 con 0x55a54f3c5c40 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c6c1e2740 10 client.0 _ll_register_callbacks cb 0x55a54f31fa20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c627fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3cf400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3cf400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:33878/0 (socket says 172.21.15.69:33878) 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c627fc640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57782/0 (socket says 172.21.15.69:57782) 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c62ffd640 1 -- 172.21.15.69:0/147153686 learned_addr learned my addr 172.21.15.69:0/147153686 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.444+0000 7f5c637fe640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 -- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 msgr2=0x55a54f3d6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.455 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 -- 172.21.15.69:0/147153686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 msgr2=0x55a54f3cf400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3cf400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 -- 172.21.15.69:0/147153686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a54f03d210 con 0x55a54f3c5c40 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c627fc640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c01b040 tx=0x7f5c4c0056f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=11dda3ce14e41786 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c4a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c4c029070 con 0x55a54f3c5c40 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c4c005500 con 0x55a54f3c5c40 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5c4c004d80 con 0x55a54f3c5c40 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5c4c035050 con 0x55a54f3c5c40 2024-05-09T09:43:56.456 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c6c1e2740 10 client.27151 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.445+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a54f109620 con 0x55a54f3c5c40 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c4c005500 con 0x55a54f3c5c40 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5c4c003d00 con 0x55a54f3c5c40 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 10 client.27151 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 10 client.27151 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 20 client.27151 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f3dad20 0x55a54f3dd150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 4 client.27151 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "get_command_descriptions"}) -- 0x55a54f336a70 con 0x55a54f3dad20 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.446+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f3dad20 0x55a54f3dd150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.457 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.447+0000 7f5c4a7fc640 10 client.27151 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:56.458 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.447+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f3dad20 0x55a54f3dd150 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=845fe4c2d5c9fb0c server_cookie=c632b2edbc3a9cf1 in_seq=0 out_seq=0 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.454+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/147153686 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+23246 (crc 0 0 0) 0x55a54f336a70 con 0x55a54f3dad20 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.454+0000 7f5c4a7fc640 10 client.27151 handle_command_reply: tid=0 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.454+0000 7f5c6c1e2740 1 client.27151 shutdown 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 msgr2=0x55a54f3cf940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f5c4c01b040 tx=0x7f5c4c0056f0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 shutdown_connections 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.459 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f3dad20 0x55a54f3dd150 crc :-1 s=READY pgs=280 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5c40 0x55a54f3cf940 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/147153686 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5250 0x55a54f3cf400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c4a7fc640 0 client.27151 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 >> 172.21.15.69:0/147153686 conn(0x55a54f3c08c0 msgr2=0x55a54f3bfa00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 shutdown_connections 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/147153686 wait complete. 2024-05-09T09:43:56.460 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.455+0000 7f5c6c1e2740 20 client.27151 trim_cache size 0 max 16384 2024-05-09T09:43:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.471+0000 7f5c6c1e2740 1 Processor -- start 2024-05-09T09:43:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.471+0000 7f5c6c1e2740 1 -- start start 2024-05-09T09:43:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.472 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3c5c40 0x55a54f3d6940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 0x55a54f3e2a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641acc50 con 0x55a54f3cadb0 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57784/0 (socket says 172.21.15.69:57784) 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c62ffd640 1 -- 172.21.15.69:0/1120490076 learned_addr learned my addr 172.21.15.69:0/1120490076 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641aa4c0 con 0x55a54f3c5c40 2024-05-09T09:43:56.473 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a8310 con 0x55a54f3c5250 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c637fe640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 0x55a54f3e2a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.472+0000 7f5c627fc640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3c5c40 0x55a54f3d6940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 -- 172.21.15.69:0/1120490076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3c5c40 msgr2=0x55a54f3d6940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3c5c40 0x55a54f3d6940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 -- 172.21.15.69:0/1120490076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 msgr2=0x55a54f3e2a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.475 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 0x55a54f3e2a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 -- 172.21.15.69:0/1120490076 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x55a54f3c1630 con 0x55a54f3c5250 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c637fe640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 0x55a54f3e2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5c540980e0 tx=0x7f5c540ccdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f55b6774b26094a2 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c58ff9640 1 -- 172.21.15.69:0/1120490076 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c540a0030 con 0x55a54f3c5250 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c58ff9640 1 -- 172.21.15.69:0/1120490076 <== mon.1 v2:172.21.15.163:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c540a48f0 con 0x55a54f3c5250 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.473+0000 7f5c58ff9640 1 -- 172.21.15.69:0/1120490076 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5409b0d0 con 0x55a54f3c5250 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 msgr2=0x55a54f3cd1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f5c540980e0 tx=0x7f5c540ccdf0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 shutdown_connections 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3cadb0 0x55a54f3e2a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3c5c40 0x55a54f3d6940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/1120490076 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3cd1a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.476 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 >> 172.21.15.69:0/1120490076 conn(0x55a54f3c08c0 msgr2=0x55a54f3bb620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 shutdown_connections 2024-05-09T09:43:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.474+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/1120490076 wait complete. 2024-05-09T09:43:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.475+0000 7f5c6c1e2740 1 -- start start 2024-05-09T09:43:56.477 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.475+0000 7f5c6c1e2740 1 Processor -- start 2024-05-09T09:43:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.475+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.480 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 0x55a54f3d43b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:57790/0 (socket says 172.21.15.69:57790) 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c627fc640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 0x55a54f3d43b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 -- 172.21.15.69:0/818420250 learned_addr learned my addr 172.21.15.69:0/818420250 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d53c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a8980 con 0x55a54f3c5c40 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a9df0 con 0x55a54f3cadb0 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5c641a9050 con 0x55a54f3c5250 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c6c1e2740 10 client.0 _ll_register_callbacks cb 0x7f5c4c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c637fe640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 -- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 msgr2=0x55a54f3d53c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d53c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.481 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 -- 172.21.15.69:0/818420250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 msgr2=0x55a54f3d43b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 0x55a54f3d43b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({osdmap=0}) -- 0x55a54f3d59b0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c627fc640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 0x55a54f3d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c62ffd640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5c5409c470 tx=0x7f5c54003bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c578ec3a75d99375 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.476+0000 7f5c4a7fc640 10 client.0 ms_handle_connect on v2:172.21.15.163:3300/0 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mon.1 v2:172.21.15.163:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5c5409b2d0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mon.1 v2:172.21.15.163:3300/0 2 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5c540979d0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_get_version(what=fsmap handle=1) -- 0x7f5c540a48f0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mon.1 v2:172.21.15.163:3300/0 3 ==== mon_get_version_reply(handle=1 version=49) ==== 24+0+0 (secure 0 0 0) 0x7f5c54099d20 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 10 client.27157 fetch_fsmap learned FSMap version 49 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_subscribe({config=0+,fsmap=49,monmap=4+}) -- 0x55a54f3d6c80 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mon.1 v2:172.21.15.163:3300/0 4 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5c540a48f0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mon.1 v2:172.21.15.163:3300/0 5 ==== fsmap(e 49) ==== 8354+0+0 (secure 0 0 0) 0x7f5c540ba2c0 con 0x55a54f3c5250 2024-05-09T09:43:56.482 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 10 client.27157 fetch_fsmap finished waiting for FSMap version 49 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 10 client.27157 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 20 client.27157 populate_metadata read hostname 'smithi069' 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f441a00 0x55a54f443e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 4 client.27157 mds_command: new command op to 24527 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.477+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 --> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] -- command(tid 0: {"prefix": "flush journal"}) -- 0x55a54f330dc0 con 0x55a54f441a00 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.478+0000 7f5c627fc640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f441a00 0x55a54f443e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.478+0000 7f5c4a7fc640 10 client.27157 ms_handle_connect on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:56.483 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:56.478+0000 7f5c627fc640 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f441a00 0x55a54f443e20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=236643150fe3caae server_cookie=c2508bdf62dee344 in_seq=0 out_seq=0 2024-05-09T09:43:56.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 357615 26.33 MB/sec execute 594 sec latency 310.911 ms 2024-05-09T09:43:57.140 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c4a7fc640 1 -- 172.21.15.69:0/818420250 <== mds.0 v2:172.21.15.103:6838/189860875 1 ==== command_reply(tid 0: 0 ) ==== 8+0+44 (crc 0 0 0) 0x55a54f330dc0 con 0x55a54f441a00 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c4a7fc640 10 client.27157 handle_command_reply: tid=0 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 client.27157 shutdown 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 msgr2=0x55a54f3d3e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f5c5409c470 tx=0x7f5c54003bc0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 shutdown_connections 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x55a54f3cadb0 0x55a54f3d53c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x55a54f3c5c40 0x55a54f3d43b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.103:6838/189860875,v1:172.21.15.103:6839/189860875] conn(0x55a54f441a00 0x55a54f443e20 crc :-1 s=READY pgs=281 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c4a7fc640 0 client.27157 ms_handle_reset on v2:172.21.15.103:6838/189860875 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 --2- 172.21.15.69:0/818420250 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x55a54f3c5250 0x55a54f3d3e70 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 >> 172.21.15.69:0/818420250 conn(0x55a54f3c08c0 msgr2=0x55a54f3cf3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:57.141 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.139+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 shutdown_connections 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.140+0000 7f5c6c1e2740 1 -- 172.21.15.69:0/818420250 wait complete. 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.140+0000 7f5c6c1e2740 20 client.27157 trim_cache size 0 max 16384 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:{ 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "message": "", 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout: "return_code": 0 2024-05-09T09:43:57.142 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stdout:} 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 msgr2=0x7f5c500a50f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 0x7f5c500a50f0 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x55a54f3c64a0 tx=0x7f5c5c005d80 comp rx=0 tx=0).stop 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 msgr2=0x7f5c64197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 crypto rx=0x7f5c5400e9c0 tx=0x7f5c5400ee80 comp rx=0 tx=0).stop 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 shutdown_connections 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5c500a2c30 0x7f5c500a50f0 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.143 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5c64102340 0x7f5c6419f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5c64101980 0x7f5c64197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 --2- 172.21.15.69:0/3010330612 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5c64100790 0x7f5c64197a70 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:43:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 >> 172.21.15.69:0/3010330612 conn(0x7f5c640fbd00 msgr2=0x7f5c64068ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:43:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 shutdown_connections 2024-05-09T09:43:57.144 INFO:teuthology.task.background_exec.ubuntu@smithi069.front.sepia.ceph.com.smithi069.stderr:2024-05-09T09:43:57.141+0000 7f5c693f9640 1 -- 172.21.15.69:0/3010330612 wait complete. 2024-05-09T09:43:57.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:57 smithi163 ceph-mon[34354]: pgmap v1121: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 727 op/s 2024-05-09T09:43:57.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:57 smithi103 ceph-mon[34421]: pgmap v1121: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 727 op/s 2024-05-09T09:43:57.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:57 smithi069 ceph-mon[30123]: pgmap v1121: 129 pgs: 129 active+clean; 669 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.5 MiB/s rd, 7.3 MiB/s wr, 727 op/s 2024-05-09T09:43:57.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 358153 26.34 MB/sec execute 595 sec latency 549.631 ms 2024-05-09T09:43:58.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 358709 26.35 MB/sec execute 596 sec latency 83.353 ms 2024-05-09T09:43:59.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:43:59 smithi163 ceph-mon[34354]: pgmap v1122: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 514 op/s 2024-05-09T09:43:59.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:43:59 smithi103 ceph-mon[34421]: pgmap v1122: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 514 op/s 2024-05-09T09:43:59.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:43:59 smithi069 ceph-mon[30123]: pgmap v1122: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 6.8 MiB/s rd, 2.6 MiB/s wr, 514 op/s 2024-05-09T09:43:59.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 359369 26.37 MB/sec execute 597 sec latency 49.303 ms 2024-05-09T09:44:00.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 359771 26.35 MB/sec execute 598 sec latency 78.910 ms 2024-05-09T09:44:01.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:01 smithi103 ceph-mon[34421]: pgmap v1123: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 535 op/s 2024-05-09T09:44:01.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:01 smithi069 ceph-mon[30123]: pgmap v1123: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 535 op/s 2024-05-09T09:44:01.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:01 smithi163 ceph-mon[34354]: pgmap v1123: 129 pgs: 129 active+clean; 668 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 MiB/s rd, 2.7 MiB/s wr, 535 op/s 2024-05-09T09:44:01.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 360398 26.38 MB/sec execute 599 sec latency 108.707 ms 2024-05-09T09:44:02.847 INFO:tasks.workunit.client.0.smithi069.stdout: 10 cleanup 600 sec 2024-05-09T09:44:03.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:03 smithi103 ceph-mon[34421]: pgmap v1124: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.1 MiB/s wr, 564 op/s 2024-05-09T09:44:03.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:03 smithi069 ceph-mon[30123]: pgmap v1124: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.1 MiB/s wr, 564 op/s 2024-05-09T09:44:03.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:03 smithi163 ceph-mon[34354]: pgmap v1124: 129 pgs: 129 active+clean; 674 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.7 MiB/s rd, 3.1 MiB/s wr, 564 op/s 2024-05-09T09:44:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: 0 cleanup 601 sec 2024-05-09T09:44:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:44:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: Operation Count AvgLat MaxLat 2024-05-09T09:44:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: ---------------------------------------- 2024-05-09T09:44:03.818 INFO:tasks.workunit.client.0.smithi069.stdout: NTCreateX 507625 4.981 3154.482 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Close 372714 0.003 0.399 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Rename 21516 22.629 2218.464 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Unlink 102613 5.632 519.884 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Qpathinfo 460342 3.275 3160.968 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Qfileinfo 80237 0.001 0.043 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Qfsinfo 84443 0.191 12.437 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Sfileinfo 41359 0.246 187.014 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Find 177898 0.301 286.023 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: WriteX 250787 0.109 2012.524 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: ReadX 796417 0.086 2188.973 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: LockX 1654 3.543 66.688 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: UnlockX 1654 2.791 23.185 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: Flush 35587 19.837 3542.262 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout: 2024-05-09T09:44:03.819 INFO:tasks.workunit.client.0.smithi069.stdout:Throughput 26.3761 MB/sec 10 clients 10 procs max_latency=3542.269 ms 2024-05-09T09:44:03.820 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-05-09T09:44:03.820 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0/tmp 2024-05-09T09:44:05.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:05 smithi103 ceph-mon[34421]: pgmap v1125: 129 pgs: 129 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.2 MiB/s wr, 487 op/s 2024-05-09T09:44:05.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:05 smithi069 ceph-mon[30123]: pgmap v1125: 129 pgs: 129 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.2 MiB/s wr, 487 op/s 2024-05-09T09:44:05.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:05 smithi163 ceph-mon[34354]: pgmap v1125: 129 pgs: 129 active+clean; 680 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.1 MiB/s rd, 3.2 MiB/s wr, 487 op/s 2024-05-09T09:44:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:07 smithi103 ceph-mon[34421]: pgmap v1126: 129 pgs: 129 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-05-09T09:44:07.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:07 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:07 smithi069 ceph-mon[30123]: pgmap v1126: 129 pgs: 129 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-05-09T09:44:07.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:07 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:07 smithi163 ceph-mon[34354]: pgmap v1126: 129 pgs: 129 active+clean; 696 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 9.3 MiB/s rd, 4.9 MiB/s wr, 521 op/s 2024-05-09T09:44:07.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:07 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:08.730 INFO:tasks.workunit:Stopping ['suites/dbench.sh'] on client.0... 2024-05-09T09:44:08.730 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-05-09T09:44:09.296 DEBUG:teuthology.parallel:result is None 2024-05-09T09:44:09.296 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T09:44:09.357 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.0/client.0 2024-05-09T09:44:09.357 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-05-09T09:44:09.368 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:44:09.379 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-05-09T09:44:09.389 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-05-09T09:44:09.389 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_1ae7ad82388e92a475afff437d49054826c019a1/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph_be85c202969d25af6e130678e7d73a20e9c4ca11/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: rank damage found: {'backtrace'} 2024-05-09T09:44:09.390 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-05-09T09:44:09.399 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:44:09.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.597+0000 7f80a8027640 1 -- 172.21.15.69:0/2136790846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a0103450 msgr2=0x7f80a0109ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:09.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.597+0000 7f80a8027640 1 --2- 172.21.15.69:0/2136790846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a0103450 0x7f80a0109ce0 secure :-1 s=READY pgs=1132 cs=0 l=1 rev1=1 crypto rx=0x7f8098009ca0 tx=0x7f809802a8f0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.598+0000 7f80a8027640 1 -- 172.21.15.69:0/2136790846 shutdown_connections 2024-05-09T09:44:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.598+0000 7f80a8027640 1 --2- 172.21.15.69:0/2136790846 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a0103450 0x7f80a0109ce0 unknown :-1 s=CLOSED pgs=1132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.598+0000 7f80a8027640 1 --2- 172.21.15.69:0/2136790846 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0102a90 0x7f80a0102f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.598+0000 7f80a8027640 1 --2- 172.21.15.69:0/2136790846 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a01018a0 0x7f80a0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.598+0000 7f80a8027640 1 -- 172.21.15.69:0/2136790846 >> 172.21.15.69:0/2136790846 conn(0x7f80a00fce30 msgr2=0x7f80a00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:09.605 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.604+0000 7f80a8027640 1 -- 172.21.15.69:0/2136790846 shutdown_connections 2024-05-09T09:44:09.608 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.607+0000 7f80a8027640 1 -- 172.21.15.69:0/2136790846 wait complete. 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 Processor -- start 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 -- start start 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 0x7f80a00700b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a0102a90 0x7f80a00705f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f80a010b470 con 0x7f80a01018a0 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f80a010b2f0 con 0x7f80a0103450 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a8027640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f80a010b5f0 con 0x7f80a0102a90 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a65c4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:09.609 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a65c4640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:50408/0 (socket says 172.21.15.69:50408) 2024-05-09T09:44:09.610 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.608+0000 7f80a65c4640 1 -- 172.21.15.69:0/2456757764 learned_addr learned my addr 172.21.15.69:0/2456757764 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:09.613 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a5dc3640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 0x7f80a00700b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:09 smithi103 ceph-mon[34421]: pgmap v1127: 129 pgs: 129 active+clean; 716 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 6.1 MiB/s wr, 462 op/s 2024-05-09T09:44:09.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:09 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a65c4640 1 -- 172.21.15.69:0/2456757764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a0102a90 msgr2=0x7f80a00705f0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a65c4640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a0102a90 0x7f80a00705f0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a65c4640 1 -- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 msgr2=0x7f80a00700b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a65c4640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 0x7f80a00700b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.613+0000 7f80a65c4640 1 -- 172.21.15.69:0/2456757764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f80a0073770 con 0x7f80a0103450 2024-05-09T09:44:09.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.615+0000 7f80a5dc3640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 0x7f80a00700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:09.621 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.621+0000 7f80a65c4640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80980098d0 tx=0x7f809802fba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.624+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098003970 con 0x7f80a0103450 2024-05-09T09:44:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.624+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f809802d070 con 0x7f80a0103450 2024-05-09T09:44:09.625 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.624+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8098030850 con 0x7f80a0103450 2024-05-09T09:44:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.626+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f80a01103a0 con 0x7f80a0103450 2024-05-09T09:44:09.627 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.626+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f80a01108c0 con 0x7f80a0103450 2024-05-09T09:44:09.629 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.628+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f80840052c0 con 0x7f80a0103450 2024-05-09T09:44:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.633+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8098003b10 con 0x7f80a0103450 2024-05-09T09:44:09.634 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.634+0000 7f8092ffd640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 0x7f808c0a5260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:09.635 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.634+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f80980f09e0 con 0x7f80a0103450 2024-05-09T09:44:09.638 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.638+0000 7f80a5dc3640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 0x7f808c0a5260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:09.640 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.640+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f80980bd560 con 0x7f80a0103450 2024-05-09T09:44:09.647 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.647+0000 7f80a5dc3640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 0x7f808c0a5260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f80a01028f0 tx=0x7f809c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:09 smithi069 ceph-mon[30123]: pgmap v1127: 129 pgs: 129 active+clean; 716 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 6.1 MiB/s wr, 462 op/s 2024-05-09T09:44:09.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:09 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:09.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:09 smithi163 ceph-mon[34354]: pgmap v1127: 129 pgs: 129 active+clean; 716 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.4 MiB/s rd, 6.1 MiB/s wr, 462 op/s 2024-05-09T09:44:09.831 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:09 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:09.986 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:09.985+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f80840055b0 con 0x7f80a0103450 2024-05-09T09:44:10.066 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.066+0000 7f8092ffd640 1 -- 172.21.15.69:0/2456757764 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24708 (secure 0 0 0) 0x7f80980bce60 con 0x7f80a0103450 2024-05-09T09:44:10.067 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:10.067 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":98,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:27:27.102222+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:44:10.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 msgr2=0x7f808c0a5260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 0x7f808c0a5260 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f80a01028f0 tx=0x7f809c008040 comp rx=0 tx=0).stop 2024-05-09T09:44:10.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 msgr2=0x7f80a010fe60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.070 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f80980098d0 tx=0x7f809802fba0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 shutdown_connections 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f808c0a2da0 0x7f808c0a5260 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f80a0103450 0x7f80a010fe60 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f80a0102a90 0x7f80a00705f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 --2- 172.21.15.69:0/2456757764 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f80a01018a0 0x7f80a00700b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 >> 172.21.15.69:0/2456757764 conn(0x7f80a00fce30 msgr2=0x7f80a0105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.069+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 shutdown_connections 2024-05-09T09:44:10.071 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.070+0000 7f80a8027640 1 -- 172.21.15.69:0/2456757764 wait complete. 2024-05-09T09:44:10.089 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:44:10.089 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/361940022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0102340 msgr2=0x7fb2b0108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/361940022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0102340 0x7fb2b0108bd0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb2a8009ca0 tx=0x7fb2a802ecc0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/361940022 shutdown_connections 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/361940022 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0102340 0x7fb2b0108bd0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/361940022 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b0101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/361940022 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0100790 0x7fb2b0100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/361940022 >> 172.21.15.69:0/361940022 conn(0x7fb2b00fbd20 msgr2=0x7fb2b00fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.285+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/361940022 shutdown_connections 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/361940022 wait complete. 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 Processor -- start 2024-05-09T09:44:10.287 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 -- start start 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0100790 0x7fb2b019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.286+0000 7fb2b6e2a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 0x7fb2b01a56d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b6e2a640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b010b2d0 con 0x7fb2b0102340 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b6e2a640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b010b150 con 0x7fb2b0100790 2024-05-09T09:44:10.288 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b6e2a640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb2b010b450 con 0x7fb2b0101980 2024-05-09T09:44:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2affff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.289 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 0x7fb2b01a56d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2affff640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b019e660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.163:3300/0 says I am v2:172.21.15.69:58242/0 (socket says 172.21.15.69:58242) 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2affff640 1 -- 172.21.15.69:0/296652352 learned_addr learned my addr 172.21.15.69:0/296652352 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b4bc6640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0100790 0x7fb2b019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 -- 172.21.15.69:0/296652352 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 msgr2=0x7fb2b019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 -- 172.21.15.69:0/296652352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0100790 msgr2=0x7fb2b019e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0100790 0x7fb2b019e120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 -- 172.21.15.69:0/296652352 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb2b01a5d40 con 0x7fb2b0102340 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2b53c7640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 0x7fb2b01a56d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb2a802f1e0 tx=0x7fb2a8003f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.287+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a8045070 con 0x7fb2b0102340 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.288+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb2a80105d0 con 0x7fb2b0102340 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.288+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb2a803c030 con 0x7fb2b0102340 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.288+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb2b01a5fd0 con 0x7fb2b0102340 2024-05-09T09:44:10.290 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.288+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb2b01a6490 con 0x7fb2b0102340 2024-05-09T09:44:10.291 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.289+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb2b0107ab0 con 0x7fb2b0102340 2024-05-09T09:44:10.292 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.291+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb2a8035c10 con 0x7fb2b0102340 2024-05-09T09:44:10.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.292+0000 7fb2adffb640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 0x7fb2980a5150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.292+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb2a80ef550 con 0x7fb2b0102340 2024-05-09T09:44:10.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.293+0000 7fb2b4bc6640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 0x7fb2980a5150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.293 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.293+0000 7fb2b4bc6640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 0x7fb2980a5150 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb2b01017e0 tx=0x7fb2a400f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.296 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.296+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb2a80bba80 con 0x7fb2b0102340 2024-05-09T09:44:10.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.474+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fb2b002d370 con 0x7fb2b0102340 2024-05-09T09:44:10.475 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.475+0000 7fb2adffb640 1 -- 172.21.15.69:0/296652352 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7fb2a80bfd80 con 0x7fb2b0102340 2024-05-09T09:44:10.475 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:44:10.477 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 msgr2=0x7fb2980a5150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 0x7fb2980a5150 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7fb2b01017e0 tx=0x7fb2a400f040 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 msgr2=0x7fb2b01a56d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 0x7fb2b01a56d0 secure :-1 s=READY pgs=1133 cs=0 l=1 rev1=1 crypto rx=0x7fb2a802f1e0 tx=0x7fb2a8003f90 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 shutdown_connections 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb2980a2c90 0x7fb2980a5150 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb2b0102340 0x7fb2b01a56d0 unknown :-1 s=CLOSED pgs=1133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb2b0101980 0x7fb2b019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 --2- 172.21.15.69:0/296652352 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb2b0100790 0x7fb2b019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 >> 172.21.15.69:0/296652352 conn(0x7fb2b00fbd20 msgr2=0x7fb2b0104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 shutdown_connections 2024-05-09T09:44:10.478 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.477+0000 7fb2b6e2a640 1 -- 172.21.15.69:0/296652352 wait complete. 2024-05-09T09:44:10.489 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:44:10.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 -- 172.21.15.69:0/2111089625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 msgr2=0x7f1da0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.595 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/2111089625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0100b90 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 crypto rx=0x7f1d90009900 tx=0x7f1d9002a410 comp rx=0 tx=0).stop 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 -- 172.21.15.69:0/2111089625 shutdown_connections 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/2111089625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/2111089625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 0x7f1da0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/2111089625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0100b90 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 -- 172.21.15.69:0/2111089625 >> 172.21.15.69:0/2111089625 conn(0x7f1da00fbce0 msgr2=0x7f1da00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 -- 172.21.15.69:0/2111089625 shutdown_connections 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.592+0000 7f1da4ea5640 1 -- 172.21.15.69:0/2111089625 wait complete. 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 Processor -- start 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 -- start start 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 0x7f1da0198120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.596 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da019f190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f1da010c3e0 con 0x7f1da0100770 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f1da010c260 con 0x7f1da0102320 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.593+0000 7f1da4ea5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f1da010c560 con 0x7f1da0101960 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57556/0 (socket says 172.21.15.69:57556) 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 -- 172.21.15.69:0/3703686992 learned_addr learned my addr 172.21.15.69:0/3703686992 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d95d9b640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 0x7f1da0198120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9ed9d640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da019f190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 -- 172.21.15.69:0/3703686992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 msgr2=0x7f1da0198120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 0x7f1da0198120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 -- 172.21.15.69:0/3703686992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 msgr2=0x7f1da019f190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da019f190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 -- 172.21.15.69:0/3703686992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f1da019f800 con 0x7f1da0100770 2024-05-09T09:44:10.597 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9ed9d640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d9e59c640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1d90031610 tx=0x7f1d900043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d9003f030 con 0x7f1da0100770 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1da019fa90 con 0x7f1da0100770 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f1d90003ce0 con 0x7f1da0100770 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1d90035070 con 0x7f1da0100770 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.594+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f1da019ffd0 con 0x7f1da0100770 2024-05-09T09:44:10.598 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.595+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1da01a0200 con 0x7f1da0100770 2024-05-09T09:44:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.596+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f1d90042030 con 0x7f1da0100770 2024-05-09T09:44:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.597+0000 7f1d97fff640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 0x7f1d8c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.597+0000 7f1d95d9b640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 0x7f1d8c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.597+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f1d900ef960 con 0x7f1da0100770 2024-05-09T09:44:10.599 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.597+0000 7f1d95d9b640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 0x7f1d8c0a5050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1da01990f0 tx=0x7f1d8800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.601 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.601+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1d900bc540 con 0x7f1da0100770 2024-05-09T09:44:10.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:10 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/2456757764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:10.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:10 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/2456757764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:10.782 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.782+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7f1da01a03d0 con 0x7f1da0100770 2024-05-09T09:44:10.783 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.782+0000 7f1d97fff640 1 -- 172.21.15.69:0/3703686992 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7f1d900bbe40 con 0x7f1da0100770 2024-05-09T09:44:10.783 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 msgr2=0x7f1d8c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 0x7f1d8c0a5050 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f1da01990f0 tx=0x7f1d8800b390 comp rx=0 tx=0).stop 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 msgr2=0x7f1da0197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 crypto rx=0x7f1d90031610 tx=0x7f1d900043b0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 shutdown_connections 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f1d8c0a2b90 0x7f1d8c0a5050 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1da0102320 0x7f1da019f190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1da0101960 0x7f1da0198120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.785 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 --2- 172.21.15.69:0/3703686992 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1da0100770 0x7f1da0197be0 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 >> 172.21.15.69:0/3703686992 conn(0x7f1da00fbce0 msgr2=0x7f1da00fd7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 shutdown_connections 2024-05-09T09:44:10.786 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.784+0000 7f1da4ea5640 1 -- 172.21.15.69:0/3703686992 wait complete. 2024-05-09T09:44:10.796 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:44:10.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:10 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/2456757764' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:10.903 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 -- 172.21.15.69:0/2148134252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c101090 msgr2=0x7f314c101510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 --2- 172.21.15.69:0/2148134252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c101090 0x7f314c101510 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 crypto rx=0x7f314000aa10 tx=0x7f314002b720 comp rx=0 tx=0).stop 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 -- 172.21.15.69:0/2148134252 shutdown_connections 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 --2- 172.21.15.69:0/2148134252 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f314c101a50 0x7f314c108370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 --2- 172.21.15.69:0/2148134252 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c101090 0x7f314c101510 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 --2- 172.21.15.69:0/2148134252 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c0ffeb0 0x7f314c1002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.900+0000 7f3153710640 1 -- 172.21.15.69:0/2148134252 >> 172.21.15.69:0/2148134252 conn(0x7f314c0fb470 msgr2=0x7f314c0fd8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- 172.21.15.69:0/2148134252 shutdown_connections 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- 172.21.15.69:0/2148134252 wait complete. 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 Processor -- start 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- start start 2024-05-09T09:44:10.904 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f314c101090 0x7f314c195f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c101a50 0x7f314c19cfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10b380 con 0x7f314c0ffeb0 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10b200 con 0x7f314c101a50 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f3153710640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f314c10b500 con 0x7f314c101090 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f31514ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f31514ac640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57582/0 (socket says 172.21.15.69:57582) 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.901+0000 7f31514ac640 1 -- 172.21.15.69:0/95763775 learned_addr learned my addr 172.21.15.69:0/95763775 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f3151cad640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c101a50 0x7f314c19cfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 -- 172.21.15.69:0/95763775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f314c101090 msgr2=0x7f314c195f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f314c101090 0x7f314c195f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.905 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 -- 172.21.15.69:0/95763775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c101a50 msgr2=0x7f314c19cfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c101a50 0x7f314c19cfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 -- 172.21.15.69:0/95763775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f314c19d610 con 0x7f314c0ffeb0 2024-05-09T09:44:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f31514ac640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f313c00eb90 tx=0x7f313c007590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.906 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c01c070 con 0x7f314c0ffeb0 2024-05-09T09:44:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f313c00ec80 con 0x7f314c0ffeb0 2024-05-09T09:44:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f314c19d900 con 0x7f314c0ffeb0 2024-05-09T09:44:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f313c017670 con 0x7f314c0ffeb0 2024-05-09T09:44:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.902+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f314c19df20 con 0x7f314c0ffeb0 2024-05-09T09:44:10.907 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.903+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f314c107250 con 0x7f314c0ffeb0 2024-05-09T09:44:10.908 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.904+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f313c01f070 con 0x7f314c0ffeb0 2024-05-09T09:44:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.905+0000 7f314a7fc640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 0x7f31340a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.905+0000 7f3150cab640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 0x7f31340a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.905+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f313c0d09e0 con 0x7f314c0ffeb0 2024-05-09T09:44:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.905+0000 7f3150cab640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 0x7f31340a50d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f314000a7d0 tx=0x7f3140030510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:10.909 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:10.908+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f313c09d5c0 con 0x7f314c0ffeb0 2024-05-09T09:44:11.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.084+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f314c062430 con 0x7f314c0ffeb0 2024-05-09T09:44:11.085 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.084+0000 7f314a7fc640 1 -- 172.21.15.69:0/95763775 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f313c00ce20 con 0x7f314c0ffeb0 2024-05-09T09:44:11.085 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 msgr2=0x7f31340a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 0x7f31340a50d0 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7f314000a7d0 tx=0x7f3140030510 comp rx=0 tx=0).stop 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 msgr2=0x7f314c1959f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 crypto rx=0x7f313c00eb90 tx=0x7f313c007590 comp rx=0 tx=0).stop 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 shutdown_connections 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f31340a2c10 0x7f31340a50d0 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f314c101a50 0x7f314c19cfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f314c101090 0x7f314c195f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 --2- 172.21.15.69:0/95763775 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f314c0ffeb0 0x7f314c1959f0 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 >> 172.21.15.69:0/95763775 conn(0x7f314c0fb470 msgr2=0x7f314c104420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 shutdown_connections 2024-05-09T09:44:11.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.086+0000 7f3153710640 1 -- 172.21.15.69:0/95763775 wait complete. 2024-05-09T09:44:11.098 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:44:11.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 -- 172.21.15.69:0/1802638426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910102320 msgr2=0x7f9910108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.204 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 --2- 172.21.15.69:0/1802638426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910102320 0x7f9910108bb0 secure :-1 s=READY pgs=1138 cs=0 l=1 rev1=1 crypto rx=0x7f9908009ca0 tx=0x7f990802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 -- 172.21.15.69:0/1802638426 shutdown_connections 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 --2- 172.21.15.69:0/1802638426 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910102320 0x7f9910108bb0 unknown :-1 s=CLOSED pgs=1138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 --2- 172.21.15.69:0/1802638426 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910101960 0x7f9910101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 --2- 172.21.15.69:0/1802638426 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 0x7f9910100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 -- 172.21.15.69:0/1802638426 >> 172.21.15.69:0/1802638426 conn(0x7f99100fbce0 msgr2=0x7f99100fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.201+0000 7f9916829640 1 -- 172.21.15.69:0/1802638426 shutdown_connections 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 -- 172.21.15.69:0/1802638426 wait complete. 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 Processor -- start 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 -- start start 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 0x7f9910197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 0x7f991019f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f991010c3d0 con 0x7f9910101960 2024-05-09T09:44:11.205 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f991010c250 con 0x7f9910102320 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9916829640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f991010c550 con 0x7f9910100770 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9907fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9907fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57588/0 (socket says 172.21.15.69:57588) 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f9907fff640 1 -- 172.21.15.69:0/1934562688 learned_addr learned my addr 172.21.15.69:0/1934562688 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.202+0000 7f990ffff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 0x7f9910197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9914dc6640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 0x7f991019f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 -- 172.21.15.69:0/1934562688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 msgr2=0x7f9910197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 0x7f9910197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 -- 172.21.15.69:0/1934562688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 msgr2=0x7f991019f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 0x7f991019f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 -- 172.21.15.69:0/1934562688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f991019f810 con 0x7f9910101960 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9914dc6640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 0x7f991019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9907fff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e9f0 tx=0x7f98f800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f800cd80 con 0x7f9910101960 2024-05-09T09:44:11.206 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f98f8010910 con 0x7f9910101960 2024-05-09T09:44:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f98f8018990 con 0x7f9910101960 2024-05-09T09:44:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f991019fb00 con 0x7f9910101960 2024-05-09T09:44:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.203+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f99101a00a0 con 0x7f9910101960 2024-05-09T09:44:11.207 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.204+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f99101a0270 con 0x7f9910101960 2024-05-09T09:44:11.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.205+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f98f8010ac0 con 0x7f9910101960 2024-05-09T09:44:11.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.206+0000 7f990dffb640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 0x7f98fc0a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.206+0000 7f990ffff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 0x7f98fc0a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.206+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f98f80cf790 con 0x7f9910101960 2024-05-09T09:44:11.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.206+0000 7f990ffff640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 0x7f98fc0a50d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9900005fd0 tx=0x7f9900005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:11.210 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.210+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f98f809c370 con 0x7f9910101960 2024-05-09T09:44:11.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:11 smithi069 ceph-mon[30123]: pgmap v1128: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.1 MiB/s wr, 541 op/s 2024-05-09T09:44:11.348 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:11 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/296652352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:11 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3703686992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.349 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:11 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/95763775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.386+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7f99101a04e0 con 0x7f9910101960 2024-05-09T09:44:11.387 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.386+0000 7f990dffb640 1 -- 172.21.15.69:0/1934562688 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v98) ==== 103+0+11 (secure 0 0 0) 0x7f98f809bc70 con 0x7f9910101960 2024-05-09T09:44:11.387 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 msgr2=0x7f98fc0a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 0x7f98fc0a50d0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f9900005fd0 tx=0x7f9900005850 comp rx=0 tx=0).stop 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 msgr2=0x7f9910198130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 crypto rx=0x7f98f800e9f0 tx=0x7f98f800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 shutdown_connections 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f98fc0a2c10 0x7f98fc0a50d0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f9910102320 0x7f991019f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.389 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f9910101960 0x7f9910198130 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 --2- 172.21.15.69:0/1934562688 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f9910100770 0x7f9910197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 >> 172.21.15.69:0/1934562688 conn(0x7f99100fbce0 msgr2=0x7f99100fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 shutdown_connections 2024-05-09T09:44:11.390 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.388+0000 7f9916829640 1 -- 172.21.15.69:0/1934562688 wait complete. 2024-05-09T09:44:11.400 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 -- 172.21.15.69:0/1203329749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0102330 msgr2=0x7faef0108bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 --2- 172.21.15.69:0/1203329749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0102330 0x7faef0108bc0 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 crypto rx=0x7faee8009ca0 tx=0x7faee802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 -- 172.21.15.69:0/1203329749 shutdown_connections 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 --2- 172.21.15.69:0/1203329749 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0102330 0x7faef0108bc0 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 --2- 172.21.15.69:0/1203329749 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0101970 0x7faef0101df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 --2- 172.21.15.69:0/1203329749 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef0100ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 -- 172.21.15.69:0/1203329749 >> 172.21.15.69:0/1203329749 conn(0x7faef00fbd10 msgr2=0x7faef00fe130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 -- 172.21.15.69:0/1203329749 shutdown_connections 2024-05-09T09:44:11.504 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.501+0000 7faef7d34640 1 -- 172.21.15.69:0/1203329749 wait complete. 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 Processor -- start 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 -- start start 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef006aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 0x7faef019eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7faef010c400 con 0x7faef0101970 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7faef010c280 con 0x7faef0102330 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef7d34640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7faef010c580 con 0x7faef0100780 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:57596/0 (socket says 172.21.15.69:57596) 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 -- 172.21.15.69:0/1595444260 learned_addr learned my addr 172.21.15.69:0/1595444260 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef62d1640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 0x7faef019eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef5ad0640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef006aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.505 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 -- 172.21.15.69:0/1595444260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 msgr2=0x7faef006aa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef006aa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 -- 172.21.15.69:0/1595444260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 msgr2=0x7faef019eed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 0x7faef019eed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef52cf640 1 -- 172.21.15.69:0/1595444260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7faef019f7e0 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef62d1640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 0x7faef019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.502+0000 7faef5ad0640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef006aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faef52cf640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7faee000c890 tx=0x7faee000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee0004220 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7faee00043c0 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7faef019fa70 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7faee00049e0 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.503+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7faef019ffc0 con 0x7faef0101970 2024-05-09T09:44:11.506 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.504+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7faef0107a40 con 0x7faef0101970 2024-05-09T09:44:11.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.505+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7faee00079b0 con 0x7faef0101970 2024-05-09T09:44:11.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.505+0000 7faee6ffd640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 0x7faedc0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:11.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.505+0000 7faef5ad0640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 0x7faedc0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:11.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.506+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7faee00cf4d0 con 0x7faef0101970 2024-05-09T09:44:11.507 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.506+0000 7faef5ad0640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 0x7faedc0a5050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faeec0058c0 tx=0x7faeec005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:11.510 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.510+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7faee0014030 con 0x7faef0101970 2024-05-09T09:44:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:11 smithi103 ceph-mon[34421]: pgmap v1128: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.1 MiB/s wr, 541 op/s 2024-05-09T09:44:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:11 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/296652352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:11 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3703686992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:11 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/95763775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:11.733 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.733+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) -- 0x7faef0062430 con 0x7faef0101970 2024-05-09T09:44:11.741 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.741+0000 7faee6ffd640 1 -- 172.21.15.69:0/1595444260 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v18) ==== 81+0+740371 (secure 0 0 0) 0x7faee009c0b0 con 0x7faef0101970 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "epoch": 18, 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "active_gid": 14152, 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "active_name": "x", 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "active_addrs": { 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-05-09T09:44:11.752 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6800", 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3550836912 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v1", 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:6801", 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3550836912 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "active_addr": "172.21.15.69:6801/3550836912", 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "active_change": "2024-05-09T09:09:29.020203+0000", 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "active_mgr_features": 4540701547738038271, 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "available": true, 2024-05-09T09:44:11.753 INFO:teuthology.orchestra.run.smithi069.stdout: "standbys": [ 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "gid": 24109, 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "z", 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_features": 4540701547738038271, 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "available_modules": [ 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "alerts", 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.754 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.755 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.756 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.757 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.758 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.759 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_destination": { 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_destination", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.760 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_from_name": { 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_from_name", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "Ceph", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.761 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email From: name", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_host": { 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_host", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.762 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP server", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_password": { 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_password", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.763 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Password to authenticate with", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_port": { 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_port", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "465", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.764 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP port", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_sender": { 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_sender", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:44:11.765 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_ssl": { 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_ssl", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.766 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_user": { 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_user", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.767 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "User to authenticate as", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "balancer", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "active": { 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "active", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.768 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_time": { 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_time", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.769 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0000", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_weekday": { 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_weekday", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.770 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_max_iterations": { 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.771 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "25", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "250", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_metrics": { 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_metrics", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.772 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_step": { 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_step", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.5", 2024-05-09T09:44:11.773 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0.001", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "0.999", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "end_time": { 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_time", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2359", 2024-05-09T09:44:11.774 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "end_weekday": { 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_weekday", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:11.775 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.776 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.777 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.778 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.779 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.780 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "min_score": { 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_score", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.781 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "upmap", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.782 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-compat", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "none", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "read", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap-read" 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Balancer mode", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_ids": { 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_ids", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.783 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.784 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_deviation": { 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_deviation", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.785 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_optimizations": { 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:11.786 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.787 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_down_multiplier": { 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_down_multiplier", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.788 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_refresh_rate": { 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_refresh_rate", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "20", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.789 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_starting_port": { 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_starting_port", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4721", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_ptrace": { 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_ptrace", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.791 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_interval": { 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_interval", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:44:11.792 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.7", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm_log_destination": { 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm_log_destination", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.793 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "file", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "file,syslog", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "syslog" 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.794 INFO:teuthology.orchestra.run.smithi069.stdout: "cgroups_split": { 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cgroups_split", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.795 INFO:teuthology.orchestra.run.smithi069.stdout: "config_checks_enabled": { 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_checks_enabled", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "config_dashboard": { 2024-05-09T09:44:11.796 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_dashboard", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_alertmanager": { 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_alertmanager", 2024-05-09T09:44:11.797 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_base": { 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_base", 2024-05-09T09:44:11.798 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_elasticsearch": { 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.799 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "elasticsearch container image", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_grafana": { 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_grafana", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.800 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_haproxy": { 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_haproxy", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.801 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "HAproxy container image", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_agent": { 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.802 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_collector": { 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.803 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_query": { 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.804 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger query container image", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_keepalived": { 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_keepalived", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.805 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Keepalived container image", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_loki": { 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_loki", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.806 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Loki container image", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_node_exporter": { 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_node_exporter", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.807 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_nvmeof": { 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_nvmeof", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:44:11.808 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Nvme-of container image", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_prometheus": { 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_prometheus", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.809 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_promtail": { 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_promtail", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:44:11.810 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Promtail container image", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_samba": { 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_samba", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.811 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_snmp_gateway": { 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.812 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "container_init": { 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_init", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.813 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "daemon_cache_timeout": { 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.814 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "900", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.815 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "default_registry": { 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_registry", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.816 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "device_cache_timeout": { 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_cache_timeout", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1800", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.817 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "device_enhanced_scan": { 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_enhanced_scan", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.818 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "facts_cache_timeout": { 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "facts_cache_timeout", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.819 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "grafana_dashboards_path": { 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.820 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "host_check_interval": { 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "host_check_interval", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:44:11.821 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "hw_monitoring": { 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hw_monitoring", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:44:11.822 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "inventory_list_all": { 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "inventory_list_all", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.823 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.824 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "log_refresh_metadata": { 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_refresh_metadata", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.825 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.826 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.827 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.828 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.830 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "*", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "max_count_per_host": { 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_count_per_host", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.832 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd_draining_count": { 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_osd_draining_count", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "migration_current": { 2024-05-09T09:44:11.833 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "migration_current", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "internal - do not modify", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:11.834 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "root", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm-package", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "root" 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "oob_default_addr": { 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "oob_default_addr", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "169.254.1.1", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.836 INFO:teuthology.orchestra.run.smithi069.stdout: "prometheus_alerts_path": { 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.837 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_insecure": { 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_insecure", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.838 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_password": { 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_password", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_url": { 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_url", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.840 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_username": { 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_username", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.841 INFO:teuthology.orchestra.run.smithi069.stdout: "secure_monitoring_stack": { 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "service_discovery_port": { 2024-05-09T09:44:11.842 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "service_discovery_port", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8765", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "ssh_config_file": { 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssh_config_file", 2024-05-09T09:44:11.843 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "use_agent": { 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_agent", 2024-05-09T09:44:11.844 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "use_repo_digest": { 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_repo_digest", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.845 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_failed_host_check": { 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.846 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_daemons": { 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.847 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_hosts": { 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.848 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crash", 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.849 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.851 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.852 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.853 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "retain_interval": { 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retain_interval", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "31536000", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.854 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_recent_interval": { 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_recent_interval", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1209600", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:44:11.855 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dashboard", 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.856 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.857 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.858 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.859 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.860 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.862 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.863 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.864 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.865 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.866 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.867 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.868 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.869 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.870 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.871 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_URL": { 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.872 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.873 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.874 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.875 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.876 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.877 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.878 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.879 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.880 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.881 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.882 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.883 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:44:11.884 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.885 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.886 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.887 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.888 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.889 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.890 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "45", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.891 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.892 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.893 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.894 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.895 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.896 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.897 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "cross_origin_url": { 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cross_origin_url", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.898 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "crt_file": { 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crt_file", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.899 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "debug": { 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "debug", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.900 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "jwt_token_ttl": { 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "jwt_token_ttl", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "28800", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.901 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.902 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.903 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.904 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.905 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.906 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "motd": { 2024-05-09T09:44:11.907 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "motd", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "The message of the day", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.908 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.909 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.910 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8080", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.911 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl_server_port": { 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl_server_port", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.912 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8443", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.913 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "redirect", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect" 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:11.914 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "url_prefix": { 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url_prefix", 2024-05-09T09:44:11.915 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.916 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "devicehealth", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_monitoring": { 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_monitoring", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "monitor device health metrics", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.917 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.918 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.919 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.920 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.922 INFO:teuthology.orchestra.run.smithi069.stdout: "mark_out_threshold": { 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mark_out_threshold", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2419200", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": { 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_name", 2024-05-09T09:44:11.923 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "device_health_metrics", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "retention_period": { 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retention_period", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.924 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15552000", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_frequency": { 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_frequency", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.925 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "self_heal": { 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "self_heal", 2024-05-09T09:44:11.926 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.927 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_threshold": { 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_threshold", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.928 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7257600", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "diskprediction_local", 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.929 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.930 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.931 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.932 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.933 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.934 INFO:teuthology.orchestra.run.smithi069.stdout: "predict_interval": { 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predict_interval", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "predictor_model": { 2024-05-09T09:44:11.935 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predictor_model", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "prophetstor", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:11.936 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.937 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "influx", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": false, 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "batch_size": { 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "batch_size", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5000", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:44:11.938 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "database": { 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "database", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "ceph", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.939 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": { 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hostname", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:44:11.940 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "30", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "5", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:44:11.941 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.942 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.943 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.944 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.945 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "password": { 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "password", 2024-05-09T09:44:11.946 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "port": { 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "port", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.947 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8086", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server port", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.948 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "false", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "threads": { 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threads", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.949 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "32", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "username": { 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "username", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.950 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "verify_ssl": { 2024-05-09T09:44:11.951 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "verify_ssl", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "true", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "insights", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.953 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.954 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.955 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.956 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.957 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "iostat", 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.958 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.959 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.960 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.961 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.962 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "k8sevents", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.963 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_event_retention_days": { 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "config_check_secs": { 2024-05-09T09:44:11.964 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_check_secs", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "10", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.965 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.966 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.967 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.968 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.969 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "localpool", 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.970 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "failure_domain": { 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "failure_domain", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "host", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.971 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.972 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.973 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.974 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.975 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": { 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_size", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.976 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "num_rep": { 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "num_rep", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.977 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": { 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_num", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.978 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "128", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "prefix": { 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prefix", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.979 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "subtree": { 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "subtree", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "rack", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.980 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mds_autoscaler", 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.981 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.982 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.983 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.984 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.985 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirroring", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.987 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.988 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.989 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.990 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.991 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "nfs", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.992 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.993 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.994 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.995 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.996 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:11.997 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "fail_fs": { 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "fail_fs", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:11.998 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:11.999 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.000 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.001 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.002 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator": { 2024-05-09T09:44:12.003 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "rook", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "test_orchestrator" 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Orchestrator backend", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.004 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_perf_query", 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.005 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.006 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.007 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.008 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.009 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_support", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.010 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.011 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.012 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.013 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.014 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.015 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_autoscaler", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.016 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.017 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.018 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.019 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.020 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.021 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "threshold": { 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threshold", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1.0", 2024-05-09T09:44:12.022 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "scaling threshold", 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "progress", 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.023 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_pg_recovery_event": { 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.024 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.025 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.026 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.027 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.028 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.029 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "max_completed_events": { 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_completed_events", 2024-05-09T09:44:12.030 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "50", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.031 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.032 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "cache": { 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cache", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.033 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "exclude_perf_counters": { 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "exclude_perf_counters", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.034 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.035 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.036 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.037 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.038 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.039 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools": { 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:44:12.040 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "300", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_interval": { 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_interval", 2024-05-09T09:44:12.041 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15.0", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.042 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.043 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "9283", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "stale_cache_strategy": { 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stale_cache_strategy", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.044 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "log", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.045 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "default", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "default", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "error" 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:12.046 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.047 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_support", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.048 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.049 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.050 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.051 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.052 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_snap_create": { 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.053 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.054 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "trash_purge_schedule": { 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "trash_purge_schedule", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.055 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "restful", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_auth": { 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_auth", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.056 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.057 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.058 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.059 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.060 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.061 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.062 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.063 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.064 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rgw", 2024-05-09T09:44:12.065 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.066 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.067 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.068 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.069 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.070 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rook", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.071 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.072 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.073 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.074 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.075 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_class": { 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "storage_class", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "local", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.076 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "selftest", 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.077 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.078 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.079 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.080 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.081 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "roption1": { 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption1", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.082 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "roption2": { 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption2", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "xyz", 2024-05-09T09:44:12.083 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption1": { 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption1", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.084 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption2": { 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption2", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.085 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption3": { 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption3", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.086 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption4": { 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption4", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.087 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption5": { 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption5", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.088 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption6": { 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption6", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.089 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption7": { 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption7", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "42", 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.090 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "testkey": { 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testkey", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.091 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "testlkey": { 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testlkey", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.092 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "testnewline": { 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testnewline", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.093 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snap_schedule", 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_m_granularity": { 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_m_granularity", 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.094 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "dump_on_update": { 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dump_on_update", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.095 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.096 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.097 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.098 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.099 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.100 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stats", 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.101 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.102 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.103 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.104 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.105 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.106 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "status", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.107 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.108 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.109 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.110 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.111 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telegraf", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "address": { 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "address", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:44:12.112 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.113 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.114 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.115 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.116 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.117 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.118 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telemetry", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_basic": { 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_basic", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.119 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_crash": { 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_crash", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.120 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_device": { 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_device", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.121 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_ident": { 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_ident", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.122 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_perf": { 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_perf", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.123 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "contact": { 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "contact", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.124 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "description": { 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "description", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.125 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "device_url": { 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_url", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.126 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.127 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.128 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "24", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "8", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "last_opt_revision": { 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "last_opt_revision", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.129 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard": { 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.130 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard_description": { 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard_description", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.131 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.132 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.133 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.134 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.135 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.136 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "organization": { 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "organization", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.137 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "proxy": { 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "proxy", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.138 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "url": { 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.139 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "test_orchestrator", 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.140 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.141 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.142 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.143 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.144 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "volumes", 2024-05-09T09:44:12.145 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.146 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.147 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.148 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.149 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_clones": { 2024-05-09T09:44:12.150 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_clones", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "periodic_async_work": { 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "periodic_async_work", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.151 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Periodically check for async work", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_delay": { 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.152 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.153 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.154 INFO:teuthology.orchestra.run.smithi069.stdout: "discovery_interval": { 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "discovery_interval", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "100", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "identifier": { 2024-05-09T09:44:12.155 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "identifier", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.156 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.157 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.158 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.159 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.160 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.161 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_host": { 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_host", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.162 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_port": { 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_port", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10051", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.163 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_sender": { 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_sender", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.164 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "gid": 24113, 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "y", 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "mgr_features": 4540701547738038271, 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: "available_modules": [ 2024-05-09T09:44:12.165 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "alerts", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.166 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.167 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.168 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.169 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.170 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.171 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_destination": { 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_destination", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.172 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_from_name": { 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_from_name", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "Ceph", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email From: name", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_host": { 2024-05-09T09:44:12.173 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_host", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP server", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_password": { 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_password", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.174 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Password to authenticate with", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_port": { 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_port", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.175 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "465", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP port", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_sender": { 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_sender", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.176 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_ssl": { 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_ssl", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.177 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_user": { 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_user", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.178 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "User to authenticate as", 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "balancer", 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "active": { 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "active", 2024-05-09T09:44:12.179 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_time": { 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_time", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.180 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0000", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_weekday": { 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_weekday", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:12.181 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_max_iterations": { 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "25", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "250", 2024-05-09T09:44:12.182 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_metrics": { 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_metrics", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:44:12.183 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_step": { 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_step", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.5", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0.001", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "0.999", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.184 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "end_time": { 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_time", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2359", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "end_weekday": { 2024-05-09T09:44:12.185 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_weekday", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.186 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.187 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.188 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.189 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.190 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "min_score": { 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_score", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.191 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "upmap", 2024-05-09T09:44:12.192 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-compat", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "none", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "read", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap-read" 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Balancer mode", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_ids": { 2024-05-09T09:44:12.193 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_ids", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.194 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_deviation": { 2024-05-09T09:44:12.195 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_deviation", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.196 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_optimizations": { 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.197 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_down_multiplier": { 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_down_multiplier", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.198 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_refresh_rate": { 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_refresh_rate", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "20", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.199 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_starting_port": { 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_starting_port", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4721", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.200 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_ptrace": { 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_ptrace", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:44:12.201 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_interval": { 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_interval", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.202 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.7", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.203 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm_log_destination": { 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm_log_destination", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "file", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "file,syslog", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "syslog" 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:44:12.204 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "cgroups_split": { 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cgroups_split", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.205 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "config_checks_enabled": { 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_checks_enabled", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.206 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "config_dashboard": { 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_dashboard", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.207 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_alertmanager": { 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_alertmanager", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_base": { 2024-05-09T09:44:12.208 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_base", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_elasticsearch": { 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.209 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "elasticsearch container image", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_grafana": { 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_grafana", 2024-05-09T09:44:12.210 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_haproxy": { 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_haproxy", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.211 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "HAproxy container image", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_agent": { 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:44:12.212 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_collector": { 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.213 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_query": { 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:44:12.214 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger query container image", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_keepalived": { 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_keepalived", 2024-05-09T09:44:12.215 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Keepalived container image", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_loki": { 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_loki", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.216 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Loki container image", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_node_exporter": { 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_node_exporter", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.217 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_nvmeof": { 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_nvmeof", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.218 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Nvme-of container image", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_prometheus": { 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_prometheus", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.219 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_promtail": { 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_promtail", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.220 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Promtail container image", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_samba": { 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_samba", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.221 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_snmp_gateway": { 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.222 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "container_init": { 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_init", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.223 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "daemon_cache_timeout": { 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.224 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "900", 2024-05-09T09:44:12.225 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "default_registry": { 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_registry", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io", 2024-05-09T09:44:12.226 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "device_cache_timeout": { 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_cache_timeout", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1800", 2024-05-09T09:44:12.227 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "device_enhanced_scan": { 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_enhanced_scan", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.228 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "facts_cache_timeout": { 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "facts_cache_timeout", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.229 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "grafana_dashboards_path": { 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:44:12.230 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "host_check_interval": { 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "host_check_interval", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.231 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "hw_monitoring": { 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hw_monitoring", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.232 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "inventory_list_all": { 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "inventory_list_all", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.233 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.234 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "log_refresh_metadata": { 2024-05-09T09:44:12.235 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_refresh_metadata", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.236 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.237 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.238 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.239 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.240 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "*", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.241 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "max_count_per_host": { 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_count_per_host", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.242 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd_draining_count": { 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_osd_draining_count", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.243 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "migration_current": { 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "migration_current", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "internal - do not modify", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.244 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "root", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm-package", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "root" 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:44:12.245 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "oob_default_addr": { 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "oob_default_addr", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "169.254.1.1", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:44:12.246 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "prometheus_alerts_path": { 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:44:12.247 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_insecure": { 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_insecure", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:44:12.248 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_password": { 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_password", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:44:12.249 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_url": { 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_url", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:44:12.250 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_username": { 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_username", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.251 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "secure_monitoring_stack": { 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.252 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "service_discovery_port": { 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "service_discovery_port", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8765", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.253 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "ssh_config_file": { 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssh_config_file", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.254 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "use_agent": { 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_agent", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.255 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "use_repo_digest": { 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_repo_digest", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.256 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_failed_host_check": { 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_daemons": { 2024-05-09T09:44:12.257 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_hosts": { 2024-05-09T09:44:12.258 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.259 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crash", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.260 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.261 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.262 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.263 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.264 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "retain_interval": { 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retain_interval", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "31536000", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:44:12.265 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_recent_interval": { 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_recent_interval", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1209600", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:44:12.266 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dashboard", 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.267 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.268 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.269 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.270 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.271 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.272 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.273 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.274 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.275 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.276 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.277 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.278 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.279 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.280 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.281 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.282 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_URL": { 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.283 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.284 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.285 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.286 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.287 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.288 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.289 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.290 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.291 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:44:12.292 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:44:12.293 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:44:12.294 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:44:12.295 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:44:12.296 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.297 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.298 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.299 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.300 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.301 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "45", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.302 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.303 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.304 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.305 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.306 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.307 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.308 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "cross_origin_url": { 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cross_origin_url", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.309 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "crt_file": { 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crt_file", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.310 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "debug": { 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "debug", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.311 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "jwt_token_ttl": { 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "jwt_token_ttl", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "28800", 2024-05-09T09:44:12.312 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.313 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.314 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.315 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.316 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.317 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.318 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "motd": { 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "motd", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.319 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "The message of the day", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.320 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.321 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8080", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.322 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.323 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl_server_port": { 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl_server_port", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8443", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.324 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "redirect", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect" 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.325 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.326 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "url_prefix": { 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url_prefix", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.327 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "devicehealth", 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_monitoring": { 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_monitoring", 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.328 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "monitor device health metrics", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.329 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.330 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.331 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.332 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.333 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "mark_out_threshold": { 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mark_out_threshold", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.334 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2419200", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": { 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_name", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.335 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "device_health_metrics", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "retention_period": { 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retention_period", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15552000", 2024-05-09T09:44:12.336 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_frequency": { 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_frequency", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.337 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "self_heal": { 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "self_heal", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.338 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:44:12.339 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_threshold": { 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_threshold", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7257600", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.340 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "diskprediction_local", 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.341 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.342 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.343 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.344 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.345 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "predict_interval": { 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predict_interval", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.346 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "predictor_model": { 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predictor_model", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "prophetstor", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.347 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.348 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "influx", 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": false, 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "batch_size": { 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "batch_size", 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.349 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5000", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "database": { 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "database", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.350 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "ceph", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": { 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hostname", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.351 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.352 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "30", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "5", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.353 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.354 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.355 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.356 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.357 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "password": { 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "password", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.358 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "port": { 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "port", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8086", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.359 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server port", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "false", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.360 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "threads": { 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threads", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "32", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:44:12.361 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "username": { 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "username", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:44:12.362 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "verify_ssl": { 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "verify_ssl", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "true", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.363 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "insights", 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.364 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.365 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.366 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.367 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.368 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "iostat", 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.369 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.370 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.371 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.372 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.373 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "k8sevents", 2024-05-09T09:44:12.374 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_event_retention_days": { 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.375 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "config_check_secs": { 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_check_secs", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "10", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.376 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.377 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.378 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.379 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.380 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.381 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "localpool", 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.382 INFO:teuthology.orchestra.run.smithi069.stdout: "failure_domain": { 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "failure_domain", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "host", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.383 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.384 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.385 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.386 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.387 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": { 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_size", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.388 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "num_rep": { 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "num_rep", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.389 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": { 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_num", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.390 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "128", 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.391 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "prefix": { 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prefix", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.392 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "subtree": { 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "subtree", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "rack", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:44:12.393 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mds_autoscaler", 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.394 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.395 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.396 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.397 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.398 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirroring", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.399 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.400 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.401 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.402 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.403 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.404 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "nfs", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.405 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.406 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.407 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.408 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.409 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "fail_fs": { 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "fail_fs", 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.410 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.411 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.412 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.413 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.414 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.415 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator": { 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.416 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm", 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "rook", 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "test_orchestrator" 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Orchestrator backend", 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_perf_query", 2024-05-09T09:44:12.417 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.418 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.419 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.420 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.421 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.422 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_support", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.423 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.424 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.425 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.426 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.427 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_autoscaler", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.428 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.429 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.430 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.431 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.432 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.433 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "threshold": { 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threshold", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1.0", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "scaling threshold", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.434 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "progress", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_pg_recovery_event": { 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.435 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.436 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.437 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.438 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.439 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.440 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.441 INFO:teuthology.orchestra.run.smithi069.stdout: "max_completed_events": { 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_completed_events", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "50", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.442 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.443 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "cache": { 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cache", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.444 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "exclude_perf_counters": { 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "exclude_perf_counters", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:44:12.445 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.446 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.447 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.448 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.449 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.450 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools": { 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.451 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "300", 2024-05-09T09:44:12.452 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_interval": { 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_interval", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15.0", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.453 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.454 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "9283", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.455 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "stale_cache_strategy": { 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stale_cache_strategy", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "log", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.456 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "default", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "default", 2024-05-09T09:44:12.457 INFO:teuthology.orchestra.run.smithi069.stdout: "error" 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:12.458 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_support", 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.459 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.460 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.461 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.462 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.463 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_snap_create": { 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.464 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.465 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "trash_purge_schedule": { 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "trash_purge_schedule", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.466 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "restful", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.467 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_auth": { 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_auth", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.468 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.469 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.470 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.471 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.472 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.473 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.474 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.475 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.476 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rgw", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.477 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.478 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.479 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.480 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.481 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rook", 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.482 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.483 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.484 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.485 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.486 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_class": { 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "storage_class", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "local", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.487 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "selftest", 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.488 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.489 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.490 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.491 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.492 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "roption1": { 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption1", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.493 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "roption2": { 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption2", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "xyz", 2024-05-09T09:44:12.494 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption1": { 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption1", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.495 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption2": { 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption2", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.496 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption3": { 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption3", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.497 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption4": { 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption4", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.498 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption5": { 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption5", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.499 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption6": { 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption6", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.500 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption7": { 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption7", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "42", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.501 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "testkey": { 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testkey", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.502 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "testlkey": { 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testlkey", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.503 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "testnewline": { 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testnewline", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.504 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snap_schedule", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_m_granularity": { 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_m_granularity", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.505 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "dump_on_update": { 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dump_on_update", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.506 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.507 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.508 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.509 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.510 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.511 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stats", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.512 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.513 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.514 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.515 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.516 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "status", 2024-05-09T09:44:12.517 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.518 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.519 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.520 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.521 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.522 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telegraf", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "address": { 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "address", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.523 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.524 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15", 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.525 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.526 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.527 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.528 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.529 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.530 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telemetry", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_basic": { 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_basic", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.531 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_crash": { 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_crash", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.532 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_device": { 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_device", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:44:12.533 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_ident": { 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_ident", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:44:12.534 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_perf": { 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_perf", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.535 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "contact": { 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "contact", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.536 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "description": { 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "description", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.537 INFO:teuthology.orchestra.run.smithi069.stdout: "device_url": { 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_url", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.538 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.539 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "24", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "8", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "last_opt_revision": { 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "last_opt_revision", 2024-05-09T09:44:12.540 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard": { 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.541 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard_description": { 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard_description", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.542 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.543 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.544 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.545 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.546 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.547 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "organization": { 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "organization", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.548 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "proxy": { 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "proxy", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.549 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "url": { 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.550 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.551 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "test_orchestrator", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.552 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.553 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.554 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.555 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.556 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "volumes", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.557 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.558 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.559 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.560 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.561 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_clones": { 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_clones", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.562 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "periodic_async_work": { 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "periodic_async_work", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Periodically check for async work", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.563 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_delay": { 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.564 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.565 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "discovery_interval": { 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "discovery_interval", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "100", 2024-05-09T09:44:12.566 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "identifier": { 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "identifier", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.567 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.568 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.569 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.570 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.571 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.572 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.573 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_host": { 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_host", 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.574 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_port": { 2024-05-09T09:44:12.577 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_port", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10051", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_sender": { 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_sender", 2024-05-09T09:44:12.578 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:12.579 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "modules": [ 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "dashboard", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "iostat", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "nfs", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "restful" 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "available_modules": [ 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "alerts", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.580 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How frequently to reexamine health status", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.581 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.582 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.583 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.584 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.585 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_destination": { 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_destination", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.586 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email address to send alerts to", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_from_name": { 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_from_name", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.587 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "Ceph", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Email From: name", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_host": { 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_host", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.588 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP server", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_password": { 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_password", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.589 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Password to authenticate with", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_port": { 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_port", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.590 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "465", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP port", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_sender": { 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_sender", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.591 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SMTP envelope sender", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_ssl": { 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_ssl", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.592 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use SSL to connect to SMTP server", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "smtp_user": { 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "smtp_user", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.593 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "User to authenticate as", 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.594 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "balancer", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "active": { 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "active", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically balance PGs across cluster", 2024-05-09T09:44:12.595 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_time": { 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_time", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0000", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "beginning time of day to automatically balance", 2024-05-09T09:44:12.596 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "begin_weekday": { 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "begin_weekday", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-05-09T09:44:12.597 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_max_iterations": { 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_max_iterations", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "25", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "250", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.598 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_metrics": { 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_metrics", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "pgs,objects,bytes", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-05-09T09:44:12.599 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "crush_compat_step": { 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crush_compat_step", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.5", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0.001", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "0.999", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "aggressiveness of optimization", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.600 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "end_time": { 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_time", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2359", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ending time of day to automatically balance", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.601 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "end_weekday": { 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "end_weekday", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "0", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "6", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.602 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.603 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.604 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.605 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.606 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.607 INFO:teuthology.orchestra.run.smithi069.stdout: "min_score": { 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_score", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.608 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "upmap", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "crush-compat", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "none", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "read", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap", 2024-05-09T09:44:12.609 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap-read" 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Balancer mode", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_ids": { 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_ids", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.610 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.611 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_deviation": { 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_deviation", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.612 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "deviation below which no optimization is attempted", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "upmap_max_optimizations": { 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "upmap_max_optimizations", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-05-09T09:44:12.613 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm", 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_down_multiplier": { 2024-05-09T09:44:12.614 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_down_multiplier", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_refresh_rate": { 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_refresh_rate", 2024-05-09T09:44:12.615 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "20", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "agent_starting_port": { 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "agent_starting_port", 2024-05-09T09:44:12.616 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4721", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_ptrace": { 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_ptrace", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.617 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_interval": { 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_interval", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.618 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to autotune daemon memory", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "autotune_memory_target_ratio": { 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "autotune_memory_target_ratio", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.619 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0.7", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm_log_destination": { 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cephadm_log_destination", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.620 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "file", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "file,syslog", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "syslog" 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "cgroups_split": { 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cgroups_split", 2024-05-09T09:44:12.621 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "config_checks_enabled": { 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_checks_enabled", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.622 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "config_dashboard": { 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_dashboard", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.623 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_alertmanager": { 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_alertmanager", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.624 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_base": { 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_base", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.625 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/ceph", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Container image name, without the tag", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_elasticsearch": { 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_elasticsearch", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.626 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "elasticsearch container image", 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.627 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_grafana": { 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_grafana", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/grafana:9.4.12", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.628 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_haproxy": { 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_haproxy", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "HAproxy container image", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.629 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_agent": { 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_agent", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger agent container image", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.630 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_collector": { 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_collector", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger collector container image", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_jaeger_query": { 2024-05-09T09:44:12.631 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_jaeger_query", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Jaeger query container image", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_keepalived": { 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_keepalived", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.632 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Keepalived container image", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_loki": { 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_loki", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-05-09T09:44:12.633 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Loki container image", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_node_exporter": { 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_node_exporter", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.634 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_nvmeof": { 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_nvmeof", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/ceph/nvmeof:1.2.1", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.635 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Nvme-of container image", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_prometheus": { 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_prometheus", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Prometheus container image", 2024-05-09T09:44:12.636 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_promtail": { 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_promtail", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Promtail container image", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.637 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_samba": { 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_samba", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "quay.io/samba.org/samba-server:devbuilds-centos-amd64", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Samba/SMB container image", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.638 INFO:teuthology.orchestra.run.smithi069.stdout: "container_image_snmp_gateway": { 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_image_snmp_gateway", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "SNMP Gateway container image", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.639 INFO:teuthology.orchestra.run.smithi069.stdout: "container_init": { 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "container_init", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Run podman/docker with `--init`", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "daemon_cache_timeout": { 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "daemon_cache_timeout", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.640 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "default_cephadm_command_timeout": { 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_cephadm_command_timeout", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.641 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "900", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "default_registry": { 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "default_registry", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "docker.io", 2024-05-09T09:44:12.642 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "device_cache_timeout": { 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_cache_timeout", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.643 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1800", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache device inventory", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "device_enhanced_scan": { 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_enhanced_scan", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.644 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use libstoragemgmt during device scans", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "facts_cache_timeout": { 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "facts_cache_timeout", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.645 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "seconds to cache host facts data", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "grafana_dashboards_path": { 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "grafana_dashboards_path", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/grafana/dashboards/ceph-dashboard/", 2024-05-09T09:44:12.646 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of dashboards to include in grafana deployments", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "host_check_interval": { 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "host_check_interval", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.647 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to perform a host check", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "hw_monitoring": { 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hw_monitoring", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-05-09T09:44:12.648 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "inventory_list_all": { 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "inventory_list_all", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.649 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.650 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "log_refresh_metadata": { 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_refresh_metadata", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.651 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.652 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.653 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.654 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf": { 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.655 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.656 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "*", 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "max_count_per_host": { 2024-05-09T09:44:12.657 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_count_per_host", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of daemons per service per host", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.658 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "max_osd_draining_count": { 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_osd_draining_count", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-05-09T09:44:12.659 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "migration_current": { 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "migration_current", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.660 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "internal - do not modify", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "mode": { 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mode", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "root", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.661 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm-package", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "root" 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "mode for remote execution of cephadm", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "oob_default_addr": { 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "oob_default_addr", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.662 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "169.254.1.1", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Default address for RedFish API (oob management).", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "prometheus_alerts_path": { 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus_alerts_path", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.663 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_insecure": { 2024-05-09T09:44:12.664 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_insecure", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.665 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_password": { 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_password", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.666 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_url": { 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_url", 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.667 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "registry_username": { 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "registry_username", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.668 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.669 INFO:teuthology.orchestra.run.smithi069.stdout: "secure_monitoring_stack": { 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "secure_monitoring_stack", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.670 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "service_discovery_port": { 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "service_discovery_port", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8765", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "cephadm service discovery port", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.671 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "ssh_config_file": { 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssh_config_file", 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.672 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "use_agent": { 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_agent", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.673 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "use_repo_digest": { 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "use_repo_digest", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.674 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_failed_host_check": { 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_failed_host_check", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.675 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if the host check fails", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_daemons": { 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_daemons", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.676 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_on_stray_hosts": { 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_on_stray_hosts", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.677 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crash", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.678 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.679 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.680 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.681 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.682 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.683 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "retain_interval": { 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retain_interval", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "31536000", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain crashes before pruning them", 2024-05-09T09:44:12.684 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_recent_interval": { 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_recent_interval", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1209600", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "time interval in which to warn about recent crashes", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.685 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dashboard", 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.686 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_HOST": { 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_HOST", 2024-05-09T09:44:12.687 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-05-09T09:44:12.688 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_ENABLED": { 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_ENABLED", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.689 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.690 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "ENABLE_BROWSABLE_API": { 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ENABLE_BROWSABLE_API", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.691 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.692 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.693 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.694 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.695 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.696 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_NFS": { 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.697 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RBD": { 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.698 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "FEATURE_TOGGLE_RGW": { 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-05-09T09:44:12.699 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_PASSWORD": { 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_PASSWORD", 2024-05-09T09:44:12.700 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-05-09T09:44:12.701 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_URL": { 2024-05-09T09:44:12.702 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_URL", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.703 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_API_USERNAME": { 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_API_USERNAME", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.704 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.705 INFO:teuthology.orchestra.run.smithi069.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.706 INFO:teuthology.orchestra.run.smithi069.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.707 INFO:teuthology.orchestra.run.smithi069.stdout: "ISSUE_TRACKER_API_KEY": { 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.708 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_HOST": { 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_HOST", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.709 INFO:teuthology.orchestra.run.smithi069.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.710 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.711 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.712 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.713 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.714 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.715 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.716 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.717 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_ENABLED": { 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_ENABLED", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.718 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.719 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.720 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.721 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "REST_REQUESTS_TIMEOUT": { 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "45", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.722 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ACCESS_KEY": { 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ACCESS_KEY", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.723 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "admin", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.724 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SECRET_KEY": { 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SECRET_KEY", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.725 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "RGW_API_SSL_VERIFY": { 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "RGW_API_SSL_VERIFY", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.726 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.727 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.728 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.729 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "cross_origin_url": { 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cross_origin_url", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.730 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "crt_file": { 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "crt_file", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.731 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "debug": { 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "debug", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.732 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Enable/disable debug options", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "jwt_token_ttl": { 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "jwt_token_ttl", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "28800", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.733 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.734 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.735 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.736 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.737 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.738 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.739 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "motd": { 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "motd", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "The message of the day", 2024-05-09T09:44:12.740 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect_resolve_ip_addr": { 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "redirect_resolve_ip_addr", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.741 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.742 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8080", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.743 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.744 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl_server_port": { 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl_server_port", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8443", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.745 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "redirect", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.746 INFO:teuthology.orchestra.run.smithi069.stdout: "redirect" 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:12.747 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "url_prefix": { 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url_prefix", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.748 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "devicehealth", 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_monitoring": { 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_monitoring", 2024-05-09T09:44:12.749 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "monitor device health metrics", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.750 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.751 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.752 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.753 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.754 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "mark_out_threshold": { 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mark_out_threshold", 2024-05-09T09:44:12.755 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "2419200", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "pool_name": { 2024-05-09T09:44:12.756 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pool_name", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "device_health_metrics", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name of pool in which to store device health metrics", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "retention_period": { 2024-05-09T09:44:12.757 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "retention_period", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15552000", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long to retain device health metrics", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_frequency": { 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_frequency", 2024-05-09T09:44:12.758 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to scrape device health metrics", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "self_heal": { 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "self_heal", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.759 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.760 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how frequently to wake up and check device health", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "warn_threshold": { 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "warn_threshold", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.761 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7257600", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "diskprediction_local", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.762 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.763 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.764 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.765 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.766 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "predict_interval": { 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predict_interval", 2024-05-09T09:44:12.767 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "86400", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.768 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "predictor_model": { 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "predictor_model", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "prophetstor", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.769 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "600", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.770 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "influx", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": false, 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "influxdb python module not found", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "batch_size": { 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "batch_size", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5000", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.771 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "database": { 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "database", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "ceph", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.772 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "hostname": { 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "hostname", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.773 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server hostname", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "30", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "5", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.774 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.775 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.776 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.777 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.778 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.779 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "password": { 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "password", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "password of InfluxDB server user", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.780 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "port": { 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "port", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "8086", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "InfluxDB server port", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.781 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "ssl": { 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ssl", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "false", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.782 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "threads": { 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threads", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "32", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.783 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "username": { 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "username", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "username of InfluxDB server user", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.784 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "verify_ssl": { 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "verify_ssl", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "true", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.785 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "insights", 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.786 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.787 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.788 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.789 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.790 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "iostat", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.791 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.792 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.793 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.794 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.795 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.796 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "k8sevents", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "ceph_event_retention_days": { 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "ceph_event_retention_days", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "7", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Days to hold ceph event information within local cache", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.797 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "config_check_secs": { 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "config_check_secs", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "10", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.798 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.799 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.800 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.801 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.802 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.802 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.802 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.802 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.803 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.804 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "localpool", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "failure_domain": { 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "failure_domain", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "host", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "failure domain for any created local pool", 2024-05-09T09:44:12.805 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.806 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.807 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.808 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.809 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.810 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "min_size": { 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "min_size", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.811 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default min_size for any created local pool", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "num_rep": { 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "num_rep", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.812 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default replica count for any created local pool", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_num": { 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_num", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "128", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.813 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "default pg_num for any created local pool", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "prefix": { 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prefix", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.814 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "name prefix for any created local pool", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "subtree": { 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "subtree", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "rack", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "CRUSH level for which to create a local pool", 2024-05-09T09:44:12.815 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mds_autoscaler", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.816 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.817 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.818 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.819 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.820 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirroring", 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.821 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.822 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.823 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.824 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.825 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.826 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "nfs", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.827 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.828 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.829 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.830 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.831 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "fail_fs": { 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "fail_fs", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.832 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.833 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.834 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.835 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.836 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.838 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.839 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator": { 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "orchestrator", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "cephadm", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "rook", 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "test_orchestrator" 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.840 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Orchestrator backend", 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_perf_query", 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.841 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.842 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.843 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.844 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.845 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "osd_support", 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.846 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.847 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.848 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.849 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.850 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.851 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "pg_autoscaler", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.852 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.853 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.854 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.855 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.856 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.857 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "threshold": { 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "threshold", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "3.0", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1.0", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "scaling threshold", 2024-05-09T09:44:12.858 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "progress", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_pg_recovery_event": { 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_pg_recovery_event", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.859 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow the module to show pg recovery progress", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.860 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.861 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.862 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.863 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.864 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.865 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "max_completed_events": { 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_completed_events", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "50", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "number of past completed events to remember", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.866 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "sleep_interval": { 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "sleep_interval", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "5", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "how long the module is going to sleep", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.867 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "prometheus", 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "cache": { 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "cache", 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.868 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "exclude_perf_counters": { 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "exclude_perf_counters", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.869 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.870 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.871 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.872 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.873 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.874 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.875 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools": { 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.876 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_stats_pools_refresh_interval": { 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "300", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.877 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "scrape_interval": { 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "scrape_interval", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15.0", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.878 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "::", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.879 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "9283", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.880 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "stale_cache_strategy": { 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stale_cache_strategy", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "log", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.881 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_behaviour": { 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_behaviour", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "default", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "default", 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "error" 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.882 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "standby_error_status_code": { 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "standby_error_status_code", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "500", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "400", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "599", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.883 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_support", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.884 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.885 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.886 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.887 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.888 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_snap_create": { 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_snap_create", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.889 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "mirror_snapshot_schedule": { 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "mirror_snapshot_schedule", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.890 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "trash_purge_schedule": { 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "trash_purge_schedule", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.891 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "restful", 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "enable_auth": { 2024-05-09T09:44:12.892 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enable_auth", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "key_file": { 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "key_file", 2024-05-09T09:44:12.893 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.894 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.895 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.896 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.897 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.898 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "server_addr": { 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_addr", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.899 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "server_port": { 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "server_port", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.900 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rgw", 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.901 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.902 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.903 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.904 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.905 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.906 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rook", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.907 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.908 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.909 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.910 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.911 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "storage_class": { 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "storage_class", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.912 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "local", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "selftest", 2024-05-09T09:44:12.913 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.914 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.915 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.916 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.917 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.918 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "roption1": { 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption1", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.919 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "roption2": { 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "roption2", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "xyz", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.920 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption1": { 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption1", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.921 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption2": { 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption2", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.922 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption3": { 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption3", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "float", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.923 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption4": { 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption4", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.924 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption5": { 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption5", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.925 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption6": { 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption6", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.926 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "rwoption7": { 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rwoption7", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "1", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "42", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.927 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "testkey": { 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testkey", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.928 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "testlkey": { 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testlkey", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.929 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "testnewline": { 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "testnewline", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.930 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snap_schedule", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "allow_m_granularity": { 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "allow_m_granularity", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.931 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "allow minute scheduled snapshots", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "dump_on_update": { 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "dump_on_update", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.932 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "dump database to debug log on update", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.933 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.934 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.935 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.936 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.937 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "stats", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.938 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.939 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.940 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.941 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.942 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "status", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.943 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.944 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.945 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.946 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.947 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.948 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telegraf", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "address": { 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "address", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.949 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "15", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.950 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.951 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.952 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.953 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.954 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.955 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "telemetry", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_basic": { 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_basic", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share basic cluster information (size, version)", 2024-05-09T09:44:12.956 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_crash": { 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_crash", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.957 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_device": { 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_device", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.958 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_ident": { 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_ident", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-05-09T09:44:12.959 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "channel_perf": { 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "channel_perf", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.960 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Share various performance metrics of a cluster", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "contact": { 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "contact", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.961 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "description": { 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "description", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.962 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "device_url": { 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "device_url", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.963 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "enabled": { 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "enabled", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.964 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "24", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "8", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.965 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "last_opt_revision": { 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "last_opt_revision", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "1", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.966 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard": { 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.967 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "leaderboard_description": { 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "leaderboard_description", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.968 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.969 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.970 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.971 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.972 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.973 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "organization": { 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "organization", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.974 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "proxy": { 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "proxy", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.975 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "url": { 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "url", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.976 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "test_orchestrator", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.977 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.978 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.979 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.980 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.981 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "volumes", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.982 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.983 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.984 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "max_concurrent_clones": { 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "max_concurrent_clones", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "4", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Number of asynchronous cloner threads", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "periodic_async_work": { 2024-05-09T09:44:12.987 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "periodic_async_work", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Periodically check for async work", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_delay": { 2024-05-09T09:44:12.988 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_delay", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "0", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "snapshot_clone_no_wait": { 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "snapshot_clone_no_wait", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.989 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "True", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.990 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "can_run": true, 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "error_string": "", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "module_options": { 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "discovery_interval": { 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "discovery_interval", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "uint", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "100", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.991 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "identifier": { 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "identifier", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.992 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "interval": { 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "interval", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "secs", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "60", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.993 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "log_level": { 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_level", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.994 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster": { 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.995 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_cluster_level": { 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_cluster_level", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "info", 2024-05-09T09:44:12.996 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [ 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "critical", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "debug", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "error", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "info", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "warning" 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.997 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "log_to_file": { 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "log_to_file", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "bool", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 1, 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "False", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.998 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_host": { 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_host", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:12.999 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_port": { 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_port", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "int", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "10051", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:13.000 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "zabbix_sender": { 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "zabbix_sender", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "str", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "level": "advanced", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "flags": 0, 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "min": "", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "max": "", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "enum_allowed": [], 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "desc": "", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "long_desc": "", 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "tags": [], 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: "see_also": [] 2024-05-09T09:44:13.001 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "services": { 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "dashboard": "https://172.21.15.69:8443/" 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "always_on_modules": { 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "octopus": [ 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "balancer", 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "crash", 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "devicehealth", 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator", 2024-05-09T09:44:13.002 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscaler", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "progress", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_support", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "status", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "telemetry", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "volumes" 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "pacific": [ 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "balancer", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "crash", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "devicehealth", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscaler", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "progress", 2024-05-09T09:44:13.003 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_support", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "status", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "telemetry", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "volumes" 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "quincy": [ 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "balancer", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "crash", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "devicehealth", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscaler", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "progress", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_support", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "status", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "telemetry", 2024-05-09T09:44:13.004 INFO:teuthology.orchestra.run.smithi069.stdout: "volumes" 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "reef": [ 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "balancer", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "crash", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "devicehealth", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscaler", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "progress", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_support", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "status", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "telemetry", 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "volumes" 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: ], 2024-05-09T09:44:13.005 INFO:teuthology.orchestra.run.smithi069.stdout: "squid": [ 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "balancer", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "crash", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "devicehealth", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "orchestrator", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "pg_autoscaler", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "progress", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "rbd_support", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "status", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "telemetry", 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "volumes" 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.006 INFO:teuthology.orchestra.run.smithi069.stdout: "last_failure_osd_epoch": 3, 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "active_clients": [ 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "devicehealth", 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:0", 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2046838104 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "libcephsqlite", 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-05-09T09:44:13.007 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:0", 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 1282800614 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "rbd_support", 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:0", 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 2399833747 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.008 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: "name": "volumes", 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: "addrvec": [ 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: { 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: "type": "v2", 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: "addr": "172.21.15.69:0", 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: "nonce": 3930888378 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout: ] 2024-05-09T09:44:13.009 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:44:13.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1934562688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.011 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:12 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1595444260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:11 smithi163 ceph-mon[34354]: pgmap v1128: 129 pgs: 129 active+clean; 670 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.1 MiB/s rd, 8.1 MiB/s wr, 541 op/s 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:11 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/296652352' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:11 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3703686992' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:11 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/95763775' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1934562688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.012 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:12 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1595444260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 msgr2=0x7faedc0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 0x7faedc0a5050 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7faeec0058c0 tx=0x7faeec005d50 comp rx=0 tx=0).stop 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 msgr2=0x7faef019c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 crypto rx=0x7faee000c890 tx=0x7faee000cd50 comp rx=0 tx=0).stop 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 shutdown_connections 2024-05-09T09:44:13.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7faedc0a2b90 0x7faedc0a5050 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7faef0102330 0x7faef019eed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7faef0101970 0x7faef019c8b0 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 --2- 172.21.15.69:0/1595444260 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7faef0100780 0x7faef006aa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.746+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 >> 172.21.15.69:0/1595444260 conn(0x7faef00fbd10 msgr2=0x7faef0104c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.747+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 shutdown_connections 2024-05-09T09:44:13.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:11.747+0000 7faef7d34640 1 -- 172.21.15.69:0/1595444260 wait complete. 2024-05-09T09:44:13.019 DEBUG:tasks.check_counter:Getting stats from a 2024-05-09T09:44:13.019 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.a perf dump 2024-05-09T09:44:13.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1934562688' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:13.020 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:12 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1595444260' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-05-09T09:44:13.276 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:44:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:13 smithi103 ceph-mon[34421]: pgmap v1129: 129 pgs: 129 active+clean; 466 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-05-09T09:44:13.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:13 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:13.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:13 smithi069 ceph-mon[30123]: pgmap v1129: 129 pgs: 129 active+clean; 466 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-05-09T09:44:13.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:13 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:13 smithi163 ceph-mon[34354]: pgmap v1129: 129 pgs: 129 active+clean; 466 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.7 MiB/s rd, 9.1 MiB/s wr, 645 op/s 2024-05-09T09:44:13.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:13 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:15.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:15 smithi103 ceph-mon[34421]: pgmap v1130: 129 pgs: 129 active+clean; 447 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 529 op/s 2024-05-09T09:44:15.629 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:15 smithi069 ceph-mon[30123]: pgmap v1130: 129 pgs: 129 active+clean; 447 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 529 op/s 2024-05-09T09:44:15.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:15 smithi163 ceph-mon[34354]: pgmap v1130: 129 pgs: 129 active+clean; 447 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 MiB/s rd, 8.4 MiB/s wr, 529 op/s 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 0, 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 2, 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 0, 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 0, 2024-05-09T09:44:15.982 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.001066685, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000121756, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:15.983 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 6, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 341, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 3344, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 215792, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.022242163, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.017328923, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.002178094, 2024-05-09T09:44:15.984 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 340, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.018457526, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000054286 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 215792 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:15.985 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 447, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 444, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 543792, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 274832, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.052949053, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.023248361, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.018029525, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.008394126, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 443, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.019223003, 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000043392 2024-05-09T09:44:15.986 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 543792, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 274832 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-05-09T09:44:15.987 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:15.988 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:15.989 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 148704, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 29, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:15.990 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:15.991 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 449, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 480581, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 449, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 449, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 480581, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:15.992 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:15.993 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:15.993 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:15.993 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:44:16.701 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.a 2024-05-09T09:44:16.701 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.a 2024-05-09T09:44:16.702 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.a 2024-05-09T09:44:16.702 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.a 2024-05-09T09:44:16.702 DEBUG:tasks.check_counter:Getting stats from d 2024-05-09T09:44:16.702 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.d perf dump 2024-05-09T09:44:16.946 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:44:17.618 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:17 smithi103 ceph-mon[34421]: pgmap v1131: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 450 op/s 2024-05-09T09:44:17.672 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:17 smithi069 ceph-mon[30123]: pgmap v1131: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 450 op/s 2024-05-09T09:44:17.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:17 smithi163 ceph-mon[34354]: pgmap v1131: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.0 MiB/s rd, 7.8 MiB/s wr, 450 op/s 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 0, 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 2, 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 0, 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 0, 2024-05-09T09:44:19.138 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.001020404, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000135733, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:19.139 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 6, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 345, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 3344, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 216496, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.022665314, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.017986920, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001660896, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:19.140 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 344, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014516988, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000042200 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 216496 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 452, 2024-05-09T09:44:19.141 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 449, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 545152, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 278032, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.056841941, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.026063742, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.018578670, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.008594910, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 448, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.019887958, 2024-05-09T09:44:19.142 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000044392 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 545152, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 278032 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-05-09T09:44:19.143 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:19.144 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:19.145 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 148660, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 29, 2024-05-09T09:44:19.146 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:19.147 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 454, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 481231, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 454, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-05-09T09:44:19.148 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 454, 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 481231, 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:19.149 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:44:19.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:19 smithi069 ceph-mon[30123]: pgmap v1132: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 5.9 MiB/s wr, 386 op/s 2024-05-09T09:44:19.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:19 smithi103 ceph-mon[34421]: pgmap v1132: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 5.9 MiB/s wr, 386 op/s 2024-05-09T09:44:19.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:19 smithi163 ceph-mon[34354]: pgmap v1132: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 989 KiB/s rd, 5.9 MiB/s wr, 386 op/s 2024-05-09T09:44:20.075 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.d 2024-05-09T09:44:20.075 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.d 2024-05-09T09:44:20.075 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.d 2024-05-09T09:44:20.075 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.d 2024-05-09T09:44:20.075 DEBUG:tasks.check_counter:Getting stats from g 2024-05-09T09:44:20.075 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.g perf dump 2024-05-09T09:44:20.336 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:44:21.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:21 smithi069 ceph-mon[30123]: pgmap v1133: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.6 MiB/s wr, 337 op/s 2024-05-09T09:44:21.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:21 smithi163 ceph-mon[34354]: pgmap v1133: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.6 MiB/s wr, 337 op/s 2024-05-09T09:44:21.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:21 smithi103 ceph-mon[34421]: pgmap v1133: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 546 KiB/s rd, 3.6 MiB/s wr, 337 op/s 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 449, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 446, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 544336, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 276112, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.054260242, 2024-05-09T09:44:22.065 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.022612956, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.018596787, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.008739410, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 445, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.017948155, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000040332 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.066 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 544336, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 276112 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 6, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 343, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 3344, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 216144, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.021680137, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.017327140, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001481729, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:22.067 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 342, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.014754748, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000043142 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 216144 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:22.068 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 0, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 2, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 0, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 0, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.000327130, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000242918, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.069 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:22.070 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:22.071 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:22.072 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 148704, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 29, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:22.073 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-05-09T09:44:22.074 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 451, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 480841, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:22.075 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 451, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 451, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 480841, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:22.076 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:44:23.025 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.g 2024-05-09T09:44:23.025 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.g 2024-05-09T09:44:23.026 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.g 2024-05-09T09:44:23.026 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.g 2024-05-09T09:44:23.026 DEBUG:tasks.check_counter:Getting stats from j 2024-05-09T09:44:23.026 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.j perf dump 2024-05-09T09:44:23.278 INFO:teuthology.orchestra.run.smithi069.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.a/config 2024-05-09T09:44:23.652 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:23 smithi069 ceph-mon[30123]: pgmap v1134: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-05-09T09:44:23.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:23 smithi163 ceph-mon[34354]: pgmap v1134: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-05-09T09:44:23.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:23 smithi103 ceph-mon[34421]: pgmap v1134: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 276 KiB/s rd, 1.6 MiB/s wr, 244 op/s 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout:{ 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 455, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 452, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 545968, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 279952, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:25.017 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.058833285, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.025939487, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.019887929, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.009161493, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 451, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.038884736, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000086218 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:25.018 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 545968, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 279952 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 2, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 345, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 320, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 216400, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.022387806, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.017935885, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.001569691, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:25.019 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 343, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.013358757, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000038946 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 216400 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.020 INFO:teuthology.orchestra.run.smithi069.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_messages": 4, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages": 4, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_bytes": 3024, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_bytes": 448, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_total_time": 0.000698294, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_send_time": 0.000031698, 2024-05-09T09:44:25.021 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_recv_time": 0.000632985, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 3, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000666204, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000222068 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:25.023 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: "cct": { 2024-05-09T09:44:25.024 INFO:teuthology.orchestra.run.smithi069.stdout: "total_workers": 0, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "unhealthy_workers": 0 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "mempool": { 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:25.025 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:25.026 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:25.027 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:25.028 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_items": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:25.029 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_bytes": 148721, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_anon_items": 30, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_bytes": 0, 2024-05-09T09:44:25.030 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_items": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_items": 0, 2024-05-09T09:44:25.031 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "pgmap_items": 0, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "mds_co_items": 2, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:25.032 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_1_items": 0, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "unittest_2_items": 0 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: }, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "val": 0, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "max": 104857600, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "get_started": 0, 2024-05-09T09:44:25.033 INFO:teuthology.orchestra.run.smithi069.stdout: "get": 457, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "get_sum": 481621, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "get_or_fail_success": 457, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "take": 0, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "take_sum": 0, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "put": 457, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "put_sum": 481621, 2024-05-09T09:44:25.034 INFO:teuthology.orchestra.run.smithi069.stdout: "wait": { 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout: "avgcount": 0, 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout: "sum": 0.000000000, 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout: "avgtime": 0.000000000 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout: } 2024-05-09T09:44:25.035 INFO:teuthology.orchestra.run.smithi069.stdout:} 2024-05-09T09:44:25.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:25 smithi069 ceph-mon[30123]: pgmap v1135: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.0 KiB/s wr, 43 op/s 2024-05-09T09:44:25.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:25 smithi163 ceph-mon[34354]: pgmap v1135: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.0 KiB/s wr, 43 op/s 2024-05-09T09:44:25.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:25 smithi103 ceph-mon[34421]: pgmap v1135: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 5.0 KiB/s wr, 43 op/s 2024-05-09T09:44:26.075 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.j 2024-05-09T09:44:26.075 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.j 2024-05-09T09:44:26.075 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.j 2024-05-09T09:44:26.075 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.j 2024-05-09T09:44:26.075 DEBUG:tasks.check_counter:Getting stats from b 2024-05-09T09:44:26.075 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.b perf dump 2024-05-09T09:44:26.363 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:44:27.727 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:27 smithi103 ceph-mon[34421]: pgmap v1136: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-05-09T09:44:27.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:27 smithi069 ceph-mon[30123]: pgmap v1136: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-05-09T09:44:27.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:27 smithi163 ceph-mon[34354]: pgmap v1136: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 3.3 KiB/s wr, 13 op/s 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 513654, 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 534616, 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 122817351, 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 441058433, 2024-05-09T09:44:28.377 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 263, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 36.725653700, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 16.369547238, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 18.686691788, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 4.414813397, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 534615, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 84.806282954, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000158630 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 842454, 2024-05-09T09:44:28.378 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.163080821, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000193 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 708784, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 290992 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 756196, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1042650, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 226127010, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 890480752, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 14, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 5, 2024-05-09T09:44:28.379 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 41.138360673, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 17.884514017, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 23.933097340, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 4.757799189, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1042649, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 199.274810188, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000191123 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 859539, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.243240655, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000282 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 50096, 2024-05-09T09:44:28.380 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 724464 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 452841, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 522617, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 109507393, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 392285161, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 11, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 6, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 31.103564055, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 11.992682906, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 15.448842862, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 5.260045403, 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:28.381 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 522615, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 118.345889112, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000226449 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 732271, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.137532008, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000187 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-09T09:44:28.382 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 195352, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 304.699339744, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001559745 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 102156, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 22.265756983, 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000217958 2024-05-09T09:44:28.383 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 469117, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 406438, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 406438, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2801.762714261, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006893456 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 62679, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 4, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 1, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 5268, 2024-05-09T09:44:28.384 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 23, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 32, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 1157, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 588, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 559, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 10, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 317, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 1820, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 1, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 1, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 4, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 747586, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 411354, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 62650, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 142, 2024-05-09T09:44:28.385 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 6188, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 6079, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 14, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 1147, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 13, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 167, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 78, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 4, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 234, 2024-05-09T09:44:28.386 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 78, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 45, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 4, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 41, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 33, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 42, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 66593, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 220690, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:28.387 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 153176, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 48097, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 21203, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 254595 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 266, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 84, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 54, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 172, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 1219, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 17, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 59726, 2024-05-09T09:44:28.388 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 59709, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 4, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 15, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 55, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:28.389 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 314692, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 311448, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 311448, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 3244, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 3906, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 431, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 426, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 426, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 5, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 1, 2024-05-09T09:44:28.390 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 6, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 670088862, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 676491541, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 105448, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1241.343241183, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.011772089 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 51, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 60633, 2024-05-09T09:44:28.391 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 60582, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 57, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 507, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 450, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 1164, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 135167, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 134003, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 1, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 65386, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 65385, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 98156, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 231804 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 647399, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 48586, 2024-05-09T09:44:28.392 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 469117, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 8962, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 97051, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 60124, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 416.048220783, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006919836 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 40529, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 436.705379805, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.010775133 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.393 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 136830, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 640.102622558, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.004678086 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.394 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.395 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.396 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 119, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.981473725, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.008247678 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.397 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 10, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.029457330, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002945733 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.398 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 91636, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 878.347445341, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.009585178 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 14666, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 105.685636355, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.007206166 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.399 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 64, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3.408224891, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.053253513 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.400 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 139, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 4.579772239, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.032948001 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.401 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2640, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 6.376792324, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002415451 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 8, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.019521163, 2024-05-09T09:44:28.402 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002440145 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 59673, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 309.478167747, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.005186234 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:28.403 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 3, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 2, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 28271, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 28271, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 1543, 2024-05-09T09:44:28.404 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 360, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 360, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:28.405 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:28.406 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 299224, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 138, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 352, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 4, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:28.407 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 169, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 1075717, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 14957, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:28.408 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 297144, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 729421009, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 37, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 297144, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 297144, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 16986.333208746, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.057165324 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-05-09T09:44:28.409 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 297144, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 402661 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 297144, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 1776, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 295368, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 4921, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 59788, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 1689, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 107132, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 1219, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-05-09T09:44:28.410 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 23, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 120400, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 87, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 90719, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:28.411 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-05-09T09:44:28.412 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 98, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 14, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 12, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 12680, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 10, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 3993 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.413 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 17, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 12, 2024-05-09T09:44:28.414 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 119382, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 59709, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 1722687, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 314016319, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 1722687, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1722687, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 314016319, 2024-05-09T09:44:28.415 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:28.416 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 297144, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 735928934, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 297144, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 735928934, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-05-09T09:44:28.418 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 297144, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 297144, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 297144, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 297144, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.419 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 59709, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 6507993, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 59709, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1684, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 6507993, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.420 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x56492d350500": { 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 314692, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 672297237, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 314692, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 105448, 2024-05-09T09:44:28.421 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 672297237, 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:28.422 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-09T09:44:29.311 INFO:tasks.check_counter:Daemon mds.b mds.exported=14 2024-05-09T09:44:29.311 INFO:tasks.check_counter:Daemon mds.b mds.imported=13 2024-05-09T09:44:29.311 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update=266 2024-05-09T09:44:29.311 INFO:tasks.check_counter:Daemon mds.b mds_cache.dir_update_receipt=84 2024-05-09T09:44:29.311 DEBUG:tasks.check_counter:Getting stats from e 2024-05-09T09:44:29.311 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.e perf dump 2024-05-09T09:44:29.571 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:44:29.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:29 smithi103 ceph-mon[34421]: pgmap v1137: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-05-09T09:44:29.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:29 smithi069 ceph-mon[30123]: pgmap v1137: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-05-09T09:44:29.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:29 smithi163 ceph-mon[34354]: pgmap v1137: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 3.9 KiB/s wr, 9 op/s 2024-05-09T09:44:31.713 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:31 smithi103 ceph-mon[34421]: pgmap v1138: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-05-09T09:44:31.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:31 smithi069 ceph-mon[30123]: pgmap v1138: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-05-09T09:44:31.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:31 smithi163 ceph-mon[34354]: pgmap v1138: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1023 B/s wr, 8 op/s 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1856, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1858, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 214387467, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 590272, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.261415989, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.039556904, 2024-05-09T09:44:31.844 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.161769691, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.047896821, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1856, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.075988142, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000040941 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:31.845 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 1250, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 1591, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 117759034, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1165901, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 6, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.185231217, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.036191975, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.106275880, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.032134726, 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:31.846 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1590, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.058882264, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000037032 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 770304 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 3566, 2024-05-09T09:44:31.847 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 3535, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 184144820, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 1258310, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 5, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.415449666, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.088150516, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.208732165, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.096106708, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3534, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.152232712, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000043076 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.848 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 683088, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 282368 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-05-09T09:44:31.849 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 10456, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.319763590, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000030581 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.850 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 0, 2024-05-09T09:44:31.851 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 57861, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 23910, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 33951, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 23703, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 1, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 0, 2024-05-09T09:44:31.852 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 0, 2024-05-09T09:44:31.853 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:31.854 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 0 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:31.855 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:44:31.856 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 285122, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 0, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 136, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 25, 2024-05-09T09:44:31.857 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 0, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 285385819, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 518096086, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 518095487, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 285122 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 24, 2024-05-09T09:44:31.858 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 57766, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 57742, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 22, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 180, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 158, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 57861, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 58275, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 414, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 0, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 0, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 0, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 104696, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 215420 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.859 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.860 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.861 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-05-09T09:44:31.862 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.863 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.864 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.865 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.866 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:31.867 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.868 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.869 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.870 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.871 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.872 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 1, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 0, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 0, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 0, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 2902, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.873 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:31.874 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:31.875 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:31.876 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 242190, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 95, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 169, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:31.877 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 40819066, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 232927, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 6169, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 21, 2024-05-09T09:44:31.878 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 3, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 6169, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 6169, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 40.362233381, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006542751 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 6169, 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 6170 2024-05-09T09:44:31.879 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 6169, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 6169, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 3229, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 2938, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-05-09T09:44:31.880 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 1, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-05-09T09:44:31.881 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-05-09T09:44:31.882 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 98, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 9, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 11, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 11, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 0, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 2, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 0 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.883 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 0, 2024-05-09T09:44:31.884 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 6668, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 515662721, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 6668, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 6668, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 515662721, 2024-05-09T09:44:31.885 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:31.886 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 6169, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 513904883, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 6169, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 513904883, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:31.887 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:31.888 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:31.889 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:31.889 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:31.889 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 6169, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 6169, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 6169, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 6169, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:31.890 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:31.891 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x55aadc8106e0": { 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:31.892 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:31.893 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:31.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.893 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:31.893 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-09T09:44:32.731 INFO:tasks.check_counter:Daemon mds.e mds.exported=0 2024-05-09T09:44:32.731 INFO:tasks.check_counter:Daemon mds.e mds.imported=0 2024-05-09T09:44:32.731 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update=0 2024-05-09T09:44:32.732 INFO:tasks.check_counter:Daemon mds.e mds_cache.dir_update_receipt=0 2024-05-09T09:44:32.732 DEBUG:tasks.check_counter:Getting stats from h 2024-05-09T09:44:32.732 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.h perf dump 2024-05-09T09:44:32.994 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:44:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:33 smithi069 ceph-mon[30123]: pgmap v1139: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-05-09T09:44:33.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:33 smithi069 ceph-mon[30123]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:33 smithi163 ceph-mon[34354]: pgmap v1139: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-05-09T09:44:33.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:33 smithi163 ceph-mon[34354]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:33.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:33 smithi103 ceph-mon[34421]: pgmap v1139: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 938 B/s wr, 7 op/s 2024-05-09T09:44:33.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:33 smithi103 ceph-mon[34421]: fsmap cephfs:3 {0=b=up:active,1=i=up:active,2=h=up:active} 3 up:standby-replay 6 up:standby 2024-05-09T09:44:34.795 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-09T09:44:34.795 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:34.795 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 490805, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 469780, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 124229780, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 218232106, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 12, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 32.312962320, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 14.525069267, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 18.335298086, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 2.670335234, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 469783, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 34.866035461, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000074217 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.796 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 841823, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.153016058, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000181 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 697520, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 311344 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 268955, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 270302, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 64539432, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 249895727, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 5, 2024-05-09T09:44:34.797 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 18.127096216, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 5.909799054, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 9.356151277, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 4.033217922, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 270302, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 40.461739136, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000149690 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 388940, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.085421457, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000219 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.798 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 683888 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 625836, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 902429, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 189444596, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 868640084, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 18, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 8, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 37.071201575, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 16.658926150, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 19.586934862, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 4.021350982, 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:34.799 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 902422, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 107.228335542, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000118822 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 741135, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.189995099, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000256 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 1, 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-09T09:44:34.800 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-MDSRank": { 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 140762, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 226.653606139, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.001610190 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "queue_len": 0, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "complete_latency": { 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 100036, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 18.137195298, 2024-05-09T09:44:34.801 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000181306 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "mds": { 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "request": 402832, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "reply": 339649, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "reply_latency": { 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 339649, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 1722.886821873, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.005072550 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_reply": 0, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "forward": 63182, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_complete": 0, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_fetch_keys": 0, 2024-05-09T09:44:34.802 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_commit": 139, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_split": 13, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_merge": 13, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes": 1507, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_top": 1029, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_bottom": 455, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pin_tail": 23, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_pinned": 35, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_expired": 399, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "inodes_with_caps": 1, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "caps": 1, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "subtrees": 4, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse": 702451, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_hit": 323106, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_forward": 63180, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_discover": 157, 2024-05-09T09:44:34.803 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "traverse_lock": 8095, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "load_cent": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "q": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "exported": 7, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "exported_inodes": 84, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "imported": 7, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "imported_inodes": 512, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_backtrace_fetch": 20, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rfiles": 637, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rbytes": 211584902, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "root_rsnaps": 0, 2024-05-09T09:44:34.804 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_fetch": 20, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "scrub_file_inodes": 20, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_inode_file_caps": 34, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_revoke": 25149, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_grant": 206752, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps": 102845, 2024-05-09T09:44:34.805 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_caps_dirty": 44476, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_cap_release": 16103, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "process_request_cap_release": 260347 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_cache": { 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update": 408, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_update_receipt": 718, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_try_discover": 424, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_send_discover": 623, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "dir_handle_discover": 571, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays": 11, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_created": 57731, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_enqueued": 57720, 2024-05-09T09:44:34.806 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "strays_migrated": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_started": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "recovery_completed": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_exportdir": 8, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_flush": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragmentdir": 26, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: "ireq_inodestats": 0 2024-05-09T09:44:34.807 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_log": { 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evlrg": 0, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evadd": 285122, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evex": 178173, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evtrm": 172061, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "ev": 113061, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evexg": 0, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "evexd": 6112, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segadd": 341, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segex": 213, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segtrm": 205, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "seg": 136, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segmjr": 10, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segexg": 0, 2024-05-09T09:44:34.808 INFO:teuthology.orchestra.run.smithi103.stdout: "segexd": 8, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "expos": 285385819, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "wrpos": 518096086, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "rdpos": 0, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "jlat": { 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 77066, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 689.462594798, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.008946391 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "replayed": 0 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_mem": { 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "ino": 37, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "ino+": 57964, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "ino-": 57927, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "dir": 43, 2024-05-09T09:44:34.809 INFO:teuthology.orchestra.run.smithi103.stdout: "dir+": 560, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dir-": 517, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dn": 1515, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dn+": 104809, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dn-": 103294, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "cap": 1, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "cap+": 63138, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "cap-": 63137, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "rss": 86420, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "heap": 223612 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_server": { 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_client_request": 616706, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "dispatch_server_request": 22, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_request": 402832, 2024-05-09T09:44:34.810 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_client_session": 7515, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "handle_peer_request": 40300, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "req_create_latency": { 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 57484, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 351.284510797, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006110996 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getattr_latency": { 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 9533, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 161.070622584, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.016896110 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.811 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "req_link_latency": { 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookup_latency": { 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 112855, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 283.990821484, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002516422 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.812 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupino_latency": { 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000118172, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000118172 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupname_latency": { 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.813 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "req_lssnap_latency": { 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mkdir_latency": { 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.814 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mknod_latency": { 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "req_mksnap_latency": { 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "req_open_latency": { 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 2, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000834520, 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000417260 2024-05-09T09:44:34.815 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "req_readdir_latency": { 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 89820, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 595.626701977, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.006631337 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rename_latency": { 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 10876, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 87.909572955, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.008082895 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.816 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmdir_latency": { 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 30, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.412844806, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.013761493 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.817 INFO:teuthology.orchestra.run.smithi103.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setattr_latency": { 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 87, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3.290503252, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.037821876 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.818 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 1260, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 3.021854872, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.002398297 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setlayout_latency": { 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "req_setxattr_latency": { 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.819 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "req_symlink_latency": { 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "req_unlink_latency": { 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 57701, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 236.278436454, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.004094875 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.820 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_sessions": { 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "session_count": 1, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "session_add": 1, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "session_remove": 0, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_open": 1, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "sessions_stale": 0, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "total_load": 27034, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "average_load": 27034, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "avg_session_uptime": 1538, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.821 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:34.822 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:34.823 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 285007, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 127, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 176, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 2, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:34.824 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 169, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 1295337, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 18555, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "objecter": { 2024-05-09T09:44:34.825 INFO:teuthology.orchestra.run.smithi103.stdout: "op_active": 0, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_laggy": 0, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send": 198728, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_send_bytes": 527055279, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_resend": 33, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_reply": 198728, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_latency": { 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 198728, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 2682.641056053, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.013499059 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "op_inflight": 0, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "oplen_avg": { 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 198728, 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 201266 2024-05-09T09:44:34.826 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "op": 198728, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "op_r": 1775, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "op_w": 196953, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "op_rmw": 0, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "op_pg": 0, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_stat": 11, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_create": 31, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_read": 1755, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_write": 78827, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writefull": 872, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_writesame": 0, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_append": 0, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_zero": 0, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_truncate": 13, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_delete": 115728, 2024-05-09T09:44:34.827 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_mapext": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_getxattr": 20, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_setxattr": 57, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_call": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_watch": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_notify": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "osdop_other": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_active": 0, 2024-05-09T09:44:34.828 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_send": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_resend": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "linger_ping": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_active": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_send": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolop_resend": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_active": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_send": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "poolstat_resend": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_active": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_send": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "statfs_resend": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "command_active": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "command_send": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "command_resend": 0, 2024-05-09T09:44:34.829 INFO:teuthology.orchestra.run.smithi103.stdout: "map_epoch": 98, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "map_full": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "map_inc": 9, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_sessions": 12, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_open": 12, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_session_close": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_laggy": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_wr": 2926, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_rd": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_del": 1026 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "oft": { 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_objs": 1, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_updates": 0, 2024-05-09T09:44:34.830 INFO:teuthology.orchestra.run.smithi103.stdout: "omap_total_removes": 0 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "osd-slow-ops": { 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "slow_ops_count": 0 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "purge_queue": { 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_ops_high_water": 17, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing": 0, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executing_high_water": 14, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed_ops": 115421, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_executed": 57720, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:34.831 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 1385592, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 262217974, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 1385592, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1385592, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 262217974, 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:34.832 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:34.833 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 198728, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 533346607, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 198728, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 533346607, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 1024, 2024-05-09T09:44:34.834 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 198728, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 198728, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 198728, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 198728, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.835 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 57720, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 6291328, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 57720, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 1761, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 6291328, 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:34.836 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-write_buf_throttle-0x555923986500": { 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 3758096384, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 285122, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 513901782, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 285122, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:34.837 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 77066, 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 513901782, 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:34.838 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-09T09:44:35.627 INFO:tasks.check_counter:Daemon mds.h mds.exported=7 2024-05-09T09:44:35.627 INFO:tasks.check_counter:Daemon mds.h mds.imported=7 2024-05-09T09:44:35.627 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update=408 2024-05-09T09:44:35.627 INFO:tasks.check_counter:Daemon mds.h mds_cache.dir_update_receipt=718 2024-05-09T09:44:35.627 DEBUG:tasks.check_counter:Getting stats from k 2024-05-09T09:44:35.628 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.k perf dump 2024-05-09T09:44:35.649 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:35 smithi103 ceph-mon[34421]: pgmap v1140: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:35.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:35 smithi069 ceph-mon[30123]: pgmap v1140: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:35.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:35 smithi163 ceph-mon[34354]: pgmap v1140: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:35.885 INFO:teuthology.orchestra.run.smithi103.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.b/config 2024-05-09T09:44:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:44:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:44:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:44:36.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:36 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:44:36.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:44:36.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:44:36.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:44:36.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:36 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:44:36.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-05-09T09:44:36.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-05-09T09:44:36.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-05-09T09:44:36.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:36 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' 2024-05-09T09:44:37.500 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:37 smithi103 ceph-mon[34421]: pgmap v1141: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:37.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:37 smithi069 ceph-mon[30123]: pgmap v1141: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:37.788 INFO:teuthology.orchestra.run.smithi103.stdout:{ 2024-05-09T09:44:37.788 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:37.788 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 458, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 455, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 546784, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 281872, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.056052157, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.022082131, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.020272454, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.010101743, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 454, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.020822862, 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000045865 2024-05-09T09:44:37.789 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 546784, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 281872 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 2, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 348, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 320, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 216928, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:37.790 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.015279466, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.011656412, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.001080849, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 346, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.016149230, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000046674 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.791 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 320, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 216928 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_messages": 4, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages": 4, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_bytes": 3024, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_bytes": 448, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_total_time": 0.000701351, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_send_time": 0.000029019, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_recv_time": 0.000614481, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:37.792 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 3, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000651953, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000217317 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "cct": { 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "total_workers": 0, 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: "unhealthy_workers": 0 2024-05-09T09:44:37.793 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "mempool": { 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:37.794 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:37.795 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_bytes": 148744, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_anon_items": 30, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_bytes": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_items": 0, 2024-05-09T09:44:37.796 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "pgmap_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "mds_co_items": 2, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_1_items": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:37.797 INFO:teuthology.orchestra.run.smithi103.stdout: "unittest_2_items": 0 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: }, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "val": 0, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "max": 104857600, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "get_started": 0, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "get": 460, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "get_sum": 482011, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "get_or_fail_success": 460, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "take": 0, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "take_sum": 0, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "put": 460, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "put_sum": 482011, 2024-05-09T09:44:37.798 INFO:teuthology.orchestra.run.smithi103.stdout: "wait": { 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout: "avgcount": 0, 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout: "sum": 0.000000000, 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout: "avgtime": 0.000000000 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout: } 2024-05-09T09:44:37.799 INFO:teuthology.orchestra.run.smithi103.stdout:} 2024-05-09T09:44:37.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:37 smithi163 ceph-mon[34354]: pgmap v1141: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:38.564 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.k 2024-05-09T09:44:38.564 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.k 2024-05-09T09:44:38.564 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.k 2024-05-09T09:44:38.564 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.k 2024-05-09T09:44:38.564 DEBUG:tasks.check_counter:Getting stats from c 2024-05-09T09:44:38.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.c perf dump 2024-05-09T09:44:38.897 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:44:39.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:39 smithi069 ceph-mon[30123]: pgmap v1142: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:39.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:39 smithi163 ceph-mon[34354]: pgmap v1142: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:39.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:39 smithi103 ceph-mon[34421]: pgmap v1142: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 767 B/s wr, 6 op/s 2024-05-09T09:44:40.777 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-09T09:44:40.988 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 2356, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 2356, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 249350847, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 748394, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.322221999, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.046663864, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.195442252, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.064523469, 2024-05-09T09:44:40.989 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 2355, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.088904399, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000037751 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.990 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 3024, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 448 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 1151, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 1136, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 143120512, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 442925, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 6, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 5, 2024-05-09T09:44:40.991 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.172090513, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.031678930, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.104011894, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.029269335, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1135, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.069973541, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000061650 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:40.992 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 294672, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 171296 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 950, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 1159, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 176403407, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 760208, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 6, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 5, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.176116880, 2024-05-09T09:44:40.993 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.027361513, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.115382245, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.025584166, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1157, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.064819323, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000056023 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:40.994 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 458416 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "cct": { 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "total_workers": 1, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "unhealthy_workers": 0 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-MDSRank": { 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 6956, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.656572070, 2024-05-09T09:44:40.995 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000094389 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": { 2024-05-09T09:44:40.996 INFO:teuthology.orchestra.run.smithi163.stdout: "request": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "reply": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "reply_latency": { 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_reply": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "forward": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_complete": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_keys": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_commit": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_split": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_merge": 0, 2024-05-09T09:44:40.997 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes": 52938, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_top": 36247, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_bottom": 16691, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pin_tail": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pinned": 1156, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_expired": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_with_caps": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "subtrees": 2, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_hit": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_forward": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_discover": 0, 2024-05-09T09:44:40.998 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_lock": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "load_cent": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "q": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "exported": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "exported_inodes": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "imported": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "imported_inodes": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rfiles": 4, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rbytes": 234, 2024-05-09T09:44:40.999 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rsnaps": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_file_inodes": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:41.000 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_cap_release": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "process_request_cap_release": 0 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache": { 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update_receipt": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_try_discover": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_send_discover": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_handle_discover": 0, 2024-05-09T09:44:41.001 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_created": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_enqueued": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_migrated": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_started": 0, 2024-05-09T09:44:41.002 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_completed": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_exportdir": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_flush": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:44:41.003 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:41.004 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_inodestats": 0 2024-05-09T09:44:41.004 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.004 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log": { 2024-05-09T09:44:41.004 INFO:teuthology.orchestra.run.smithi163.stdout: "evlrg": 0, 2024-05-09T09:44:41.004 INFO:teuthology.orchestra.run.smithi163.stdout: "evadd": 0, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "evex": 0, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "evtrm": 0, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "ev": 266076, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "evexg": 0, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "evexd": 0, 2024-05-09T09:44:41.005 INFO:teuthology.orchestra.run.smithi163.stdout: "segadd": 0, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "segex": 0, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "segtrm": 0, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "seg": 5, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "segmjr": 73, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "segexg": 0, 2024-05-09T09:44:41.006 INFO:teuthology.orchestra.run.smithi163.stdout: "segexd": 0, 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "expos": 670088862, 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "wrpos": 676491541, 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "rdpos": 676489044, 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "jlat": { 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.007 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: "replayed": 266076 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_mem": { 2024-05-09T09:44:41.008 INFO:teuthology.orchestra.run.smithi163.stdout: "ino": 34, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "ino+": 53174, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "ino-": 53140, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dir": 40, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dir+": 345, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dir-": 305, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dn": 52938, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dn+": 54330, 2024-05-09T09:44:41.009 INFO:teuthology.orchestra.run.smithi163.stdout: "dn-": 1392, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "cap": 0, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "cap+": 0, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "cap-": 0, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "rss": 99724, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "heap": 215396 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.010 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_server": { 2024-05-09T09:44:41.011 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_client_request": 0, 2024-05-09T09:44:41.011 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_server_request": 0, 2024-05-09T09:44:41.011 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_request": 0, 2024-05-09T09:44:41.011 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_session": 0, 2024-05-09T09:44:41.011 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_peer_request": 0, 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: "req_create_latency": { 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.012 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getattr_latency": { 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:41.013 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "req_link_latency": { 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookup_latency": { 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.014 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupino_latency": { 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.015 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupname_latency": { 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:41.016 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lssnap_latency": { 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.017 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mkdir_latency": { 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mknod_latency": { 2024-05-09T09:44:41.018 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mksnap_latency": { 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "req_open_latency": { 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.019 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "req_readdir_latency": { 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rename_latency": { 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:41.020 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmdir_latency": { 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.021 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.022 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setattr_latency": { 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.023 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setlayout_latency": { 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setxattr_latency": { 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.024 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "req_symlink_latency": { 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "req_unlink_latency": { 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:41.025 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.026 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sessions": { 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "session_count": 1, 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "session_add": 1, 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "session_remove": 0, 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_open": 0, 2024-05-09T09:44:41.027 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_stale": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "total_load": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "average_load": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "avg_session_uptime": 2894, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool": { 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:41.028 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:41.029 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:41.030 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_bytes": 239276, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_items": 93, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:41.031 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_items": 169, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_items": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_bytes": 37400704, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_items": 213929, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_items": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_items": 0 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.032 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter": { 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_active": 0, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_laggy": 0, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send": 4155, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send_bytes": 26, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_resend": 4, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_reply": 4155, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_latency": { 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 4155, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 40.967093126, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.009859709 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "op_inflight": 0, 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "oplen_avg": { 2024-05-09T09:44:41.033 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 4155, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 4156 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "op": 4155, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "op_r": 4155, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "op_w": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "op_rmw": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "op_pg": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_stat": 2113, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_create": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_read": 2039, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_write": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writefull": 0, 2024-05-09T09:44:41.034 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writesame": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_append": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_zero": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_truncate": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_delete": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_mapext": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_getxattr": 2, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_setxattr": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_call": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_watch": 0, 2024-05-09T09:44:41.035 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_notify": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_other": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_active": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_send": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_resend": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_ping": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_active": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_send": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_resend": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_active": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_send": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_resend": 0, 2024-05-09T09:44:41.036 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_active": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_send": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_resend": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "command_active": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "command_send": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "command_resend": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "map_epoch": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "map_full": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "map_inc": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_sessions": 11, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_open": 11, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_close": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_laggy": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_wr": 0, 2024-05-09T09:44:41.037 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_rd": 2, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_del": 0 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "oft": { 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_objs": 0, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_updates": 0, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_removes": 0 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "osd-slow-ops": { 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_ops_count": 0 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "purge_queue": { 2024-05-09T09:44:41.038 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_high_water": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed_ops": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 4453, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 568455734, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:41.039 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 4453, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 4453, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 568455734, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:41.040 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 4155, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 567436510, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 4155, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 567436510, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:41.041 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 1024, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 4155, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 4155, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 4155, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 4155, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.042 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 0, 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:41.043 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle-0x5650102d06e0": { 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 0, 2024-05-09T09:44:41.044 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 0, 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:41.045 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-09T09:44:41.617 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:41 smithi103 ceph-mon[34421]: pgmap v1143: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:41.664 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:41 smithi163 ceph-mon[34354]: pgmap v1143: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:41.700 INFO:tasks.check_counter:Daemon mds.c mds.exported=0 2024-05-09T09:44:41.701 INFO:tasks.check_counter:Daemon mds.c mds.imported=0 2024-05-09T09:44:41.701 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update=0 2024-05-09T09:44:41.701 INFO:tasks.check_counter:Daemon mds.c mds_cache.dir_update_receipt=0 2024-05-09T09:44:41.701 DEBUG:tasks.check_counter:Getting stats from f 2024-05-09T09:44:41.701 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.f perf dump 2024-05-09T09:44:41.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:41 smithi069 ceph-mon[30123]: pgmap v1143: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:41.952 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:44:43.632 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:43 smithi163 ceph-mon[34354]: pgmap v1144: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:43.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:43 smithi069 ceph-mon[30123]: pgmap v1144: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:43.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:43 smithi103 ceph-mon[34421]: pgmap v1144: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 4018, 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 4019, 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 173121046, 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 1277413, 2024-05-09T09:44:43.887 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.397561802, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.091832282, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.173639146, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.107142239, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 4017, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.157990716, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000039330 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.888 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 1222, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 1567, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 134529578, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 1075924, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 7, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 6, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.189058169, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.034817650, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.113724305, 2024-05-09T09:44:43.889 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.030540889, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1565, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.069186540, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000044208 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 687584 2024-05-09T09:44:43.890 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 1510, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 1476, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 143572337, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 604004, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 6, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 4, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.218742801, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.043750802, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.130398248, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.036331068, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1476, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.066442187, 2024-05-09T09:44:43.891 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000045015 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 691024, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 284096 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "cct": { 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "total_workers": 1, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "unhealthy_workers": 0 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.892 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-MDSRank": { 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 10607, 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.288238779, 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000027174 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:43.893 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": { 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "request": 0, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "reply": 0, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "reply_latency": { 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.894 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_reply": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "forward": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_complete": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_keys": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_commit": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_split": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_merge": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes": 45991, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_top": 16323, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_bottom": 29668, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pin_tail": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pinned": 22672, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_expired": 1, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_with_caps": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": 0, 2024-05-09T09:44:43.895 INFO:teuthology.orchestra.run.smithi163.stdout: "subtrees": 1, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_hit": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_forward": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_discover": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_lock": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "load_cent": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "q": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "exported": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "exported_inodes": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "imported": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "imported_inodes": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:43.896 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_backtrace_fetch": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rfiles": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rbytes": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rsnaps": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_fetch": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_file_inodes": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_inode_file_caps": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_revoke": 0, 2024-05-09T09:44:43.897 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_grant": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps_dirty": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_cap_release": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "process_request_cap_release": 0 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache": { 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update_receipt": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_try_discover": 0, 2024-05-09T09:44:43.898 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_send_discover": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_handle_discover": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_created": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_enqueued": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_migrated": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:43.899 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_started": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_completed": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_exportdir": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_flush": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragmentdir": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_inodestats": 0 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log": { 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "evlrg": 0, 2024-05-09T09:44:43.900 INFO:teuthology.orchestra.run.smithi163.stdout: "evadd": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "evex": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "evtrm": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "ev": 228151, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "evexg": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "evexd": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segadd": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segex": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segtrm": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "seg": 128, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segmjr": 20, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segexg": 0, 2024-05-09T09:44:43.901 INFO:teuthology.orchestra.run.smithi163.stdout: "segexd": 0, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "expos": 234978112, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "wrpos": 453003367, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "rdpos": 453001716, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "jlat": { 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "replayed": 228151 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_mem": { 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "ino": 27, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "ino+": 45777, 2024-05-09T09:44:43.902 INFO:teuthology.orchestra.run.smithi163.stdout: "ino-": 45750, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dir": 25, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dir+": 248, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dir-": 223, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dn": 45991, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dn+": 46428, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dn-": 437, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "cap": 0, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "cap+": 0, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "cap-": 0, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "rss": 93684, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "heap": 215420 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_server": { 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_client_request": 0, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_server_request": 0, 2024-05-09T09:44:43.903 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_request": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_session": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_peer_request": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "req_create_latency": { 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getattr_latency": { 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.904 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "req_link_latency": { 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookup_latency": { 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.905 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupino_latency": { 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupname_latency": { 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.906 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lssnap_latency": { 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.907 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mkdir_latency": { 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mknod_latency": { 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.908 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mksnap_latency": { 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "req_open_latency": { 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "req_readdir_latency": { 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.909 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rename_latency": { 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.910 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmdir_latency": { 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.911 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setattr_latency": { 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.912 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setlayout_latency": { 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setxattr_latency": { 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "req_symlink_latency": { 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.913 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "req_unlink_latency": { 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.914 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sessions": { 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "session_count": 1, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "session_add": 2, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "session_remove": 1, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_open": 1, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_stale": 0, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "total_load": 0, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "average_load": 0, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "avg_session_uptime": 2896, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool": { 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:43.915 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:43.916 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_items": 0, 2024-05-09T09:44:43.917 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_bytes": 241517, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_items": 96, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:43.918 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_items": 169, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_items": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_bytes": 32473015, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_items": 185701, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_items": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_items": 0 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter": { 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "op_active": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "op_laggy": 0, 2024-05-09T09:44:43.919 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send_bytes": 21, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_resend": 3, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_reply": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_latency": { 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 33.906248819, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.005430212 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_inflight": 0, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "oplen_avg": { 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 6245 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_r": 6244, 2024-05-09T09:44:43.920 INFO:teuthology.orchestra.run.smithi163.stdout: "op_w": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "op_rmw": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "op_pg": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_stat": 3305, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_create": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_read": 2937, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_write": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writefull": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writesame": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_append": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_zero": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_truncate": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_delete": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_mapext": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:43.921 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_getxattr": 1, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_setxattr": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_call": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_watch": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_notify": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_other": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_active": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_send": 0, 2024-05-09T09:44:43.922 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_resend": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_ping": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_active": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_send": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_resend": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_active": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_send": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_resend": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_active": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_send": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_resend": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "command_active": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "command_send": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "command_resend": 0, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "map_epoch": 98, 2024-05-09T09:44:43.923 INFO:teuthology.orchestra.run.smithi163.stdout: "map_full": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "map_inc": 9, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_sessions": 11, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_open": 12, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_close": 1, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_laggy": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_wr": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_rd": 2, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_del": 0 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "oft": { 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_objs": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_updates": 0, 2024-05-09T09:44:43.924 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_removes": 0 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "osd-slow-ops": { 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_ops_count": 0 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "purge_queue": { 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops_high_water": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_high_water": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed_ops": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed": 0, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:43.925 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 6746, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 450587231, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 6746, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 6746, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 450587231, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.926 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 6244, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 448813027, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 6244, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 448813027, 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:43.927 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 1024, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 6244, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 6244, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 6244, 2024-05-09T09:44:43.928 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 6244, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:43.929 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle-0x55ee1105e6e0": { 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:43.930 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:43.931 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-09T09:44:44.612 INFO:tasks.check_counter:Daemon mds.f mds.exported=0 2024-05-09T09:44:44.612 INFO:tasks.check_counter:Daemon mds.f mds.imported=0 2024-05-09T09:44:44.612 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update=0 2024-05-09T09:44:44.613 INFO:tasks.check_counter:Daemon mds.f mds_cache.dir_update_receipt=0 2024-05-09T09:44:44.613 DEBUG:tasks.check_counter:Getting stats from i 2024-05-09T09:44:44.613 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.i perf dump 2024-05-09T09:44:44.860 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:44:45.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:45 smithi163 ceph-mon[34354]: pgmap v1145: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:45.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:45 smithi069 ceph-mon[30123]: pgmap v1145: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:45.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:45 smithi103 ceph-mon[34421]: pgmap v1145: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:46.680 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-09T09:44:46.680 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:46.680 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 539214, 2024-05-09T09:44:46.680 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 749126, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 156811031, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 736050460, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 24, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 6, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 28.461514803, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 12.902969289, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 16.944748472, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 2.495902582, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 749120, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 69.446152228, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000092703 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:46.681 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 638784, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.152370528, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000238 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 928, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 736 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 456838, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 387407, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 115912762, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 247108528, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 8, 2024-05-09T09:44:46.682 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 5, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 23.961179275, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 9.104920065, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 15.774119332, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 2.403095538, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 387409, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 27.609460304, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000071266 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 729445, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.112628833, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000154 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 160, 2024-05-09T09:44:46.683 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 689056 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 262560, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 261519, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 68164236, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 232168433, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 7, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 6, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 16.233836363, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 6.113922115, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 8.408982332, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 2.771073397, 2024-05-09T09:44:46.684 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 261519, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 24.592644072, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000094037 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 386617, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.060482735, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000156 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 703936, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 312752 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "cct": { 2024-05-09T09:44:46.685 INFO:teuthology.orchestra.run.smithi163.stdout: "total_workers": 1, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "unhealthy_workers": 0 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-MDSRank": { 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 108351, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 178.365907173, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.001646186 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.686 INFO:teuthology.orchestra.run.smithi163.stdout: "finisher-PurgeQueue": { 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "queue_len": 0, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "complete_latency": { 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 82047, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 13.038809150, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000158918 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "mds": { 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "request": 350233, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "reply": 284157, 2024-05-09T09:44:46.687 INFO:teuthology.orchestra.run.smithi163.stdout: "reply_latency": { 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 284157, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 1567.687571516, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.005516976 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_reply": 0, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "forward": 66075, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_complete": 0, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_fetch_keys": 0, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_commit": 196, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_split": 19, 2024-05-09T09:44:46.688 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_merge": 25, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes": 2189, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_top": 1496, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_bottom": 653, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pin_tail": 40, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_pinned": 44, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_expired": 421, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "inodes_with_caps": 1, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "caps": 1, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "subtrees": 4, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse": 571091, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_hit": 269564, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_forward": 66075, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_discover": 238, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_dir_fetch": 0, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_remote_ino": 0, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "traverse_lock": 4994, 2024-05-09T09:44:46.689 INFO:teuthology.orchestra.run.smithi163.stdout: "load_cent": 1000, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "q": 1, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "exported": 7, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "exported_inodes": 98, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "imported": 8, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "imported_inodes": 650, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_dir_fetch": 0, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_backtrace_fetch": 20, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "openino_peer_discover": 0, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rfiles": 637, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rbytes": 211584902, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "root_rsnaps": 0, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_fetch": 20, 2024-05-09T09:44:46.690 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_set_tag": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_backtrace_repaired": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_inotable_repaired": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_inodes": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dir_base_inodes": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_dirfrag_rstats": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "scrub_file_inodes": 20, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_inode_file_caps": 36, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_revoke": 29484, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_grant": 170659, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_trunc": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_cap_op_flush_ack": 0, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps": 103233, 2024-05-09T09:44:46.691 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_caps_dirty": 36905, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_cap_release": 18868, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "process_request_cap_release": 201665 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_cache": { 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update": 177, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_update_receipt": 1187, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_try_discover": 660, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_send_discover": 1211, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "dir_handle_discover": 216, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays": 14, 2024-05-09T09:44:46.692 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_delayed": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "num_strays_enqueuing": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_created": 45736, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_enqueued": 45722, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_reintegrated": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "strays_migrated": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_processing": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_enqueued": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "num_recovering_prioritized": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_started": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "recovery_completed": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_path": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_quiesce_inode": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_enqueue_scrub": 0, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_exportdir": 7, 2024-05-09T09:44:46.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_flush": 0, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragmentdir": 44, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_fragstats": 0, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "ireq_inodestats": 0 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_log": { 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evlrg": 0, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evadd": 228151, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evex": 122901, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evtrm": 122901, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "ev": 105250, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evexg": 0, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "evexd": 12290, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "segadd": 276, 2024-05-09T09:44:46.694 INFO:teuthology.orchestra.run.smithi163.stdout: "segex": 148, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "segtrm": 148, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "seg": 128, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "segmjr": 9, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "segexg": 0, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "segexd": 16, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "expos": 234978112, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "wrpos": 453003367, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "rdpos": 0, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "jlat": { 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 60263, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 632.789679512, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.010500467 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: "replayed": 0 2024-05-09T09:44:46.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_mem": { 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ino": 40, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ino+": 45952, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "ino-": 45912, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dir": 46, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dir+": 909, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dir-": 863, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dn": 2189, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dn+": 90079, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "dn-": 87890, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "cap": 1, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "cap+": 50946, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "cap-": 50945, 2024-05-09T09:44:46.696 INFO:teuthology.orchestra.run.smithi163.stdout: "rss": 83280, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "heap": 223612 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_server": { 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_client_request": 496652, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "dispatch_server_request": 3136, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_request": 350233, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_client_session": 5562, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "handle_peer_request": 17660, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "req_create_latency": { 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 45476, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 348.327533767, 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.007659590 2024-05-09T09:44:46.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getattr_latency": { 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 15462, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 265.827901547, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.017192336 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getfilelock_latency": { 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.698 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "req_link_latency": { 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookup_latency": { 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 95683, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 185.993973209, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.001943855 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookuphash_latency": { 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.699 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupino_latency": { 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000189292, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000189292 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupname_latency": { 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupparent_latency": { 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.700 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lookupsnap_latency": { 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.701 INFO:teuthology.orchestra.run.smithi163.stdout: "req_lssnap_latency": { 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mkdir_latency": { 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mknod_latency": { 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.702 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "req_mksnap_latency": { 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "req_open_latency": { 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.008785290, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.008785290 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "req_readdir_latency": { 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 71879, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 497.927615230, 2024-05-09T09:44:46.703 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.006927303 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rename_latency": { 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 8543, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 75.977177989, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.008893500 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "req_renamesnap_latency": { 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "req_snapdiff_latency": { 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.704 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmdir_latency": { 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 30, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.745134688, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.024837822 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmsnap_latency": { 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "req_rmxattr_latency": { 2024-05-09T09:44:46.705 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setattr_latency": { 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 44, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 1.475608371, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.033536553 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setdirlayout_latency": { 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.706 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setfilelock_latency": { 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 1332, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 2.601442032, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.001953034 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setlayout_latency": { 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "req_setxattr_latency": { 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.707 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "req_symlink_latency": { 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "req_unlink_latency": { 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 45706, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 188.802210101, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.004130797 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_revoke_eviction": 0, 2024-05-09T09:44:46.708 INFO:teuthology.orchestra.run.smithi163.stdout: "cap_acquisition_throttle": 0, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "req_getvxattr_latency": { 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_sessions": { 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "session_count": 1, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "session_add": 2, 2024-05-09T09:44:46.709 INFO:teuthology.orchestra.run.smithi163.stdout: "session_remove": 1, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_open": 1, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "sessions_stale": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "total_load": 27104, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "average_load": 27104, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "avg_session_uptime": 1559, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "mdthresh_evicted": 0 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool": { 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:46.710 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:46.711 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_items": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:46.712 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_bytes": 291196, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_items": 131, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_bytes": 440, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_items": 5, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bytes": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_items": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_bytes": 8800, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_items": 169, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:46.713 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_items": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_bytes": 1817700, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_items": 26118, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_items": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_items": 0 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "objecter": { 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "op_active": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "op_laggy": 0, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send": 156907, 2024-05-09T09:44:46.714 INFO:teuthology.orchestra.run.smithi163.stdout: "op_send_bytes": 460233982, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_resend": 28, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_reply": 156907, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_latency": { 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 156907, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 2687.477539168, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.017127837 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_inflight": 0, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "oplen_avg": { 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 156907, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 159187 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op": 156907, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_r": 1340, 2024-05-09T09:44:46.715 INFO:teuthology.orchestra.run.smithi163.stdout: "op_w": 155567, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "op_rmw": 0, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "op_pg": 0, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_stat": 8, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_create": 37, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_read": 1320, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_write": 61583, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writefull": 688, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_writesame": 0, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_append": 0, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_zero": 0, 2024-05-09T09:44:46.716 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_truncate": 19, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_delete": 91762, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_mapext": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_sparse_read": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_clonerange": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_getxattr": 20, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_setxattr": 66, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_cmpxattr": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_rmxattr": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_resetxattrs": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_call": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_watch": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_notify": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_src_cmpxattr": 0, 2024-05-09T09:44:46.717 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_pgls_filter": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "osdop_other": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_active": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_send": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_resend": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "linger_ping": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_active": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_send": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolop_resend": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_active": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_send": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "poolstat_resend": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_active": 0, 2024-05-09T09:44:46.718 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_send": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "statfs_resend": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "command_active": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "command_send": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "command_resend": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "map_epoch": 98, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "map_full": 0, 2024-05-09T09:44:46.719 INFO:teuthology.orchestra.run.smithi163.stdout: "map_inc": 9, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_sessions": 12, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_open": 12, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_session_close": 0, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_laggy": 0, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_wr": 2877, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_rd": 0, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_del": 807 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "oft": { 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_objs": 1, 2024-05-09T09:44:46.720 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_kv_pairs": 0, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_updates": 0, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "omap_total_removes": 0 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "osd-slow-ops": { 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "slow_ops_count": 0 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "purge_queue": { 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops": 0, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_ops_high_water": 17, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing": 0, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executing_high_water": 12, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed_ops": 91428, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_executed": 45722, 2024-05-09T09:44:46.721 INFO:teuthology.orchestra.run.smithi163.stdout: "pq_item_in_journal": 0 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 1258609, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 235417505, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 1258609, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 1258609, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 235417505, 2024-05-09T09:44:46.722 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_bytes": { 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 156907, 2024-05-09T09:44:46.723 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 465217552, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 156907, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 465217552, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-objecter_ops": { 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 1024, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 0, 2024-05-09T09:44:46.724 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 0, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 0, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 156907, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 156907, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 156907, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 156907, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle": { 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:46.725 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 45722, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 4983570, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 45722, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 1320, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 4983570, 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:46.726 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-write_buf_throttle-0x55d5ee42a500": { 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 3758096384, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 228151, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 448809063, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 228151, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:46.727 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 60263, 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 448809063, 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:46.728 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-09T09:44:47.483 INFO:tasks.check_counter:Daemon mds.i mds.exported=7 2024-05-09T09:44:47.483 INFO:tasks.check_counter:Daemon mds.i mds.imported=8 2024-05-09T09:44:47.483 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update=177 2024-05-09T09:44:47.483 INFO:tasks.check_counter:Daemon mds.i mds_cache.dir_update_receipt=1187 2024-05-09T09:44:47.483 DEBUG:tasks.check_counter:Getting stats from l 2024-05-09T09:44:47.483 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc shell --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 -- ceph daemon mds.l perf dump 2024-05-09T09:44:47.727 INFO:teuthology.orchestra.run.smithi163.stderr:Inferring config /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/mon.c/config 2024-05-09T09:44:47.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:47 smithi069 ceph-mon[30123]: pgmap v1146: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:47.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:47 smithi163 ceph-mon[34354]: pgmap v1146: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:47.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:47 smithi103 ceph-mon[34421]: pgmap v1146: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:49.432 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-0": { 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 0, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 2, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 0, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 0, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 0, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.001161154, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.000000000, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.000338451, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:49.433 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:49.434 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 0, 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 0 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-1": { 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 6, 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 353, 2024-05-09T09:44:49.435 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 3344, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 217904, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 4, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 2, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.017069679, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.012693421, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.001295719, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 352, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.022303796, 2024-05-09T09:44:49.436 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000063363 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.437 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 3344, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 217904 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "AsyncMessenger::Worker-2": { 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_messages": 462, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages": 459, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_bytes": 547872, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_bytes": 284432, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_created_connections": 2, 2024-05-09T09:44:49.438 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_active_connections": 1, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_total_time": 0.058592681, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_send_time": 0.023189321, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_recv_time": 0.021604185, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_running_fast_dispatch_time": 0.010319853, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_messages_queue_lat": { 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 458, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.024508534, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000053512 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_handle_ack_lat": { 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:49.439 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_recv_encrypted_bytes": 547872, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "msgr_send_encrypted_bytes": 284432 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "cct": { 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "total_workers": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "unhealthy_workers": 0 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "mempool": { 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_bytes": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bloom_filter_items": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_bytes": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_alloc_items": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_bytes": 0, 2024-05-09T09:44:49.440 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_data_items": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_bytes": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_onode_items": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_bytes": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_meta_items": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_bytes": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_other_items": 0, 2024-05-09T09:44:49.441 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_bytes": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_cache_buffer_items": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_bytes": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_extent_items": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_bytes": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_blob_items": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_bytes": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_shared_blob_items": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_bytes": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_inline_bl_items": 0, 2024-05-09T09:44:49.442 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_fsck_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_txc_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_deferred_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluestore_writing_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_reader_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_bytes": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "bluefs_file_writer_items": 0, 2024-05-09T09:44:49.443 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_bytes": 148744, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_anon_items": 30, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "buffer_meta_items": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_items": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_mapbl_items": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_pglog_items": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_items": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_bytes": 0, 2024-05-09T09:44:49.444 INFO:teuthology.orchestra.run.smithi163.stdout: "osdmap_mapping_items": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_bytes": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "pgmap_items": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_bytes": 928, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_co_items": 2, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_bytes": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_1_items": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_bytes": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "unittest_2_items": 0 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "val": 0, 2024-05-09T09:44:49.445 INFO:teuthology.orchestra.run.smithi163.stdout: "max": 104857600, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "get_started": 0, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "get": 464, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "get_sum": 482531, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_fail": 0, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "get_or_fail_success": 464, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "take": 0, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "take_sum": 0, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "put": 464, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "put_sum": 482531, 2024-05-09T09:44:49.446 INFO:teuthology.orchestra.run.smithi163.stdout: "wait": { 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout: "avgcount": 0, 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout: "sum": 0.000000000, 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout: "avgtime": 0.000000000 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout: } 2024-05-09T09:44:49.447 INFO:teuthology.orchestra.run.smithi163.stdout:} 2024-05-09T09:44:49.670 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:49 smithi103 ceph-mon[34421]: pgmap v1147: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:49.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:49 smithi069 ceph-mon[30123]: pgmap v1147: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:49.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:49 smithi163 ceph-mon[34354]: pgmap v1147: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-05-09T09:44:50.138 WARNING:tasks.check_counter:Counter 'mds.exported' not found on daemon mds.l 2024-05-09T09:44:50.138 WARNING:tasks.check_counter:Counter 'mds.imported' not found on daemon mds.l 2024-05-09T09:44:50.138 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update' not found on daemon mds.l 2024-05-09T09:44:50.138 WARNING:tasks.check_counter:Counter 'mds_cache.dir_update_receipt' not found on daemon mds.l 2024-05-09T09:44:50.139 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-05-09T09:44:50.150 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T09:44:50.150 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:44:50.150 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:44:50.181 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:44:50.181 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:44:50.241 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 -- 172.21.15.69:0/1823366002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64101960 msgr2=0x7f6e64101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/1823366002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64101960 0x7f6e64101de0 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 crypto rx=0x7f6e4c009900 tx=0x7f6e4c02a410 comp rx=0 tx=0).stop 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 -- 172.21.15.69:0/1823366002 shutdown_connections 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/1823366002 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64102320 0x7f6e64108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/1823366002 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64101960 0x7f6e64101de0 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/1823366002 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 0x7f6e64100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 -- 172.21.15.69:0/1823366002 >> 172.21.15.69:0/1823366002 conn(0x7f6e640fbce0 msgr2=0x7f6e640fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:50.399 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 -- 172.21.15.69:0/1823366002 shutdown_connections 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.394+0000 7f6e6a09e640 1 -- 172.21.15.69:0/1823366002 wait complete. 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 Processor -- start 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 -- start start 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 0x7f6e64197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64101960 0x7f6e641980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6410d510 con 0x7f6e64102320 2024-05-09T09:44:50.400 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6410d390 con 0x7f6e64100770 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e6a09e640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f6e6410d690 con 0x7f6e64101960 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e63fff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55962/0 (socket says 172.21.15.69:55962) 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e63fff640 1 -- 172.21.15.69:0/4052404020 learned_addr learned my addr 172.21.15.69:0/4052404020 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e5bfff640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64101960 0x7f6e641980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.395+0000 7f6e637fe640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 0x7f6e64197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 -- 172.21.15.69:0/4052404020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64101960 msgr2=0x7f6e641980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64101960 0x7f6e641980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 -- 172.21.15.69:0/4052404020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 msgr2=0x7f6e64197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 0x7f6e64197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 -- 172.21.15.69:0/4052404020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f6e6419f790 con 0x7f6e64102320 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e63fff640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009fd0 tx=0x7f6e5c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c017070 con 0x7f6e64102320 2024-05-09T09:44:50.401 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f6e6419fa80 con 0x7f6e64102320 2024-05-09T09:44:50.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f6e5c003e50 con 0x7f6e64102320 2024-05-09T09:44:50.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f6e5c0106c0 con 0x7f6e64102320 2024-05-09T09:44:50.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.396+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f6e6419ffa0 con 0x7f6e64102320 2024-05-09T09:44:50.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.397+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f6e64107b10 con 0x7f6e64102320 2024-05-09T09:44:50.402 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.398+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f6e5c003970 con 0x7f6e64102320 2024-05-09T09:44:50.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.399+0000 7f6e617fa640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 0x7f6e500a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.399+0000 7f6e637fe640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 0x7f6e500a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.399+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f6e5c0cf0d0 con 0x7f6e64102320 2024-05-09T09:44:50.403 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.400+0000 7f6e637fe640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 0x7f6e500a5050 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6e641017c0 tx=0x7f6e5400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:50.404 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.404+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f6e5c09b660 con 0x7f6e64102320 2024-05-09T09:44:50.584 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.584+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f6e64062430 con 0x7f6e64102320 2024-05-09T09:44:50.586 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.586+0000 7f6e617fa640 1 -- 172.21.15.69:0/4052404020 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24708 (secure 0 0 0) 0x7f6e5c09f960 con 0x7f6e64102320 2024-05-09T09:44:50.586 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:50.586 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":98,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:27:27.102222+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:44:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.587+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 msgr2=0x7f6e500a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.587+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 0x7f6e500a5050 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f6e641017c0 tx=0x7f6e5400b390 comp rx=0 tx=0).stop 2024-05-09T09:44:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.587+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 msgr2=0x7f6e6419f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.588 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.587+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c009fd0 tx=0x7f6e5c00b260 comp rx=0 tx=0).stop 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 shutdown_connections 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f6e500a2b90 0x7f6e500a5050 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f6e64102320 0x7f6e6419f120 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f6e64101960 0x7f6e641980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 --2- 172.21.15.69:0/4052404020 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f6e64100770 0x7f6e64197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 >> 172.21.15.69:0/4052404020 conn(0x7f6e640fbce0 msgr2=0x7f6e64104ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 shutdown_connections 2024-05-09T09:44:50.589 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.588+0000 7f6e6a09e640 1 -- 172.21.15.69:0/4052404020 wait complete. 2024-05-09T09:44:50.600 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-05-09T09:11:51.348723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-05-09T09:16:30.601729+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-05-09T09:16:31.484689+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-05-09T09:16:35.125832+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-05-09T09:44:50.600 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- 172.21.15.69:0/2956355290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 msgr2=0x7fe910108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- 172.21.15.69:0/2956355290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe910108bb0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 crypto rx=0x7fe908009ca0 tx=0x7fe90802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- 172.21.15.69:0/2956355290 shutdown_connections 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- 172.21.15.69:0/2956355290 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe910108bb0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- 172.21.15.69:0/2956355290 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910101960 0x7fe910101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- 172.21.15.69:0/2956355290 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910100770 0x7fe910100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.708 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- 172.21.15.69:0/2956355290 >> 172.21.15.69:0/2956355290 conn(0x7fe9100fbce0 msgr2=0x7fe9100fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- 172.21.15.69:0/2956355290 shutdown_connections 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- 172.21.15.69:0/2956355290 wait complete. 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 Processor -- start 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- start start 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 0x7fe9101969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910101960 0x7fe910196f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fe91010c3d0 con 0x7fe910102320 2024-05-09T09:44:50.709 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fe91010c250 con 0x7fe910100770 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe916f29640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fe91010c550 con 0x7fe910101960 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.705+0000 7fe9154c6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:55992/0 (socket says 172.21.15.69:55992) 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 -- 172.21.15.69:0/336349470 learned_addr learned my addr 172.21.15.69:0/336349470 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe914cc5640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 0x7fe9101969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe90ffff640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910101960 0x7fe910196f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 -- 172.21.15.69:0/336349470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910101960 msgr2=0x7fe910196f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910101960 0x7fe910196f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 -- 172.21.15.69:0/336349470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 msgr2=0x7fe9101969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 0x7fe9101969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.710 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 -- 172.21.15.69:0/336349470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fe91019e600 con 0x7fe910102320 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe914cc5640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 0x7fe9101969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe9154c6640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe908005e30 tx=0x7fe908031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe908003c30 con 0x7fe910102320 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fe91019e890 con 0x7fe910102320 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fe908003dd0 con 0x7fe910102320 2024-05-09T09:44:50.711 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fe90802fde0 con 0x7fe910102320 2024-05-09T09:44:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.706+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fe91019ede0 con 0x7fe910102320 2024-05-09T09:44:50.712 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.707+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fe910107a90 con 0x7fe910102320 2024-05-09T09:44:50.713 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.708+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fe908004050 con 0x7fe910102320 2024-05-09T09:44:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.709+0000 7fe90dffb640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 0x7fe9000a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.709+0000 7fe914cc5640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 0x7fe9000a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.709+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fe9080f0a90 con 0x7fe910102320 2024-05-09T09:44:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.710+0000 7fe914cc5640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 0x7fe9000a5050 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe9101017c0 tx=0x7fe90400a5e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:50.714 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.713+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fe9080bd610 con 0x7fe910102320 2024-05-09T09:44:50.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.889+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) -- 0x7fe91002d350 con 0x7fe910102320 2024-05-09T09:44:50.890 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.890+0000 7fe90dffb640 1 -- 172.21.15.69:0/336349470 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v98) ==== 93+0+10 (secure 0 0 0) 0x7fe9080bcf10 con 0x7fe910102320 2024-05-09T09:44:50.890 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 1 2024-05-09T09:44:50.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.891+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 msgr2=0x7fe9000a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.891+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 0x7fe9000a5050 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7fe9101017c0 tx=0x7fe90400a5e0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.891+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 msgr2=0x7fe91019df90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:50.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 secure :-1 s=READY pgs=1145 cs=0 l=1 rev1=1 crypto rx=0x7fe908005e30 tx=0x7fe908031ee0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.892 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 shutdown_connections 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fe9000a2b90 0x7fe9000a5050 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fe910102320 0x7fe91019df90 unknown :-1 s=CLOSED pgs=1145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fe910101960 0x7fe910196f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 --2- 172.21.15.69:0/336349470 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fe910100770 0x7fe9101969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 >> 172.21.15.69:0/336349470 conn(0x7fe9100fbce0 msgr2=0x7fe910104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 shutdown_connections 2024-05-09T09:44:50.893 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:50.892+0000 7fe916f29640 1 -- 172.21.15.69:0/336349470 wait complete. 2024-05-09T09:44:50.904 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-05-09T09:44:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 -- 172.21.15.69:0/1768030087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd00104cc0 msgr2=0x7efd001070b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.012 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 --2- 172.21.15.69:0/1768030087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd00104cc0 0x7efd001070b0 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 crypto rx=0x7efcf0009900 tx=0x7efcf002a410 comp rx=0 tx=0).stop 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 -- 172.21.15.69:0/1768030087 shutdown_connections 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 --2- 172.21.15.69:0/1768030087 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd001075f0 0x7efd001099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 --2- 172.21.15.69:0/1768030087 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd00104cc0 0x7efd001070b0 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 --2- 172.21.15.69:0/1768030087 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00104780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 -- 172.21.15.69:0/1768030087 >> 172.21.15.69:0/1768030087 conn(0x7efd000fbd20 msgr2=0x7efd000fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 -- 172.21.15.69:0/1768030087 shutdown_connections 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.007+0000 7efd07f4f640 1 -- 172.21.15.69:0/1768030087 wait complete. 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 Processor -- start 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 -- start start 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00197bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.013 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd00104cc0 0x7efd00198100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7efd0010d510 con 0x7efd001075f0 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7efd0010d390 con 0x7efd00102390 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd07f4f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7efd0010d690 con 0x7efd00104cc0 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56014/0 (socket says 172.21.15.69:56014) 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 -- 172.21.15.69:0/582680320 learned_addr learned my addr 172.21.15.69:0/582680320 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd05ceb640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00197bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 -- 172.21.15.69:0/582680320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd00104cc0 msgr2=0x7efd00198100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd054ea640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd00104cc0 0x7efd00198100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd00104cc0 0x7efd00198100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 -- 172.21.15.69:0/582680320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 msgr2=0x7efd00197bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00197bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.014 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 -- 172.21.15.69:0/582680320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7efd0019f7e0 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd05ceb640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.008+0000 7efd064ec640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00c970 tx=0x7efcfc00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.009+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc007bb0 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.009+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7efcfc007d50 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.009+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7efd0019fad0 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.009+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7efcfc00f450 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.009+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7efd001a0020 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.010+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7efd000fd810 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.011+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7efcfc004f20 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.012+0000 7efcf6ffd640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 0x7efcec0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.012+0000 7efd05ceb640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 0x7efcec0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.012+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7efcfc0cf060 con 0x7efd001075f0 2024-05-09T09:44:51.015 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.012+0000 7efd05ceb640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 0x7efcec0a5050 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005fd0 tx=0x7efcf8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.016 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.016+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7efcfc014030 con 0x7efd001075f0 2024-05-09T09:44:51.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.192+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) -- 0x7efd000624d0 con 0x7efd001075f0 2024-05-09T09:44:51.192 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.192+0000 7efcf6ffd640 1 -- 172.21.15.69:0/582680320 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v98) ==== 104+0+11 (secure 0 0 0) 0x7efcfc09bc40 con 0x7efd001075f0 2024-05-09T09:44:51.193 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 msgr2=0x7efcec0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 0x7efcec0a5050 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7efcf8005fd0 tx=0x7efcf8009500 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 msgr2=0x7efd0019f170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 secure :-1 s=READY pgs=1147 cs=0 l=1 rev1=1 crypto rx=0x7efcfc00c970 tx=0x7efcfc00ce30 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 shutdown_connections 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7efcec0a2b90 0x7efcec0a5050 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7efd001075f0 0x7efd0019f170 unknown :-1 s=CLOSED pgs=1147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7efd00104cc0 0x7efd00198100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 --2- 172.21.15.69:0/582680320 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7efd00102390 0x7efd00197bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 >> 172.21.15.69:0/582680320 conn(0x7efd000fbd20 msgr2=0x7efd00102c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.195 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 shutdown_connections 2024-05-09T09:44:51.196 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.194+0000 7efd07f4f640 1 -- 172.21.15.69:0/582680320 wait complete. 2024-05-09T09:44:51.206 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-05-09T09:44:51.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/3849106075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58101880 msgr2=0x7f3c58101ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/3849106075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58101880 0x7f3c58101ca0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 crypto rx=0x7f3c54009900 tx=0x7f3c5402a410 comp rx=0 tx=0).stop 2024-05-09T09:44:51.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/3849106075 shutdown_connections 2024-05-09T09:44:51.313 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/3849106075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58103430 0x7f3c58109cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/3849106075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58102a70 0x7f3c58102ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/3849106075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58101880 0x7f3c58101ca0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/3849106075 >> 172.21.15.69:0/3849106075 conn(0x7f3c580fce10 msgr2=0x7f3c580ff230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/3849106075 shutdown_connections 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.309+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/3849106075 wait complete. 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 Processor -- start 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 -- start start 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58101880 0x7f3c58197c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.314 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58103430 0x7f3c5819f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5810d550 con 0x7f3c58102a70 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5810d3d0 con 0x7f3c58103430 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ecc3640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f3c5810d6d0 con 0x7f3c58101880 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56026/0 (socket says 172.21.15.69:56026) 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 -- 172.21.15.69:0/1399249205 learned_addr learned my addr 172.21.15.69:0/1399249205 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5d260640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58103430 0x7f3c5819f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c5ca5f640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58101880 0x7f3c58197c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.315 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 -- 172.21.15.69:0/1399249205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58101880 msgr2=0x7f3c58197c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58101880 0x7f3c58197c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 -- 172.21.15.69:0/1399249205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58103430 msgr2=0x7f3c5819f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58103430 0x7f3c5819f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 -- 172.21.15.69:0/1399249205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f3c5819f860 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.310+0000 7f3c4ffff640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d950 tx=0x7f3c4800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.311+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c48017070 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.311+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f3c48009c30 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.311+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f3c5819fb50 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.311+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f3c4800f6b0 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.311+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f3c581a00a0 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.312+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f3c58108c20 con 0x7f3c58102a70 2024-05-09T09:44:51.316 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.313+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f3c480109d0 con 0x7f3c58102a70 2024-05-09T09:44:51.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.313+0000 7f3c4dffb640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 0x7f3c400a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.313+0000 7f3c5ca5f640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 0x7f3c400a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.313+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f3c480cf0a0 con 0x7f3c58102a70 2024-05-09T09:44:51.317 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.314+0000 7f3c5ca5f640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 0x7f3c400a5050 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3c540314e0 tx=0x7f3c54000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.318 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.318+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f3c48009de0 con 0x7f3c58102a70 2024-05-09T09:44:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.495+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) -- 0x7f3c58062430 con 0x7f3c58102a70 2024-05-09T09:44:51.495 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.495+0000 7f3c4dffb640 1 -- 172.21.15.69:0/1399249205 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v98) ==== 100+0+11 (secure 0 0 0) 0x7f3c4809bc80 con 0x7f3c58102a70 2024-05-09T09:44:51.496 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 64 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 msgr2=0x7f3c400a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 0x7f3c400a5050 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3c540314e0 tx=0x7f3c54000bc0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 msgr2=0x7f3c58198190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 crypto rx=0x7f3c4800d950 tx=0x7f3c4800de10 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 shutdown_connections 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f3c400a2b90 0x7f3c400a5050 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f3c58103430 0x7f3c5819f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f3c58102a70 0x7f3c58198190 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 --2- 172.21.15.69:0/1399249205 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f3c58101880 0x7f3c58197c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.498 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 >> 172.21.15.69:0/1399249205 conn(0x7f3c580fce10 msgr2=0x7f3c58105df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 shutdown_connections 2024-05-09T09:44:51.499 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.497+0000 7f3c5ecc3640 1 -- 172.21.15.69:0/1399249205 wait complete. 2024-05-09T09:44:51.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:51 smithi069 ceph-mon[30123]: pgmap v1148: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:51.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/4052404020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:51.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/336349470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.508 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:51 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/582680320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.509 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/2742430336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 msgr2=0x7fd8c4109a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/2742430336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c4109a00 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc009ca0 tx=0x7fd8bc02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/2742430336 shutdown_connections 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/2742430336 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c4109a00 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/2742430336 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c4104ce0 0x7fd8c41070d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/2742430336 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c41023b0 0x7fd8c41047a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/2742430336 >> 172.21.15.69:0/2742430336 conn(0x7fd8c40fbd00 msgr2=0x7fd8c40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/2742430336 shutdown_connections 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/2742430336 wait complete. 2024-05-09T09:44:51.616 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.613+0000 7fd8cc4bd640 1 Processor -- start 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 -- start start 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c41023b0 0x7fd8c4197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c4104ce0 0x7fd8c4198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410d510 con 0x7fd8c4107610 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410d390 con 0x7fd8c4104ce0 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd8c410d690 con 0x7fd8c41023b0 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56050/0 (socket says 172.21.15.69:56050) 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 -- 172.21.15.69:0/733314075 learned_addr learned my addr 172.21.15.69:0/733314075 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:51.617 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8c9a58640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c4104ce0 0x7fd8c4198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8ca259640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c41023b0 0x7fd8c4197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 -- 172.21.15.69:0/733314075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c41023b0 msgr2=0x7fd8c4197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c41023b0 0x7fd8c4197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 -- 172.21.15.69:0/733314075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c4104ce0 msgr2=0x7fd8c4198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c4104ce0 0x7fd8c4198090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 -- 172.21.15.69:0/733314075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd8c419f770 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8caa5a640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc031ae0 tx=0x7fd8bc031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc003c30 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd8c419fa00 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd8bc003dd0 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.614+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd8bc02fcb0 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.615+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd8c419ffa0 con 0x7fd8c4107610 2024-05-09T09:44:51.618 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.615+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd8c40fd7f0 con 0x7fd8c4107610 2024-05-09T09:44:51.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.617+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd8bc004050 con 0x7fd8c4107610 2024-05-09T09:44:51.619 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.617+0000 7fd8bb7fe640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 0x7fd8b00a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.617+0000 7fd8ca259640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 0x7fd8b00a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.617+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fd8bc0f0610 con 0x7fd8c4107610 2024-05-09T09:44:51.620 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.618+0000 7fd8ca259640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 0x7fd8b00a5050 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009bd0 tx=0x7fd8b4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.622 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.621+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd8bc0bd310 con 0x7fd8c4107610 2024-05-09T09:44:51.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.797+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) -- 0x7fd8c4062430 con 0x7fd8c4107610 2024-05-09T09:44:51.797 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.797+0000 7fd8bb7fe640 1 -- 172.21.15.69:0/733314075 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v98) ==== 103+0+11 (secure 0 0 0) 0x7fd8bc0bcc10 con 0x7fd8c4107610 2024-05-09T09:44:51.798 INFO:teuthology.orchestra.run.smithi069.stdout:pg_num: 32 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 msgr2=0x7fd8b00a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 0x7fd8b00a5050 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7fd8b4009bd0 tx=0x7fd8b4005f90 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 msgr2=0x7fd8c419f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 secure :-1 s=READY pgs=1151 cs=0 l=1 rev1=1 crypto rx=0x7fd8bc031ae0 tx=0x7fd8bc031ee0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 shutdown_connections 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd8b00a2b90 0x7fd8b00a5050 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd8c4107610 0x7fd8c419f100 unknown :-1 s=CLOSED pgs=1151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd8c4104ce0 0x7fd8c4198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 --2- 172.21.15.69:0/733314075 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd8c41023b0 0x7fd8c4197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 >> 172.21.15.69:0/733314075 conn(0x7fd8c40fbd00 msgr2=0x7fd8c4102cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 shutdown_connections 2024-05-09T09:44:51.800 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.799+0000 7fd8cc4bd640 1 -- 172.21.15.69:0/733314075 wait complete. 2024-05-09T09:44:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:51 smithi163 ceph-mon[34354]: pgmap v1148: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/4052404020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:51.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/336349470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:51 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/582680320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.811 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-05-09T09:44:51.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:51 smithi103 ceph-mon[34421]: pgmap v1148: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 6 op/s 2024-05-09T09:44:51.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/4052404020' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:51.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/336349470' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:51 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/582680320' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 -- 172.21.15.69:0/1160966702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684101410 msgr2=0x7f5684108cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 --2- 172.21.15.69:0/1160966702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684101410 0x7f5684108cb0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 crypto rx=0x7f567c009ca0 tx=0x7f567c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 -- 172.21.15.69:0/1160966702 shutdown_connections 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 --2- 172.21.15.69:0/1160966702 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684101410 0x7f5684108cb0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 --2- 172.21.15.69:0/1160966702 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684100a50 0x7f5684100ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.920 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 --2- 172.21.15.69:0/1160966702 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101d20 0x7f5684102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 -- 172.21.15.69:0/1160966702 >> 172.21.15.69:0/1160966702 conn(0x7f56840fb460 msgr2=0x7f56840fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 -- 172.21.15.69:0/1160966702 shutdown_connections 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.914+0000 7f56891a9640 1 -- 172.21.15.69:0/1160966702 wait complete. 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 Processor -- start 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 -- start start 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101410 0x7f5684198090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 0x7f568419f100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.921 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f568410d500 con 0x7f5684100a50 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f568410d380 con 0x7f5684101d20 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f56891a9640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f568410d680 con 0x7f5684101410 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56080/0 (socket says 172.21.15.69:56080) 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 -- 172.21.15.69:0/834905557 learned_addr learned my addr 172.21.15.69:0/834905557 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:51.922 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f568359e640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 0x7f568419f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f567a59c640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101410 0x7f5684198090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 -- 172.21.15.69:0/834905557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101410 msgr2=0x7f5684198090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101410 0x7f5684198090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 -- 172.21.15.69:0/834905557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 msgr2=0x7f568419f100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 0x7f568419f100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f5682d9d640 1 -- 172.21.15.69:0/834905557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f568419f770 con 0x7f5684100a50 2024-05-09T09:44:51.923 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.915+0000 7f568359e640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 0x7f568419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f5682d9d640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f567400d950 tx=0x7f567400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5674017020 con 0x7f5684100a50 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f568419fa60 con 0x7f5684100a50 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5674009c30 con 0x7f5684100a50 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f567400f740 con 0x7f5684100a50 2024-05-09T09:44:51.924 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.916+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f568419ffb0 con 0x7f5684100a50 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.917+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5684107b90 con 0x7f5684100a50 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.918+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f567401a020 con 0x7f5684100a50 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.919+0000 7f5680d99640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 0x7f56700a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.919+0000 7f567a59c640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 0x7f56700a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.919+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f56740ced80 con 0x7f5684100a50 2024-05-09T09:44:51.925 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.919+0000 7f567a59c640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 0x7f56700a5050 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f566c005fd0 tx=0x7f566c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:51.926 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:51.923+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5674021070 con 0x7f5684100a50 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.109+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) -- 0x7f568402d1c0 con 0x7f5684100a50 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.110+0000 7f5680d99640 1 -- 172.21.15.69:0/834905557 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v98) ==== 81+0+590 (secure 0 0 0) 0x7f5674009de0 con 0x7f5684100a50 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/4182744829 2024-05-10T09:09:29.020087+0000 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1673751021 2024-05-10T09:09:29.020087+0000 2024-05-09T09:44:52.111 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1911852009 2024-05-10T09:09:02.961220+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/1056628795 2024-05-10T09:09:02.961220+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/936285351 2024-05-10T09:09:29.020087+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6800/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/2520675955 2024-05-10T09:09:02.961220+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:6801/2869747387 2024-05-10T09:09:02.961220+0000 2024-05-09T09:44:52.112 INFO:teuthology.orchestra.run.smithi069.stdout:172.21.15.69:0/3187678781 2024-05-10T09:09:29.020087+0000 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:listed 10 entries 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 msgr2=0x7f56700a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 0x7f56700a5050 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f566c005fd0 tx=0x7f566c005850 comp rx=0 tx=0).stop 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 msgr2=0x7f5684197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 crypto rx=0x7f567400d950 tx=0x7f567400de10 comp rx=0 tx=0).stop 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 shutdown_connections 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f56700a2b90 0x7f56700a5050 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.113 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5684101d20 0x7f568419f100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5684101410 0x7f5684198090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 --2- 172.21.15.69:0/834905557 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5684100a50 0x7f5684197b50 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 >> 172.21.15.69:0/834905557 conn(0x7f56840fb460 msgr2=0x7f5684104d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:52.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 shutdown_connections 2024-05-09T09:44:52.114 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.112+0000 7f56891a9640 1 -- 172.21.15.69:0/834905557 wait complete. 2024-05-09T09:44:52.124 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-05-09T09:44:52.124 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.124 DEBUG:teuthology.orchestra.run.smithi069:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-05-09T09:44:52.316 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T09:44:52.316 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.316 DEBUG:teuthology.orchestra.run.smithi069:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-05-09T09:44:52.369 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.369 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-05-09T09:44:52.409 INFO:teuthology.orchestra.run.smithi069.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-05-09T09:44:52.410 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.410 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-05-09T09:44:52.515 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.515 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-05-09T09:44:52.600 INFO:tasks.kclient:Unmounting kernel clients... 2024-05-09T09:44:52.601 DEBUG:teuthology.orchestra.run.smithi069:> set -ex 2024-05-09T09:44:52.601 DEBUG:teuthology.orchestra.run.smithi069:> dd if=/proc/self/mounts of=/dev/stdout 2024-05-09T09:44:52.619 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4056292,mode=755,inode64 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev,inode64 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6497604k,nr_inodes=819200,mode=755,inode64 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=16859 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 none /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 none /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248800k,nr_inodes=812200,mode=700,uid=1000,gid=1316,inode64 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/95246700a94798b7c758d12dcade34b75241a1ef7d9896bfb7e1836770986a92/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/95246700a94798b7c758d12dcade34b75241a1ef7d9896bfb7e1836770986a92/diff,workdir=/var/lib/containers/storage/overlay/95246700a94798b7c758d12dcade34b75241a1ef7d9896bfb7e1836770986a92/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2ba8168c32eeefe00cb48e2bc27daf254bfa9a8ee16a738a49f5fd0e291e94de/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/2ba8168c32eeefe00cb48e2bc27daf254bfa9a8ee16a738a49f5fd0e291e94de/diff,workdir=/var/lib/containers/storage/overlay/2ba8168c32eeefe00cb48e2bc27daf254bfa9a8ee16a738a49f5fd0e291e94de/work,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3248800k,nr_inodes=812200,mode=700,inode64 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/f0e7162e2a3b41de44d9b3bfe8b1b74c3b94ca79d431296f32e3e52ecbc86c8b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/f0e7162e2a3b41de44d9b3bfe8b1b74c3b94ca79d431296f32e3e52ecbc86c8b/diff,workdir=/var/lib/containers/storage/overlay/f0e7162e2a3b41de44d9b3bfe8b1b74c3b94ca79d431296f32e3e52ecbc86c8b/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9a7c37d14ed3c946936e73d2223c6c2a36e292e6ffdfd6b87cf7d74e39f7f9f1/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/9a7c37d14ed3c946936e73d2223c6c2a36e292e6ffdfd6b87cf7d74e39f7f9f1/diff,workdir=/var/lib/containers/storage/overlay/9a7c37d14ed3c946936e73d2223c6c2a36e292e6ffdfd6b87cf7d74e39f7f9f1/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/c8066468fb1fab120f9c9eaa9ce4dcff5eb90ec10b25e7a8970e50afa590c08e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/c8066468fb1fab120f9c9eaa9ce4dcff5eb90ec10b25e7a8970e50afa590c08e/diff,workdir=/var/lib/containers/storage/overlay/c8066468fb1fab120f9c9eaa9ce4dcff5eb90ec10b25e7a8970e50afa590c08e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/036e1a77f9c28f2f74b51d9f1e444a05cc3325070a4a1bae23921c987b8fd56e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/036e1a77f9c28f2f74b51d9f1e444a05cc3325070a4a1bae23921c987b8fd56e/diff,workdir=/var/lib/containers/storage/overlay/036e1a77f9c28f2f74b51d9f1e444a05cc3325070a4a1bae23921c987b8fd56e/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/f80d553c591563a29b8fd7be7c81583977bdfdb1bd0c1e2780f1fea602796d61/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/f80d553c591563a29b8fd7be7c81583977bdfdb1bd0c1e2780f1fea602796d61/diff,workdir=/var/lib/containers/storage/overlay/f80d553c591563a29b8fd7be7c81583977bdfdb1bd0c1e2780f1fea602796d61/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/9676bee77d889ac6e34921bc20cb06cc64e49c4116b7bbd8aaaa56f57f8becae/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/9676bee77d889ac6e34921bc20cb06cc64e49c4116b7bbd8aaaa56f57f8becae/diff,workdir=/var/lib/containers/storage/overlay/9676bee77d889ac6e34921bc20cb06cc64e49c4116b7bbd8aaaa56f57f8becae/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/dae6028816059875df24493cf459756dfab5fc489c85177aed4e32d33da90380/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/dae6028816059875df24493cf459756dfab5fc489c85177aed4e32d33da90380/diff,workdir=/var/lib/containers/storage/overlay/dae6028816059875df24493cf459756dfab5fc489c85177aed4e32d33da90380/work,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/d9fabc11817efd8555067d8dee95f49a3d2db50104d79803dc73f8f8e8e1b893/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/4PAVAGAMDDUPSBJRTB3BGKHG7P:/var/lib/containers/storage/overlay/l/TVDLRN5W2GUPVLIEUEHWDM2475,upperdir=/var/lib/containers/storage/overlay/d9fabc11817efd8555067d8dee95f49a3d2db50104d79803dc73f8f8e8e1b893/diff,workdir=/var/lib/containers/storage/overlay/d9fabc11817efd8555067d8dee95f49a3d2db50104d79803dc73f8f8e8e1b893/work,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6497604k,nr_inodes=819200,mode=755,inode64 0 0 2024-05-09T09:44:52.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.620 DEBUG:teuthology.orchestra.run.smithi069:> ip netns list 2024-05-09T09:44:52.675 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-05-09T09:44:52.675 DEBUG:teuthology.orchestra.run.smithi069:> sudo ip link delete ceph-brx 2024-05-09T09:44:52.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:52 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1399249205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:52 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/733314075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.696 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:52 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/834905557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:44:52.754 INFO:teuthology.orchestra.run.smithi069.stderr:Cannot find device "ceph-brx" 2024-05-09T09:44:52.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T09:44:52.756 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-05-09T09:44:52.771 INFO:tasks.cephadm:Teardown begin 2024-05-09T09:44:52.771 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-05-09T09:44:52.771 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:44:52.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:52 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1399249205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:52 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/733314075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:52 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/834905557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:44:52.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:52 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1399249205' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:52 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/733314075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-05-09T09:44:52.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:52 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/834905557' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 -- 172.21.15.69:0/3528218138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0102a90 msgr2=0x7f2eb0102f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 --2- 172.21.15.69:0/3528218138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0102a90 0x7f2eb0102f10 secure :-1 s=READY pgs=1154 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009900 tx=0x7f2ea002a410 comp rx=0 tx=0).stop 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 -- 172.21.15.69:0/3528218138 shutdown_connections 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 --2- 172.21.15.69:0/3528218138 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb0109ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 --2- 172.21.15.69:0/3528218138 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb0102a90 0x7f2eb0102f10 unknown :-1 s=CLOSED pgs=1154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 --2- 172.21.15.69:0/3528218138 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb01018a0 0x7f2eb0101cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 -- 172.21.15.69:0/3528218138 >> 172.21.15.69:0/3528218138 conn(0x7f2eb00fce10 msgr2=0x7f2eb00ff250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 -- 172.21.15.69:0/3528218138 shutdown_connections 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.873+0000 7f2eb805d640 1 -- 172.21.15.69:0/3528218138 wait complete. 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 Processor -- start 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 -- start start 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 0x7f2eb01980b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:52.876 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb019f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb010d500 con 0x7f2eb01018a0 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb010d380 con 0x7f2eb0103450 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb805d640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f2eb010d680 con 0x7f2eb0102a90 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56096/0 (socket says 172.21.15.69:56096) 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 -- 172.21.15.69:0/1392885706 learned_addr learned my addr 172.21.15.69:0/1392885706 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb55f8640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 0x7f2eb01980b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb65fa640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb019f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 -- 172.21.15.69:0/1392885706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 msgr2=0x7f2eb01980b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 0x7f2eb01980b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 -- 172.21.15.69:0/1392885706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 msgr2=0x7f2eb019f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb019f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 -- 172.21.15.69:0/1392885706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f2eb019f790 con 0x7f2eb01018a0 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb55f8640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 0x7f2eb01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:52.877 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb65fa640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.874+0000 7f2eb5df9640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00d950 tx=0x7f2eac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac017020 con 0x7f2eb01018a0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f2eac009c30 con 0x7f2eb01018a0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f2eb019fa80 con 0x7f2eb01018a0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f2eac00f740 con 0x7f2eb01018a0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f2eb019ffa0 con 0x7f2eb01018a0 2024-05-09T09:44:52.878 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.875+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f2eb0068650 con 0x7f2eb01018a0 2024-05-09T09:44:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.877+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f2eac01a020 con 0x7f2eb01018a0 2024-05-09T09:44:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.877+0000 7f2ea6ffd640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 0x7f2e9c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.877+0000 7f2eb55f8640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 0x7f2e9c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.877+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f2eac0cf1d0 con 0x7f2eb01018a0 2024-05-09T09:44:52.879 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.878+0000 7f2eb55f8640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 0x7f2e9c0a5050 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0000c00 tx=0x7f2ea00095f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:52.881 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:52.881+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f2eac021070 con 0x7f2eb01018a0 2024-05-09T09:44:53.095 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.095+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7f2eb0108c10 con 0x7f2eb01018a0 2024-05-09T09:44:53.096 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.096+0000 7f2ea6ffd640 1 -- 172.21.15.69:0/1392885706 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11883 (secure 0 0 0) 0x7f2eac009de0 con 0x7f2eb01018a0 2024-05-09T09:44:53.096 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:53.096 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:44:32.187773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":66,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":66,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25975":{"gid":25975,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3681372184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3681372184},{"type":"v1","addr":"172.21.15.163:6837","nonce":3681372184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24511,24527,24491]},"id":1}]} 2024-05-09T09:44:53.097 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 53 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 msgr2=0x7f2e9c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 0x7f2e9c0a5050 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0000c00 tx=0x7f2ea00095f0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 msgr2=0x7f2eb0197b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 secure :-1 s=READY pgs=1155 cs=0 l=1 rev1=1 crypto rx=0x7f2eac00d950 tx=0x7f2eac00de10 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 shutdown_connections 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f2e9c0a2b90 0x7f2e9c0a5050 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f2eb0103450 0x7f2eb019f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f2eb0102a90 0x7f2eb01980b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 --2- 172.21.15.69:0/1392885706 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f2eb01018a0 0x7f2eb0197b70 unknown :-1 s=CLOSED pgs=1155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 >> 172.21.15.69:0/1392885706 conn(0x7f2eb00fce10 msgr2=0x7f2eb0105de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 shutdown_connections 2024-05-09T09:44:53.099 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.098+0000 7f2eb805d640 1 -- 172.21.15.69:0/1392885706 wait complete. 2024-05-09T09:44:53.110 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-05-09T09:44:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.211+0000 7fb648a91640 1 -- 172.21.15.69:0/188659800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 msgr2=0x7fb644108bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.211+0000 7fb648a91640 1 --2- 172.21.15.69:0/188659800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644108bb0 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 crypto rx=0x7fb63c009ca0 tx=0x7fb63c02a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- 172.21.15.69:0/188659800 shutdown_connections 2024-05-09T09:44:53.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- 172.21.15.69:0/188659800 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644108bb0 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- 172.21.15.69:0/188659800 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644101960 0x7fb644101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- 172.21.15.69:0/188659800 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb644100770 0x7fb644100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- 172.21.15.69:0/188659800 >> 172.21.15.69:0/188659800 conn(0x7fb6440fbce0 msgr2=0x7fb6440fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- 172.21.15.69:0/188659800 shutdown_connections 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- 172.21.15.69:0/188659800 wait complete. 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 Processor -- start 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- start start 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644100770 0x7fb644104700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb644101960 0x7fb644102d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fb64410c450 con 0x7fb644102320 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fb64410c2d0 con 0x7fb644101960 2024-05-09T09:44:53.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.212+0000 7fb648a91640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fb64410c5d0 con 0x7fb644100770 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56124/0 (socket says 172.21.15.69:56124) 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 -- 172.21.15.69:0/3120379403 learned_addr learned my addr 172.21.15.69:0/3120379403 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb64259c640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644100770 0x7fb644104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 -- 172.21.15.69:0/3120379403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644100770 msgr2=0x7fb644104700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644100770 0x7fb644104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 -- 172.21.15.69:0/3120379403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb644101960 msgr2=0x7fb644102d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb644101960 0x7fb644102d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 -- 172.21.15.69:0/3120379403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fb644103b10 con 0x7fb644102320 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb642d9d640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005e30 tx=0x7fb63c031ee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c003c30 con 0x7fb644102320 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fb63c003dd0 con 0x7fb644102320 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fb6441a1cb0 con 0x7fb644102320 2024-05-09T09:44:53.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fb63c02fde0 con 0x7fb644102320 2024-05-09T09:44:53.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.213+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fb6441a21b0 con 0x7fb644102320 2024-05-09T09:44:53.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.214+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fb6441a23e0 con 0x7fb644102320 2024-05-09T09:44:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.215+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fb63c004050 con 0x7fb644102320 2024-05-09T09:44:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.216+0000 7fb63b7fe640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 0x7fb6300a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.216+0000 7fb64259c640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 0x7fb6300a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.216+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7fb63c0f0980 con 0x7fb644102320 2024-05-09T09:44:53.218 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.217+0000 7fb64259c640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 0x7fb6300a5050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb634005fd0 tx=0x7fb634005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.221 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.221+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fb63c0bd680 con 0x7fb644102320 2024-05-09T09:44:53.435 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.435+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) -- 0x7fb6441a2770 con 0x7fb644102320 2024-05-09T09:44:53.436 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.436+0000 7fb63b7fe640 1 -- 172.21.15.69:0/3120379403 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) ==== 94+0+11883 (secure 0 0 0) 0x7fb63c0bcf80 con 0x7fb644102320 2024-05-09T09:44:53.436 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:53.436 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":53,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14541,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6835/3886932870","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6834","nonce":3886932870},{"type":"v1","addr":"172.21.15.69:6835","nonce":3886932870}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":4},{"gid":14553,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6837/4268769273","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6836","nonce":4268769273},{"type":"v1","addr":"172.21.15.69:6837","nonce":4268769273}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":5},{"gid":14616,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6839/1794172552","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6838","nonce":1794172552},{"type":"v1","addr":"172.21.15.69:6839","nonce":1794172552}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":11},{"gid":14628,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.69:6841/4006889644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6840","nonce":4006889644},{"type":"v1","addr":"172.21.15.69:6841","nonce":4006889644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":12},{"gid":24430,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.163:6833/3615156387","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6832","nonce":3615156387},{"type":"v1","addr":"172.21.15.163:6833","nonce":3615156387}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":2},{"gid":24431,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.103:6833/1248493104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6832","nonce":1248493104},{"type":"v1","addr":"172.21.15.103:6833","nonce":1248493104}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"epoch":3}],"filesystems":[{"mdsmap":{"epoch":53,"flags":306,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-05-09T09:16:32.211035+0000","modified":"2024-05-09T09:44:32.187773+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}},"max_mds":3,"in":[0,1,2],"up":{"mds_0":24527,"mds_1":24511,"mds_2":24491},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24458":{"gid":24458,"name":"e","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.103:6835/3790151668","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6834","nonce":3790151668},{"type":"v1","addr":"172.21.15.103:6835","nonce":3790151668}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24478":{"gid":24478,"name":"f","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6835/989258568","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6834","nonce":989258568},{"type":"v1","addr":"172.21.15.163:6835","nonce":989258568}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24491":{"gid":24491,"name":"h","rank":2,"incarnation":22,"state":"up:active","state_seq":66,"addr":"172.21.15.103:6837/1084447179","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6836","nonce":1084447179},{"type":"v1","addr":"172.21.15.103:6837","nonce":1084447179}]},"join_fscid":-1,"export_targets":[0,1],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24511":{"gid":24511,"name":"i","rank":1,"incarnation":20,"state":"up:active","state_seq":66,"addr":"172.21.15.163:6839/282166335","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6838","nonce":282166335},{"type":"v1","addr":"172.21.15.163:6839","nonce":282166335}]},"join_fscid":-1,"export_targets":[0],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_24527":{"gid":24527,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.103:6839/189860875","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6838","nonce":189860875},{"type":"v1","addr":"172.21.15.103:6839","nonce":189860875}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}},"gid_25975":{"gid":25975,"name":"c","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.163:6837/3681372184","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6836","nonce":3681372184},{"type":"v1","addr":"172.21.15.163:6837","nonce":3681372184}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments","feature_12":"quiesce subvolumes"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24527,"qdb_cluster":[24511,24527,24491]},"id":1}]} 2024-05-09T09:44:53.437 INFO:teuthology.orchestra.run.smithi069.stderr:dumped fsmap epoch 53 2024-05-09T09:44:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 msgr2=0x7fb6300a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 0x7fb6300a5050 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fb634005fd0 tx=0x7fb634005c70 comp rx=0 tx=0).stop 2024-05-09T09:44:53.438 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 msgr2=0x7fb644103280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 secure :-1 s=READY pgs=1157 cs=0 l=1 rev1=1 crypto rx=0x7fb63c005e30 tx=0x7fb63c031ee0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 shutdown_connections 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fb6300a2b90 0x7fb6300a5050 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fb644102320 0x7fb644103280 unknown :-1 s=CLOSED pgs=1157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fb644101960 0x7fb644102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 --2- 172.21.15.69:0/3120379403 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fb644100770 0x7fb644104700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 >> 172.21.15.69:0/3120379403 conn(0x7fb6440fbce0 msgr2=0x7fb6440fd740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 shutdown_connections 2024-05-09T09:44:53.439 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.438+0000 7fb648a91640 1 -- 172.21.15.69:0/3120379403 wait complete. 2024-05-09T09:44:53.450 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-05-09T09:44:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 -- 172.21.15.69:0/2830003620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0100770 msgr2=0x7f5eb0100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.553 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/2830003620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0100770 0x7f5eb0100b90 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 crypto rx=0x7f5ea8009900 tx=0x7f5ea802a410 comp rx=0 tx=0).stop 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 -- 172.21.15.69:0/2830003620 shutdown_connections 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/2830003620 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0102320 0x7f5eb0108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/2830003620 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0101960 0x7f5eb0101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/2830003620 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0100770 0x7f5eb0100b90 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 -- 172.21.15.69:0/2830003620 >> 172.21.15.69:0/2830003620 conn(0x7f5eb00fbce0 msgr2=0x7f5eb00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 -- 172.21.15.69:0/2830003620 shutdown_connections 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.550+0000 7f5eb76d2640 1 -- 172.21.15.69:0/2830003620 wait complete. 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 Processor -- start 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 -- start start 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 0x7f5eb0197d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 0x7f5eb019f280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.554 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb010c4c0 con 0x7f5eb0101960 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb010c340 con 0x7f5eb0100770 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb76d2640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f5eb010c640 con 0x7f5eb0102320 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb4c6d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb4c6d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56142/0 (socket says 172.21.15.69:56142) 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb4c6d640 1 -- 172.21.15.69:0/1271309751 learned_addr learned my addr 172.21.15.69:0/1271309751 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.551+0000 7f5eb546e640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 0x7f5eb0197d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb5c6f640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 0x7f5eb019f280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 -- 172.21.15.69:0/1271309751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 msgr2=0x7f5eb019f280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 0x7f5eb019f280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 -- 172.21.15.69:0/1271309751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 msgr2=0x7f5eb0197d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 0x7f5eb0197d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 -- 172.21.15.69:0/1271309751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f5eb019f920 con 0x7f5eb0101960 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb546e640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 0x7f5eb0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:53.555 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb5c6f640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 0x7f5eb019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb4c6d640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b4a0 tx=0x7f5ea400b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4014070 con 0x7f5eb0101960 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f5ea4010af0 con 0x7f5eb0101960 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f5eb019fc10 con 0x7f5eb0101960 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f5ea4017d10 con 0x7f5eb0101960 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.552+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f5eb01a01b0 con 0x7f5eb0101960 2024-05-09T09:44:53.556 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.553+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f5eb01a03e0 con 0x7f5eb0101960 2024-05-09T09:44:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.554+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f5ea4017070 con 0x7f5eb0101960 2024-05-09T09:44:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.555+0000 7f5eae7fc640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 0x7f5e980a50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.555+0000 7f5eb546e640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 0x7f5e980a50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.555+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f5ea40ce7f0 con 0x7f5eb0101960 2024-05-09T09:44:53.557 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.555+0000 7f5eb546e640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 0x7f5e980a50d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80314e0 tx=0x7f5ea8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.559 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.559+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f5ea409b3d0 con 0x7f5eb0101960 2024-05-09T09:44:53.739 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.739+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) -- 0x7f5eb01a05b0 con 0x7f5eb0101960 2024-05-09T09:44:53.740 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.740+0000 7f5eae7fc640 1 -- 172.21.15.69:0/1271309751 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v98) ==== 74+0+24708 (secure 0 0 0) 0x7f5ea409acd0 con 0x7f5eb0101960 2024-05-09T09:44:53.741 INFO:teuthology.orchestra.run.smithi069.stdout: 2024-05-09T09:44:53.741 INFO:teuthology.orchestra.run.smithi069.stdout:{"epoch":98,"fsid":"8e3a7b52-0de3-11ef-bc98-c7b262605968","created":"2024-05-09T09:08:27.932250+0000","modified":"2024-05-09T09:27:27.102222+0000","last_up_change":"2024-05-09T09:15:42.725589+0000","last_in_change":"2024-05-09T09:15:22.340872+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-05-09T09:11:51.348723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-05-09T09:16:30.601729+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-05-09T09:16:31.484689+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-05-09T09:16:35.125832+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"54339b38-12b5-46a7-a293-1cea9feb97e6","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6802","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6803","nonce":1965346268}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6804","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6805","nonce":1965346268}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6808","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6809","nonce":1965346268}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6806","nonce":1965346268},{"type":"v1","addr":"172.21.15.69:6807","nonce":1965346268}]},"public_addr":"172.21.15.69:6803/1965346268","cluster_addr":"172.21.15.69:6805/1965346268","heartbeat_back_addr":"172.21.15.69:6809/1965346268","heartbeat_front_addr":"172.21.15.69:6807/1965346268","state":["exists","up"]},{"osd":1,"uuid":"9edd74d4-c926-46b9-b564-36ea82da5321","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6800","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6801","nonce":4055201011}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6802","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6803","nonce":4055201011}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6806","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6807","nonce":4055201011}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6804","nonce":4055201011},{"type":"v1","addr":"172.21.15.103:6805","nonce":4055201011}]},"public_addr":"172.21.15.103:6801/4055201011","cluster_addr":"172.21.15.103:6803/4055201011","heartbeat_back_addr":"172.21.15.103:6807/4055201011","heartbeat_front_addr":"172.21.15.103:6805/4055201011","state":["exists","up"]},{"osd":2,"uuid":"b8268e98-1c34-44a4-b53f-779dd3ea68ab","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6800","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6801","nonce":250759774}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6802","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6803","nonce":250759774}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6806","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6807","nonce":250759774}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6804","nonce":250759774},{"type":"v1","addr":"172.21.15.163:6805","nonce":250759774}]},"public_addr":"172.21.15.163:6801/250759774","cluster_addr":"172.21.15.163:6803/250759774","heartbeat_back_addr":"172.21.15.163:6807/250759774","heartbeat_front_addr":"172.21.15.163:6805/250759774","state":["exists","up"]},{"osd":3,"uuid":"ebea5b0f-0931-447f-848c-342af0a730a7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6810","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6811","nonce":3308296525}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6812","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6813","nonce":3308296525}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6816","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6817","nonce":3308296525}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6814","nonce":3308296525},{"type":"v1","addr":"172.21.15.69:6815","nonce":3308296525}]},"public_addr":"172.21.15.69:6811/3308296525","cluster_addr":"172.21.15.69:6813/3308296525","heartbeat_back_addr":"172.21.15.69:6817/3308296525","heartbeat_front_addr":"172.21.15.69:6815/3308296525","state":["exists","up"]},{"osd":4,"uuid":"7cf7c161-cb83-47cc-be48-1a7bcba78412","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6808","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6809","nonce":613528366}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6810","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6811","nonce":613528366}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6814","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6815","nonce":613528366}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6812","nonce":613528366},{"type":"v1","addr":"172.21.15.103:6813","nonce":613528366}]},"public_addr":"172.21.15.103:6809/613528366","cluster_addr":"172.21.15.103:6811/613528366","heartbeat_back_addr":"172.21.15.103:6815/613528366","heartbeat_front_addr":"172.21.15.103:6813/613528366","state":["exists","up"]},{"osd":5,"uuid":"3e0e1bf3-cc14-4202-a96a-a557f5041ac0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6808","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6809","nonce":3212961219}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6810","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6811","nonce":3212961219}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6814","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6815","nonce":3212961219}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6812","nonce":3212961219},{"type":"v1","addr":"172.21.15.163:6813","nonce":3212961219}]},"public_addr":"172.21.15.163:6809/3212961219","cluster_addr":"172.21.15.163:6811/3212961219","heartbeat_back_addr":"172.21.15.163:6815/3212961219","heartbeat_front_addr":"172.21.15.163:6813/3212961219","state":["exists","up"]},{"osd":6,"uuid":"ef0f7d90-b664-4519-a34a-27549cfe3406","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6818","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6819","nonce":3160153338}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6820","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6821","nonce":3160153338}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6824","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6825","nonce":3160153338}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6822","nonce":3160153338},{"type":"v1","addr":"172.21.15.69:6823","nonce":3160153338}]},"public_addr":"172.21.15.69:6819/3160153338","cluster_addr":"172.21.15.69:6821/3160153338","heartbeat_back_addr":"172.21.15.69:6825/3160153338","heartbeat_front_addr":"172.21.15.69:6823/3160153338","state":["exists","up"]},{"osd":7,"uuid":"7236537c-3738-4ad0-bc46-1a96a7bd8b70","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6816","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6817","nonce":4108396337}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6818","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6819","nonce":4108396337}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6822","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6823","nonce":4108396337}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6820","nonce":4108396337},{"type":"v1","addr":"172.21.15.103:6821","nonce":4108396337}]},"public_addr":"172.21.15.103:6817/4108396337","cluster_addr":"172.21.15.103:6819/4108396337","heartbeat_back_addr":"172.21.15.103:6823/4108396337","heartbeat_front_addr":"172.21.15.103:6821/4108396337","state":["exists","up"]},{"osd":8,"uuid":"e7cfa791-797c-4dab-84a1-d2ff938e96c4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6816","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6817","nonce":2914036945}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6818","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6819","nonce":2914036945}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6822","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6823","nonce":2914036945}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6820","nonce":2914036945},{"type":"v1","addr":"172.21.15.163:6821","nonce":2914036945}]},"public_addr":"172.21.15.163:6817/2914036945","cluster_addr":"172.21.15.163:6819/2914036945","heartbeat_back_addr":"172.21.15.163:6823/2914036945","heartbeat_front_addr":"172.21.15.163:6821/2914036945","state":["exists","up"]},{"osd":9,"uuid":"55ae1e0d-2d12-4aef-9449-e498e1cc719f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6826","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6827","nonce":2042751079}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6828","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6829","nonce":2042751079}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6832","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6833","nonce":2042751079}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.69:6830","nonce":2042751079},{"type":"v1","addr":"172.21.15.69:6831","nonce":2042751079}]},"public_addr":"172.21.15.69:6827/2042751079","cluster_addr":"172.21.15.69:6829/2042751079","heartbeat_back_addr":"172.21.15.69:6833/2042751079","heartbeat_front_addr":"172.21.15.69:6831/2042751079","state":["exists","up"]},{"osd":10,"uuid":"dbc63c27-da12-4420-b03a-0c4d5721bd9c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6824","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6825","nonce":955568662}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6826","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6827","nonce":955568662}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6830","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6831","nonce":955568662}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.103:6828","nonce":955568662},{"type":"v1","addr":"172.21.15.103:6829","nonce":955568662}]},"public_addr":"172.21.15.103:6825/955568662","cluster_addr":"172.21.15.103:6827/955568662","heartbeat_back_addr":"172.21.15.103:6831/955568662","heartbeat_front_addr":"172.21.15.103:6829/955568662","state":["exists","up"]},{"osd":11,"uuid":"9e951639-2f9f-4524-9e05-c9b222ac9a05","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6824","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6825","nonce":208079664}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6826","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6827","nonce":208079664}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6830","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6831","nonce":208079664}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.163:6828","nonce":208079664},{"type":"v1","addr":"172.21.15.163:6829","nonce":208079664}]},"public_addr":"172.21.15.163:6825/208079664","cluster_addr":"172.21.15.163:6827/208079664","heartbeat_back_addr":"172.21.15.163:6831/208079664","heartbeat_front_addr":"172.21.15.163:6829/208079664","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:01.503838+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:25.467115+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:11:49.156783+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:14.225935+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:12:39.275713+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:04.762793+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:30.179692+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:13:55.508847+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:21.149509+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:14:47.200267+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:14.120009+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-05-09T09:15:40.873381+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.1b","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11},{"from":2,"to":1}]},{"pgid":"3.0","mappings":[{"from":5,"to":9}]},{"pgid":"3.3","mappings":[{"from":4,"to":9}]},{"pgid":"3.34","mappings":[{"from":4,"to":9}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.e","mappings":[{"from":5,"to":8}]},{"pgid":"4.17","mappings":[{"from":9,"to":8}]},{"pgid":"4.1c","mappings":[{"from":9,"to":10}]},{"pgid":"4.1f","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.69:0/4182744829":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6801/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1673751021":"2024-05-10T09:09:29.020087+0000","172.21.15.69:0/1911852009":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/1056628795":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6800/936285351":"2024-05-10T09:09:29.020087+0000","172.21.15.69:6800/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/2520675955":"2024-05-10T09:09:02.961220+0000","172.21.15.69:6801/2869747387":"2024-05-10T09:09:02.961220+0000","172.21.15.69:0/3187678781":"2024-05-10T09:09:29.020087+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-05-09T09:44:53.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.742+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 msgr2=0x7f5e980a50d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.742+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 0x7f5e980a50d0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7f5ea80314e0 tx=0x7f5ea8000bc0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.743 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.742+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 msgr2=0x7f5eb0198250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.742+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 secure :-1 s=READY pgs=1159 cs=0 l=1 rev1=1 crypto rx=0x7f5ea400b4a0 tx=0x7f5ea400b960 comp rx=0 tx=0).stop 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 shutdown_connections 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f5e980a2c10 0x7f5e980a50d0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f5eb0102320 0x7f5eb019f280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f5eb0101960 0x7f5eb0198250 unknown :-1 s=CLOSED pgs=1159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 --2- 172.21.15.69:0/1271309751 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f5eb0100770 0x7f5eb0197d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 >> 172.21.15.69:0/1271309751 conn(0x7f5eb00fbce0 msgr2=0x7f5eb00689c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 shutdown_connections 2024-05-09T09:44:53.744 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.743+0000 7f5eb76d2640 1 -- 172.21.15.69:0/1271309751 wait complete. 2024-05-09T09:44:53.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:53 smithi069 ceph-mon[30123]: pgmap v1149: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:53.753 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:53 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1392885706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:53.755 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-05-09T09:44:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:53 smithi163 ceph-mon[34354]: pgmap v1149: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:53.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:53 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1392885706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:53.865 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- 172.21.15.69:0/2698787870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 msgr2=0x7f1250100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- 172.21.15.69:0/2698787870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250100b90 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 crypto rx=0x7f124c009900 tx=0x7f124c02a410 comp rx=0 tx=0).stop 2024-05-09T09:44:53.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- 172.21.15.69:0/2698787870 shutdown_connections 2024-05-09T09:44:53.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- 172.21.15.69:0/2698787870 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250102320 0x7f1250108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.866 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- 172.21.15.69:0/2698787870 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250101960 0x7f1250101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- 172.21.15.69:0/2698787870 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250100b90 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- 172.21.15.69:0/2698787870 >> 172.21.15.69:0/2698787870 conn(0x7f12500fbce0 msgr2=0x7f12500fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- 172.21.15.69:0/2698787870 shutdown_connections 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- 172.21.15.69:0/2698787870 wait complete. 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 Processor -- start 2024-05-09T09:44:53.867 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- start start 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 0x7f125019c880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250102320 0x7f125019eea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f125010c3f0 con 0x7f1250100770 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f125010c270 con 0x7f1250102320 2024-05-09T09:44:53.868 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f12586f5640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f125010c570 con 0x7f1250101960 2024-05-09T09:44:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.858+0000 7f1256491640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56160/0 (socket says 172.21.15.69:56160) 2024-05-09T09:44:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 -- 172.21.15.69:0/1628136625 learned_addr learned my addr 172.21.15.69:0/1628136625 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1255c90640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 0x7f125019c880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.869 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256c92640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250102320 0x7f125019eea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 -- 172.21.15.69:0/1628136625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 msgr2=0x7f125019c880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 0x7f125019c880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 -- 172.21.15.69:0/1628136625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250102320 msgr2=0x7f125019eea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250102320 0x7f125019eea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 -- 172.21.15.69:0/1628136625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f125019f7b0 con 0x7f1250100770 2024-05-09T09:44:53.870 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1255c90640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 0x7f125019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f1256491640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f124c031610 tx=0x7f124c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c03f030 con 0x7f1250100770 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f124c003ad0 con 0x7f1250100770 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f124c035070 con 0x7f1250100770 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f125019f9e0 con 0x7f1250100770 2024-05-09T09:44:53.871 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.859+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f125019fed0 con 0x7f1250100770 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.860+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1250107a90 con 0x7f1250100770 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.861+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f124c042030 con 0x7f1250100770 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.862+0000 7f12477fe640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 0x7f123c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.862+0000 7f1255c90640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 0x7f123c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.862+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(98..98 src has 1..98) ==== 9129+0+0 (secure 0 0 0) 0x7f124c0efba0 con 0x7f1250100770 2024-05-09T09:44:53.872 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.862+0000 7f1255c90640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 0x7f123c0a5050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f12500692a0 tx=0x7f124000a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:53.873 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:53.866+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f124c0bc780 con 0x7f1250100770 2024-05-09T09:44:53.874 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:53 smithi103 ceph-mon[34421]: pgmap v1149: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 6 op/s 2024-05-09T09:44:53.874 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:53 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1392885706' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.082 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.082+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) -- 0x7f12500624d0 con 0x7f1250100770 2024-05-09T09:44:54.084 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.084+0000 7f12477fe640 1 -- 172.21.15.69:0/1628136625 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v54) ==== 161+0+0 (secure 0 0 0) 0x7f124c0bc080 con 0x7f1250100770 2024-05-09T09:44:54.085 INFO:teuthology.orchestra.run.smithi069.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-05-09T09:44:54.087 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 msgr2=0x7f123c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 0x7f123c0a5050 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f12500692a0 tx=0x7f124000a5c0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 msgr2=0x7f1250069530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 crypto rx=0x7f124c031610 tx=0x7f124c004270 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 shutdown_connections 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f123c0a2b90 0x7f123c0a5050 secure :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7f12500692a0 tx=0x7f124000a5c0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1250102320 0x7f125019eea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1250101960 0x7f125019c880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 --2- 172.21.15.69:0/1628136625 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1250100770 0x7f1250069530 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 >> 172.21.15.69:0/1628136625 conn(0x7f12500fbce0 msgr2=0x7f1250104c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 shutdown_connections 2024-05-09T09:44:54.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.087+0000 7f12586f5640 1 -- 172.21.15.69:0/1628136625 wait complete. 2024-05-09T09:44:54.100 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-05-09T09:44:54.208 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 -- 172.21.15.69:0/858803918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20101960 msgr2=0x7f8e20101de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 --2- 172.21.15.69:0/858803918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20101960 0x7f8e20101de0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 crypto rx=0x7f8e08009900 tx=0x7f8e0802a410 comp rx=0 tx=0).stop 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 -- 172.21.15.69:0/858803918 shutdown_connections 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 --2- 172.21.15.69:0/858803918 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 0x7f8e20108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 --2- 172.21.15.69:0/858803918 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20101960 0x7f8e20101de0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.207+0000 7f8e2511b640 1 --2- 172.21.15.69:0/858803918 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20100770 0x7f8e20100b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.208+0000 7f8e2511b640 1 -- 172.21.15.69:0/858803918 >> 172.21.15.69:0/858803918 conn(0x7f8e200fbce0 msgr2=0x7f8e200fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.208+0000 7f8e2511b640 1 -- 172.21.15.69:0/858803918 shutdown_connections 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.208+0000 7f8e2511b640 1 -- 172.21.15.69:0/858803918 wait complete. 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.208+0000 7f8e2511b640 1 Processor -- start 2024-05-09T09:44:54.209 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.208+0000 7f8e2511b640 1 -- start start 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20101960 0x7f8e201980d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 0x7f8e2019f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2010d510 con 0x7f8e20100770 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2010d390 con 0x7f8e20101960 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8e2010d690 con 0x7f8e20102320 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56190/0 (socket says 172.21.15.69:56190) 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 -- 172.21.15.69:0/548383263 learned_addr learned my addr 172.21.15.69:0/548383263 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1f59e640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 0x7f8e2019f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1e59c640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20101960 0x7f8e201980d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:54.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 -- 172.21.15.69:0/548383263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 msgr2=0x7f8e2019f140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 0x7f8e2019f140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 -- 172.21.15.69:0/548383263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20101960 msgr2=0x7f8e201980d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20101960 0x7f8e201980d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 -- 172.21.15.69:0/548383263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8e2019f7b0 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e1ed9d640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e9c0 tx=0x7f8e1000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e1000cd80 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8e2019faa0 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8e100035f0 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8e10010690 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.209+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8e201a0010 con 0x7f8e20100770 2024-05-09T09:44:54.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.210+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8e20107b60 con 0x7f8e20100770 2024-05-09T09:44:54.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.212+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8e100037a0 con 0x7f8e20100770 2024-05-09T09:44:54.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.212+0000 7f8e17fff640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 0x7f8e0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:54.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.213+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9387+0+0 (secure 0 0 0) 0x7f8e100cf990 con 0x7f8e20100770 2024-05-09T09:44:54.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.213+0000 7f8e1e59c640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 0x7f8e0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:54.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.213+0000 7f8e1e59c640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 0x7f8e0c0a5050 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8e08000c00 tx=0x7f8e080315d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:54.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.216+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8e1009c500 con 0x7f8e20100770 2024-05-09T09:44:54.380 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:44:54.082+0000 7f01b69fd700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:44:54.430 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.430 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/2803619388 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x55cc4a82a4e0 con 0x55cc4a800c00 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mds[65542]: -- 172.21.15.163:0/2803619388 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55cc4a82da40 con 0x55cc4a800c00 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: starting mds.c at 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mds[65218]: -- 172.21.15.163:0/3225614317 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x55b50f7cfa40 con 0x55b50f7a3000 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: starting mds.f at 2024-05-09T09:44:54.431 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: starting mds.i at 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:44:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:44:54.178+0000 7f0701fba700 -1 PosixStack listen unable to listen on v2:0.0.0.0:6835/0: (98) Address already in use 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mds[64418]: -- 172.21.15.103:0/3421523217 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x560535b69a40 con 0x560535b3d000 2024-05-09T09:44:54.432 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: starting mds.e at 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mds[64725]: -- 172.21.15.103:0/311332301 <== mon.0 v2:172.21.15.69:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x555e6b163a40 con 0x555e6b136c00 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: starting mds.h at 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: ignoring --setuser ceph since I am not root 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: ignoring --setgroup ceph since I am not root 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mds[65037]: -- 172.21.15.103:0/935508131 <== mon.1 v2:172.21.15.163:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) ==== 194+0+0 (secure 0 0 0) 0x563e565f9a40 con 0x563e565cc800 2024-05-09T09:44:54.433 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: starting mds.b at 2024-05-09T09:44:54.443 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:54.443+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) -- 0x7f8e2002d420 con 0x7f8e20100770 2024-05-09T09:44:54.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3120379403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1271309751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T09:44:54.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:54 smithi069 ceph-mon[30123]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T09:44:54.805 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3120379403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1271309751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T09:44:54.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:54 smithi163 ceph-mon[34354]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T09:44:54.867 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3120379403' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1271309751' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: Health check cleared: MDS_DAMAGE (was: 2 MDSs report damaged metadata) 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/1628136625' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: osdmap e99: 12 total, 12 up, 12 in 2024-05-09T09:44:54.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:54 smithi103 ceph-mon[34421]: fsmap cephfs:0/3 6 up:standby, 3 failed 2024-05-09T09:44:55.086 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.086+0000 7f8e17fff640 1 -- 172.21.15.69:0/548383263 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v55) ==== 104+0+0 (secure 0 0 0) 0x7f8e1009be00 con 0x7f8e20100770 2024-05-09T09:44:55.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.089+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 msgr2=0x7f8e0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:55.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.089+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 0x7f8e0c0a5050 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7f8e08000c00 tx=0x7f8e080315d0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.089+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 msgr2=0x7f8e20197b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:55.090 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 crypto rx=0x7f8e1000e9c0 tx=0x7f8e1000ee80 comp rx=0 tx=0).stop 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 shutdown_connections 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8e0c0a2b90 0x7f8e0c0a5050 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8e20102320 0x7f8e2019f140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8e20101960 0x7f8e201980d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 --2- 172.21.15.69:0/548383263 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8e20100770 0x7f8e20197b90 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 >> 172.21.15.69:0/548383263 conn(0x7f8e200fbce0 msgr2=0x7f8e20104d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 shutdown_connections 2024-05-09T09:44:55.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.090+0000 7f8e2511b640 1 -- 172.21.15.69:0/548383263 wait complete. 2024-05-09T09:44:55.102 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-05-09T09:44:55.212 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 -- 172.21.15.69:0/1090480710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20100770 msgr2=0x7f8d20100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 --2- 172.21.15.69:0/1090480710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20100770 0x7f8d20100b90 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 crypto rx=0x7f8d18009ca0 tx=0x7f8d1802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 -- 172.21.15.69:0/1090480710 shutdown_connections 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 --2- 172.21.15.69:0/1090480710 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20102320 0x7f8d20108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 --2- 172.21.15.69:0/1090480710 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d20101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 --2- 172.21.15.69:0/1090480710 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20100770 0x7f8d20100b90 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 -- 172.21.15.69:0/1090480710 >> 172.21.15.69:0/1090480710 conn(0x7f8d200fbce0 msgr2=0x7f8d200fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 -- 172.21.15.69:0/1090480710 shutdown_connections 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 -- 172.21.15.69:0/1090480710 wait complete. 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.210+0000 7f8d2514f640 1 Processor -- start 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 -- start start 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20100770 0x7f8d20196b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d201970a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:55.213 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c440 con 0x7f8d20102320 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c2c0 con 0x7f8d20101960 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d2514f640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f8d2010c5c0 con 0x7f8d20100770 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56204/0 (socket says 172.21.15.69:56204) 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 -- 172.21.15.69:0/3000380651 learned_addr learned my addr 172.21.15.69:0/3000380651 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1e59c640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d201970a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1ed9d640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20100770 0x7f8d20196b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 -- 172.21.15.69:0/3000380651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20100770 msgr2=0x7f8d20196b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:55.214 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20100770 0x7f8d20196b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 -- 172.21.15.69:0/3000380651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 msgr2=0x7f8d201970a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d201970a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 -- 172.21.15.69:0/3000380651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f8d2019e780 con 0x7f8d20102320 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1e59c640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d201970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.211+0000 7f8d1f59e640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009fd0 tx=0x7f8d1000b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:55.215 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.212+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d10011830 con 0x7f8d20102320 2024-05-09T09:44:55.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.212+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f8d10016070 con 0x7f8d20102320 2024-05-09T09:44:55.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.212+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f8d2019ea70 con 0x7f8d20102320 2024-05-09T09:44:55.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.212+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000f2e0 con 0x7f8d20102320 2024-05-09T09:44:55.216 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.212+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f8d2019f040 con 0x7f8d20102320 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.213+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f8d20107bb0 con 0x7f8d20102320 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.214+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f8d100119d0 con 0x7f8d20102320 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.215+0000 7f8d17fff640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.215+0000 7f8d1ed9d640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.215+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(99..99 src has 1..99) ==== 9387+0+0 (secure 0 0 0) 0x7f8d100cf250 con 0x7f8d20102320 2024-05-09T09:44:55.217 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.215+0000 7f8d1ed9d640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8d18006010 tx=0x7f8d18009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:55.219 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.218+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f8d1009bd30 con 0x7f8d20102320 2024-05-09T09:44:55.405 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:55.405+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) -- 0x7f8d2002d1c0 con 0x7f8d20102320 2024-05-09T09:44:55.760 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: pgmap v1150: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: Cluster is now healthy 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6834/2505017656,v1:172.21.15.163:6837/2505017656] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6835/3683290738,v1:172.21.15.163:6838/3683290738] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.163:6836/859749775,v1:172.21.15.163:6839/859749775] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6834/60884510,v1:172.21.15.103:6837/60884510] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6836/1480446107,v1:172.21.15.103:6839/1480446107] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: mds.? [v2:172.21.15.103:6835/443877184,v1:172.21.15.103:6838/443877184] up:boot 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: fsmap 12 up:standby 2024-05-09T09:44:55.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:44:55.762 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:55 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: pgmap v1150: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: Cluster is now healthy 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6834/2505017656,v1:172.21.15.163:6837/2505017656] up:boot 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6835/3683290738,v1:172.21.15.163:6838/3683290738] up:boot 2024-05-09T09:44:55.806 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.163:6836/859749775,v1:172.21.15.163:6839/859749775] up:boot 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6834/60884510,v1:172.21.15.103:6837/60884510] up:boot 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6836/1480446107,v1:172.21.15.103:6839/1480446107] up:boot 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: mds.? [v2:172.21.15.103:6835/443877184,v1:172.21.15.103:6838/443877184] up:boot 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: fsmap 12 up:standby 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:44:55.807 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:55 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: pgmap v1150: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 255 B/s wr, 6 op/s 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: Cluster is now healthy 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/548383263' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6834/2505017656,v1:172.21.15.163:6837/2505017656] up:boot 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6835/3683290738,v1:172.21.15.163:6838/3683290738] up:boot 2024-05-09T09:44:55.868 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.163:6836/859749775,v1:172.21.15.163:6839/859749775] up:boot 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6834/60884510,v1:172.21.15.103:6837/60884510] up:boot 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6836/1480446107,v1:172.21.15.103:6839/1480446107] up:boot 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: mds.? [v2:172.21.15.103:6835/443877184,v1:172.21.15.103:6838/443877184] up:boot 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: fsmap 12 up:standby 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='mgr.14152 172.21.15.69:0/2709107867' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-05-09T09:44:55.869 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:55 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:56.088 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.087+0000 7f8d17fff640 1 -- 172.21.15.69:0/3000380651 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v100) ==== 181+0+0 (secure 0 0 0) 0x7f8d1009b630 con 0x7f8d20102320 2024-05-09T09:44:56.088 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_metadata' removed 2024-05-09T09:44:56.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 msgr2=0x7f8d0c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:56.091 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f8d18006010 tx=0x7f8d18009360 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 msgr2=0x7f8d2019e110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 secure :-1 s=READY pgs=1169 cs=0 l=1 rev1=1 crypto rx=0x7f8d10009fd0 tx=0x7f8d1000b8e0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 shutdown_connections 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f8d0c0a2b90 0x7f8d0c0a5050 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f8d20102320 0x7f8d2019e110 unknown :-1 s=CLOSED pgs=1169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f8d20101960 0x7f8d201970a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 --2- 172.21.15.69:0/3000380651 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f8d20100770 0x7f8d20196b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 >> 172.21.15.69:0/3000380651 conn(0x7f8d200fbce0 msgr2=0x7f8d20104d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:56.092 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.091+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 shutdown_connections 2024-05-09T09:44:56.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.092+0000 7f8d2514f640 1 -- 172.21.15.69:0/3000380651 wait complete. 2024-05-09T09:44:56.108 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 -- 172.21.15.69:0/3816231430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358102340 msgr2=0x7fd358108bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 --2- 172.21.15.69:0/3816231430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358102340 0x7fd358108bd0 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 crypto rx=0x7fd350009ca0 tx=0x7fd35002a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 -- 172.21.15.69:0/3816231430 shutdown_connections 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 --2- 172.21.15.69:0/3816231430 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358102340 0x7fd358108bd0 unknown :-1 s=CLOSED pgs=1170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 --2- 172.21.15.69:0/3816231430 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358101980 0x7fd358101e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 --2- 172.21.15.69:0/3816231430 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358100790 0x7fd358100bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 -- 172.21.15.69:0/3816231430 >> 172.21.15.69:0/3816231430 conn(0x7fd3580fbd00 msgr2=0x7fd3580fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 -- 172.21.15.69:0/3816231430 shutdown_connections 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.233+0000 7fd35ff90640 1 -- 172.21.15.69:0/3816231430 wait complete. 2024-05-09T09:44:56.236 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 Processor -- start 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 -- start start 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358100790 0x7fd358197c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 0x7fd35819f170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810c410 con 0x7fd358101980 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810c290 con 0x7fd358100790 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35ff90640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7fd35810c590 con 0x7fd358102340 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.69:3300/0 says I am v2:172.21.15.69:56226/0 (socket says 172.21.15.69:56226) 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 -- 172.21.15.69:0/410426681 learned_addr learned my addr 172.21.15.69:0/410426681 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35dd2c640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358100790 0x7fd358197c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35e52d640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 0x7fd35819f170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 -- 172.21.15.69:0/410426681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 msgr2=0x7fd35819f170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:56.237 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 0x7fd35819f170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 -- 172.21.15.69:0/410426681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358100790 msgr2=0x7fd358197c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358100790 0x7fd358197c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 -- 172.21.15.69:0/410426681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7fd35819f7e0 con 0x7fd358101980 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35e52d640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 0x7fd35819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.234+0000 7fd35d52b640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd34800e9f0 tx=0x7fd34800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.235+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd34800cd80 con 0x7fd358101980 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.235+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7fd35819fad0 con 0x7fd358101980 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.235+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7fd348010910 con 0x7fd358101980 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.235+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7fd3480188f0 con 0x7fd358101980 2024-05-09T09:44:56.238 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.235+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7fd3581a0070 con 0x7fd358101980 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.236+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7fd3581a0240 con 0x7fd358101980 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.237+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7fd348010ac0 con 0x7fd358101980 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.237+0000 7fd34effd640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 0x7fd3440a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.238+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 5 ==== osd_map(100..100 src has 1..100) ==== 8854+0+0 (secure 0 0 0) 0x7fd3480cf710 con 0x7fd358101980 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.238+0000 7fd35dd2c640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 0x7fd3440a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:56.239 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.238+0000 7fd35dd2c640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 0x7fd3440a5050 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd354005fd0 tx=0x7fd35400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:56.242 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.242+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7fd34809c400 con 0x7fd358101980 2024-05-09T09:44:56.421 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:56.420+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) -- 0x7fd3581a04b0 con 0x7fd358101980 2024-05-09T09:44:57.089 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.089+0000 7fd34effd640 1 -- 172.21.15.69:0/410426681 <== mon.0 v2:172.21.15.69:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v101) ==== 169+0+0 (secure 0 0 0) 0x7fd34809bd00 con 0x7fd358101980 2024-05-09T09:44:57.090 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data' removed 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 msgr2=0x7fd3440a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 0x7fd3440a5050 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7fd354005fd0 tx=0x7fd35400b390 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 msgr2=0x7fd358198150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 crypto rx=0x7fd34800e9f0 tx=0x7fd34800eeb0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 shutdown_connections 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7fd3440a2b90 0x7fd3440a5050 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7fd358102340 0x7fd35819f170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7fd358101980 0x7fd358198150 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 --2- 172.21.15.69:0/410426681 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7fd358100790 0x7fd358197c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 >> 172.21.15.69:0/410426681 conn(0x7fd3580fbd00 msgr2=0x7fd3580fd7a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:57.093 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 shutdown_connections 2024-05-09T09:44:57.094 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.092+0000 7fd35ff90640 1 -- 172.21.15.69:0/410426681 wait complete. 2024-05-09T09:44:57.111 DEBUG:teuthology.orchestra.run.smithi069:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 -- 172.21.15.69:0/2520250959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390100770 msgr2=0x7f1390100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 --2- 172.21.15.69:0/2520250959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390100770 0x7f1390100b90 secure :-1 s=READY pgs=1172 cs=0 l=1 rev1=1 crypto rx=0x7f1388009ca0 tx=0x7f138802a8c0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 -- 172.21.15.69:0/2520250959 shutdown_connections 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 --2- 172.21.15.69:0/2520250959 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390102320 0x7f1390108bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 --2- 172.21.15.69:0/2520250959 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 0x7f1390101de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 --2- 172.21.15.69:0/2520250959 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390100770 0x7f1390100b90 unknown :-1 s=CLOSED pgs=1172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 -- 172.21.15.69:0/2520250959 >> 172.21.15.69:0/2520250959 conn(0x7f13900fbd20 msgr2=0x7f13900fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 -- 172.21.15.69:0/2520250959 shutdown_connections 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.259+0000 7f1398190640 1 -- 172.21.15.69:0/2520250959 wait complete. 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 Processor -- start 2024-05-09T09:44:57.261 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 -- start start 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 --2- >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 0x7f1390198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 --2- >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390102320 0x7f139019f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 -- --> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] -- mon_getmap magic: 0 -- 0x7f139010c3e0 con 0x7f1390102320 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 -- --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_getmap magic: 0 -- 0x7f139010c260 con 0x7f1390100770 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1398190640 1 -- --> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] -- mon_getmap magic: 0 -- 0x7f139010c560 con 0x7f1390101960 2024-05-09T09:44:57.262 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1395f2c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:57.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1395f2c640 1 --2- >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.103:3300/0 says I am v2:172.21.15.69:46480/0 (socket says 172.21.15.69:46480) 2024-05-09T09:44:57.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f1395f2c640 1 -- 172.21.15.69:0/711854869 learned_addr learned my addr 172.21.15.69:0/711854869 (peer_addr_for_me v2:172.21.15.69:0/0) 2024-05-09T09:44:57.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f139672d640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390102320 0x7f139019f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:57.263 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.260+0000 7f139572b640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 0x7f1390198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:57.264 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 -- 172.21.15.69:0/711854869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 msgr2=0x7f1390198140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 0x7f1390198140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 -- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390102320 msgr2=0x7f139019f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390102320 0x7f139019f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 -- 172.21.15.69:0/711854869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({config=0+,monmap=0+}) -- 0x7f139019f7d0 con 0x7f1390100770 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1395f2c640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1388004af0 tx=0x7f13880093d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 1 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388042540 con 0x7f1390100770 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({mgrmap=0+}) -- 0x7f1390102b80 con 0x7f1390100770 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 2 ==== config(27 keys) ==== 1140+0+0 (secure 0 0 0) 0x7f13880426e0 con 0x7f1390100770 2024-05-09T09:44:57.265 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 3 ==== mon_map magic: 0 ==== 413+0+0 (secure 0 0 0) 0x7f1388030820 con 0x7f1390100770 2024-05-09T09:44:57.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.261+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_subscribe({osdmap=0}) -- 0x7f13901030d0 con 0x7f1390100770 2024-05-09T09:44:57.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.263+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) -- 0x7f1390103330 con 0x7f1390100770 2024-05-09T09:44:57.266 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.264+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 4 ==== mgrmap(e 18) ==== 137833+0+0 (secure 0 0 0) 0x7f138802f610 con 0x7f1390100770 2024-05-09T09:44:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.264+0000 7f1386ffd640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 0x7f137c0a5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-05-09T09:44:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.264+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 5 ==== osd_map(101..101 src has 1..101) ==== 8324+0+0 (secure 0 0 0) 0x7f1388101030 con 0x7f1390100770 2024-05-09T09:44:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.264+0000 7f139572b640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 0x7f137c0a5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-05-09T09:44:57.267 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.265+0000 7f139572b640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 0x7f137c0a5050 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f13901990c0 tx=0x7f1380011040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-05-09T09:44:57.269 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.269+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) ==== 72+0+186147 (secure 0 0 0) 0x7f1388062020 con 0x7f1390100770 2024-05-09T09:44:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:57 smithi103 ceph-mon[34421]: pgmap v1152: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2024-05-09T09:44:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:57 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:57 smithi103 ceph-mon[34421]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T09:44:57.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:57 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:57.467 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:57.466+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 --> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) -- 0x7f1390103b20 con 0x7f1390100770 2024-05-09T09:44:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:57 smithi069 ceph-mon[30123]: pgmap v1152: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2024-05-09T09:44:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:57 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:57 smithi069 ceph-mon[30123]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T09:44:57.510 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:57 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:57 smithi163 ceph-mon[34354]: pgmap v1152: 129 pgs: 129 active+clean; 445 MiB data, 12 GiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 716 B/s wr, 6 op/s 2024-05-09T09:44:57.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:57 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/3000380651' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:57.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:57 smithi163 ceph-mon[34354]: osdmap e100: 12 total, 12 up, 12 in 2024-05-09T09:44:57.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:57 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.105 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.104+0000 7f1386ffd640 1 -- 172.21.15.69:0/711854869 <== mon.2 v2:172.21.15.103:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v102) ==== 178+0+0 (secure 0 0 0) 0x7f13880cd970 con 0x7f1390100770 2024-05-09T09:44:58.105 INFO:teuthology.orchestra.run.smithi069.stderr:pool 'cephfs_data_ec' removed 2024-05-09T09:44:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 msgr2=0x7f137c0a5050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 0x7f137c0a5050 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f13901990c0 tx=0x7f1380011040 comp rx=0 tx=0).stop 2024-05-09T09:44:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 msgr2=0x7f1390197c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-05-09T09:44:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1388004af0 tx=0x7f13880093d0 comp rx=0 tx=0).stop 2024-05-09T09:44:58.108 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 shutdown_connections 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:6800/3550836912,v1:172.21.15.69:6801/3550836912] conn(0x7f137c0a2b90 0x7f137c0a5050 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.69:3300/0,v1:172.21.15.69:6789/0] conn(0x7f1390102320 0x7f139019f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.163:3300/0,v1:172.21.15.163:6789/0] conn(0x7f1390101960 0x7f1390198140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 --2- 172.21.15.69:0/711854869 >> [v2:172.21.15.103:3300/0,v1:172.21.15.103:6789/0] conn(0x7f1390100770 0x7f1390197c00 secure :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1388004af0 tx=0x7f13880093d0 comp rx=0 tx=0).stop 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 >> 172.21.15.69:0/711854869 conn(0x7f13900fbd20 msgr2=0x7f13900fd820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 shutdown_connections 2024-05-09T09:44:58.109 INFO:teuthology.orchestra.run.smithi069.stderr:2024-05-09T09:44:58.108+0000 7f1398190640 1 -- 172.21.15.69:0/711854869 wait complete. 2024-05-09T09:44:58.125 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:44:58.184 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:44:58.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:44:58.275 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-05-09T09:44:58.275 DEBUG:teuthology.orchestra.run.smithi069:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:44:58.294 DEBUG:teuthology.orchestra.run.smithi103:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:44:58.309 DEBUG:teuthology.orchestra.run.smithi163:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-05-09T09:44:58.330 INFO:tasks.cephadm:Stopping all daemons... 2024-05-09T09:44:58.330 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-05-09T09:44:58.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a 2024-05-09T09:44:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:58.355 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-mon[30123]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T09:44:58.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-mon[30123]: from='client.? 172.21.15.69:0/711854869' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.356 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-mon[30123]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:58 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:58.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:58 smithi103 ceph-mon[34421]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T09:44:58.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:58 smithi103 ceph-mon[34421]: from='client.? 172.21.15.69:0/711854869' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:44:58 smithi103 ceph-mon[34421]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:58 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/410426681' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-05-09T09:44:58.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:58 smithi163 ceph-mon[34354]: osdmap e101: 12 total, 12 up, 12 in 2024-05-09T09:44:58.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:58 smithi163 ceph-mon[34354]: from='client.? 172.21.15.69:0/711854869' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:44:58 smithi163 ceph-mon[34354]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-05-09T09:44:58.761 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 systemd[1]: Stopping Ceph mon.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:44:59.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:44:58.809+0000 7f01bc208700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:44:59.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a[30119]: 2024-05-09T09:44:58.809+0000 7f01bc208700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-05-09T09:44:59.260 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:58 smithi069 podman[97850]: 2024-05-09 09:44:58.969749879 +0000 UTC m=+0.366530401 container died 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc) 2024-05-09T09:44:59.650 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:59 smithi069 podman[97850]: 2024-05-09 09:44:59.365703552 +0000 UTC m=+0.762484084 container cleanup 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:44:59.650 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:59 smithi069 bash[97850]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a 2024-05-09T09:44:59.950 INFO:journalctl@ceph.mon.a.smithi069.stdout:May 09 09:44:59 smithi069 podman[97927]: 2024-05-09 09:44:59.651117756 +0000 UTC m=+0.678891785 container remove 6a7dcd9185a2d6f7d2a5bb0f3c971d8236ee9b3fd3022e1d3b8187e2a5f9b5c2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-a, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-05-09T09:44:59.968 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.a.service' 2024-05-09T09:45:00.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:00.076 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-05-09T09:45:00.076 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-05-09T09:45:00.076 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.b 2024-05-09T09:45:00.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:00 smithi103 systemd[1]: Stopping Ceph mon.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:01.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b[34417]: 2024-05-09T09:45:00.656+0000 7f9bcb65b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:01.082 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b[34417]: 2024-05-09T09:45:00.656+0000 7f9bcb65b700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-05-09T09:45:01.083 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:00 smithi103 podman[73039]: 2024-05-09 09:45:00.789786682 +0000 UTC m=+0.264107097 container died 65ad0f4b3714d4e1e3169e77e45c75f3c563e03d4b7617125b9bfaeb31383338 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T09:45:01.367 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 podman[73039]: 2024-05-09 09:45:01.207748693 +0000 UTC m=+0.682069116 container cleanup 65ad0f4b3714d4e1e3169e77e45c75f3c563e03d4b7617125b9bfaeb31383338 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:01.368 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 bash[73039]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b 2024-05-09T09:45:01.672 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.b.service' 2024-05-09T09:45:01.693 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 podman[73053]: 2024-05-09 09:45:01.442850689 +0000 UTC m=+0.656718257 container remove 65ad0f4b3714d4e1e3169e77e45c75f3c563e03d4b7617125b9bfaeb31383338 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-b, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_CLEAN=True) 2024-05-09T09:45:01.693 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.b.service: Deactivated successfully. 2024-05-09T09:45:01.694 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 systemd[1]: Stopped Ceph mon.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:45:01.694 INFO:journalctl@ceph.mon.b.smithi103.stdout:May 09 09:45:01 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.b.service: Consumed 17.947s CPU time. 2024-05-09T09:45:01.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:01.723 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-05-09T09:45:01.723 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-05-09T09:45:01.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.c 2024-05-09T09:45:02.055 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:01 smithi163 systemd[1]: Stopping Ceph mon.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:02.555 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c[34350]: 2024-05-09T09:45:02.139+0000 7f5f3ab57700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:02.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c[34350]: 2024-05-09T09:45:02.139+0000 7f5f3ab57700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-05-09T09:45:02.556 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:02 smithi163 podman[75419]: 2024-05-09 09:45:02.271987448 +0000 UTC m=+0.267055697 container died 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS) 2024-05-09T09:45:02.940 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:02 smithi163 podman[75419]: 2024-05-09 09:45:02.672879243 +0000 UTC m=+0.667947498 container cleanup 39dbdc4607cdf3267f99ab29ccba9858f8c7778e4585ecc3c6c179d61859b6f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T09:45:02.940 INFO:journalctl@ceph.mon.c.smithi163.stdout:May 09 09:45:02 smithi163 bash[75419]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mon-c 2024-05-09T09:45:03.064 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mon.c.service' 2024-05-09T09:45:03.109 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:03.109 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-05-09T09:45:03.109 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-05-09T09:45:03.109 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x 2024-05-09T09:45:03.327 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:45:03 smithi069 systemd[1]: Stopping Ceph mgr.x for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:04.011 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:45:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x[30352]: 2024-05-09T09:45:03.615+0000 7fb24760d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:04.011 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:45:03 smithi069 podman[100664]: 2024-05-09 09:45:03.646075788 +0000 UTC m=+0.162768369 container died 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T09:45:04.508 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:45:04 smithi069 podman[100664]: 2024-05-09 09:45:04.248795321 +0000 UTC m=+0.765487899 container cleanup 15f5ef413d4694c79853825a319fcb0eb30879a38b7ceabd6ffceddb654f8cd9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:96805dcf909541c4ca9a1e90258de8ae5b6fa1bc, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-05-09T09:45:04.509 INFO:journalctl@ceph.mgr.x.smithi069.stdout:May 09 09:45:04 smithi069 bash[100664]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-x 2024-05-09T09:45:04.666 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.x.service' 2024-05-09T09:45:04.704 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:04.705 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-05-09T09:45:04.705 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-05-09T09:45:04.705 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y 2024-05-09T09:45:05.041 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:04 smithi103 systemd[1]: Stopping Ceph mgr.y for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:05.367 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 podman[73164]: 2024-05-09 09:45:05.042102045 +0000 UTC m=+0.146728118 container died 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-05-09T09:45:05.707 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 podman[73164]: 2024-05-09 09:45:05.489545863 +0000 UTC m=+0.594171936 container cleanup 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc) 2024-05-09T09:45:05.707 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 bash[73164]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y 2024-05-09T09:45:05.952 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y.service' 2024-05-09T09:45:05.972 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 podman[73178]: 2024-05-09 09:45:05.707961705 +0000 UTC m=+0.656684172 container remove 1d980e30dd6a96a723bf06586e6c19849e8209b58f085fd73c2d1ace08102453 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-y, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-96805dc, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:45:05.973 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y.service: Main process exited, code=exited, status=143/n/a 2024-05-09T09:45:05.973 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y.service: Failed with result 'exit-code'. 2024-05-09T09:45:05.973 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 systemd[1]: Stopped Ceph mgr.y for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:45:05.973 INFO:journalctl@ceph.mgr.y.smithi103.stdout:May 09 09:45:05 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.y.service: Consumed 16.934s CPU time. 2024-05-09T09:45:05.985 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:05.985 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-05-09T09:45:05.986 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-05-09T09:45:05.986 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.z 2024-05-09T09:45:06.305 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:45:06 smithi163 systemd[1]: Stopping Ceph mgr.z for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:06.611 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:45:06 smithi163 podman[75543]: 2024-05-09 09:45:06.323904249 +0000 UTC m=+0.145016997 container died ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5) 2024-05-09T09:45:06.988 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:45:06 smithi163 podman[75543]: 2024-05-09 09:45:06.721232352 +0000 UTC m=+0.542345108 container cleanup ac5e3177a5666b85422c0880ca10e731b97e5ffefe6e67b6ab6c777814cfc88f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T09:45:06.988 INFO:journalctl@ceph.mgr.z.smithi163.stdout:May 09 09:45:06 smithi163 bash[75543]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mgr-z 2024-05-09T09:45:07.134 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mgr.z.service' 2024-05-09T09:45:07.167 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:07.168 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-05-09T09:45:07.168 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-05-09T09:45:07.168 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.0 2024-05-09T09:45:07.453 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:07 smithi069 systemd[1]: Stopping Ceph osd.0 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:07.760 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0[44850]: 2024-05-09T09:45:07.453+0000 7f1b5a67a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:07.760 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0[44850]: 2024-05-09T09:45:07.453+0000 7f1b5a67a700 -1 osd.0 102 *** Got signal Terminated *** 2024-05-09T09:45:07.760 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0[44850]: 2024-05-09T09:45:07.453+0000 7f1b5a67a700 -1 osd.0 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:13.320 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:13 smithi069 podman[100787]: 2024-05-09 09:45:13.034553964 +0000 UTC m=+5.675280850 container died 48a0ae1ad5aa559f3df71335b805e8cc9677e23847f693dc734eff53ad37b990 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-05-09T09:45:13.712 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:13 smithi069 podman[100787]: 2024-05-09 09:45:13.420164775 +0000 UTC m=+6.060891650 container cleanup 48a0ae1ad5aa559f3df71335b805e8cc9677e23847f693dc734eff53ad37b990 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507) 2024-05-09T09:45:13.712 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:13 smithi069 bash[100787]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0 2024-05-09T09:45:14.010 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:13 smithi069 podman[100804]: 2024-05-09 09:45:13.713424214 +0000 UTC m=+0.672641166 container remove 48a0ae1ad5aa559f3df71335b805e8cc9677e23847f693dc734eff53ad37b990 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T09:45:14.341 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:13.979335108 +0000 UTC m=+0.018532622 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:14.341 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:14.089700713 +0000 UTC m=+0.128898213 container create a471c36db9419b9c2a3d512dc63881f41ba7b3799cbfabb8e6c1df02ef9a9d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-09T09:45:14.341 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:14.342462917 +0000 UTC m=+0.381660418 container init a471c36db9419b9c2a3d512dc63881f41ba7b3799cbfabb8e6c1df02ef9a9d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0-deactivate, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-05-09T09:45:14.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:14.345980855 +0000 UTC m=+0.385178357 container start a471c36db9419b9c2a3d512dc63881f41ba7b3799cbfabb8e6c1df02ef9a9d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0-deactivate, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:14.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:14.417556492 +0000 UTC m=+0.456754010 container attach a471c36db9419b9c2a3d512dc63881f41ba7b3799cbfabb8e6c1df02ef9a9d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux ) 2024-05-09T09:45:14.745 INFO:journalctl@ceph.osd.0.smithi069.stdout:May 09 09:45:14 smithi069 podman[100899]: 2024-05-09 09:45:14.745376043 +0000 UTC m=+0.784573549 container died a471c36db9419b9c2a3d512dc63881f41ba7b3799cbfabb8e6c1df02ef9a9d67 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-0-deactivate, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:15.449 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.0.service' 2024-05-09T09:45:15.490 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:15.490 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-05-09T09:45:15.491 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-05-09T09:45:15.491 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.1 2024-05-09T09:45:15.811 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:15 smithi103 systemd[1]: Stopping Ceph osd.1 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:16.117 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:45:15.810+0000 7ff100ee9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:16.118 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:45:15.810+0000 7ff100ee9700 -1 osd.1 102 *** Got signal Terminated *** 2024-05-09T09:45:16.118 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1[39021]: 2024-05-09T09:45:15.810+0000 7ff100ee9700 -1 osd.1 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:21.745 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:21 smithi103 podman[73290]: 2024-05-09 09:45:21.439090647 +0000 UTC m=+5.760425356 container died 477b99ff28b9bd0a22e894c13c4f5285c715675d6a08defffa9b3bdf2173155a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1, GIT_CLEAN=True, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-09T09:45:22.113 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:21 smithi103 podman[73290]: 2024-05-09 09:45:21.853503457 +0000 UTC m=+6.174838165 container cleanup 477b99ff28b9bd0a22e894c13c4f5285c715675d6a08defffa9b3bdf2173155a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:22.113 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:21 smithi103 bash[73290]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1 2024-05-09T09:45:22.367 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:22 smithi103 podman[73307]: 2024-05-09 09:45:22.113684851 +0000 UTC m=+0.674204522 container remove 477b99ff28b9bd0a22e894c13c4f5285c715675d6a08defffa9b3bdf2173155a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:45:22.815 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:22 smithi103 podman[73409]: 2024-05-09 09:45:22.467611776 +0000 UTC m=+0.017933971 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:22.815 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:22 smithi103 podman[73409]: 2024-05-09 09:45:22.598104322 +0000 UTC m=+0.148426516 container create 3ed310ba67766a5a89e770952116fb41e811e1483b0cc48ef049f242ad05b1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:45:23.117 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:22 smithi103 podman[73409]: 2024-05-09 09:45:22.89357474 +0000 UTC m=+0.443896925 container init 3ed310ba67766a5a89e770952116fb41e811e1483b0cc48ef049f242ad05b1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1-deactivate, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T09:45:23.117 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:22 smithi103 podman[73409]: 2024-05-09 09:45:22.897382379 +0000 UTC m=+0.447704571 container start 3ed310ba67766a5a89e770952116fb41e811e1483b0cc48ef049f242ad05b1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-05-09T09:45:23.117 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:23 smithi103 podman[73409]: 2024-05-09 09:45:23.001814962 +0000 UTC m=+0.552137156 container attach 3ed310ba67766a5a89e770952116fb41e811e1483b0cc48ef049f242ad05b1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507) 2024-05-09T09:45:23.838 INFO:journalctl@ceph.osd.1.smithi103.stdout:May 09 09:45:23 smithi103 podman[73409]: 2024-05-09 09:45:23.494333772 +0000 UTC m=+1.044655949 container died 3ed310ba67766a5a89e770952116fb41e811e1483b0cc48ef049f242ad05b1dc (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-1-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-09T09:45:24.162 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.1.service' 2024-05-09T09:45:24.201 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:24.201 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-05-09T09:45:24.201 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-05-09T09:45:24.202 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.2 2024-05-09T09:45:24.516 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:24 smithi163 systemd[1]: Stopping Ceph osd.2 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:24.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2[39509]: 2024-05-09T09:45:24.515+0000 7fd6cf71a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:24.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2[39509]: 2024-05-09T09:45:24.515+0000 7fd6cf71a700 -1 osd.2 102 *** Got signal Terminated *** 2024-05-09T09:45:24.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2[39509]: 2024-05-09T09:45:24.515+0000 7fd6cf71a700 -1 osd.2 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:30.305 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:30 smithi163 podman[75667]: 2024-05-09 09:45:30.026799935 +0000 UTC m=+5.637104246 container died 97e859dd79bc52ee1d0f72e8f9f4e555dbce820c4f1af12cd0df33c4c50eef19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T09:45:30.702 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:30 smithi163 podman[75667]: 2024-05-09 09:45:30.426031698 +0000 UTC m=+6.036336005 container cleanup 97e859dd79bc52ee1d0f72e8f9f4e555dbce820c4f1af12cd0df33c4c50eef19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-05-09T09:45:30.702 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:30 smithi163 bash[75667]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2 2024-05-09T09:45:31.055 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:30 smithi163 podman[75697]: 2024-05-09 09:45:30.702829161 +0000 UTC m=+0.678452465 container remove 97e859dd79bc52ee1d0f72e8f9f4e555dbce820c4f1af12cd0df33c4c50eef19 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:45:31.389 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.007208147 +0000 UTC m=+0.018356963 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:31.389 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.137255085 +0000 UTC m=+0.148403900 container create 867af8be841b7afadf06a89f36123d78d041983495654045f1d9fd5a43fad682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2-deactivate, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:31.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.390117121 +0000 UTC m=+0.401265937 container init 867af8be841b7afadf06a89f36123d78d041983495654045f1d9fd5a43fad682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:45:31.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.393381881 +0000 UTC m=+0.404530698 container start 867af8be841b7afadf06a89f36123d78d041983495654045f1d9fd5a43fad682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-05-09T09:45:31.805 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.507293607 +0000 UTC m=+0.518442441 container attach 867af8be841b7afadf06a89f36123d78d041983495654045f1d9fd5a43fad682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0) 2024-05-09T09:45:32.219 INFO:journalctl@ceph.osd.2.smithi163.stdout:May 09 09:45:31 smithi163 podman[75788]: 2024-05-09 09:45:31.911385395 +0000 UTC m=+0.922534214 container died 867af8be841b7afadf06a89f36123d78d041983495654045f1d9fd5a43fad682 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-2-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:32.563 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.2.service' 2024-05-09T09:45:32.603 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:32.603 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-05-09T09:45:32.603 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-05-09T09:45:32.603 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.3 2024-05-09T09:45:32.890 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:32 smithi069 systemd[1]: Stopping Ceph osd.3 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:33.142 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:33 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:45:32.998+0000 7f0e4e6bb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:33.142 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:33 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:45:32.998+0000 7f0e4e6bb700 -1 osd.3 102 *** Got signal Terminated *** 2024-05-09T09:45:33.142 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:33 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:45:32.998+0000 7f0e4e6bb700 -1 osd.3 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:33.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:33.162+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:34.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:34 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:34.003+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:34.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:34.189+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:35.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:34.755+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:35.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:35 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:35.006+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:35.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:35 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:35.176+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:36.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:35.787+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:36.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:36 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:35.999+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:36.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:36 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:36.177+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:36.760 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:36 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:45:36.460+0000 7f0e4acaf700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.703891+0000 front 2024-05-09T09:45:11.703912+0000 (oldest deadline 2024-05-09T09:45:35.804085+0000) 2024-05-09T09:45:37.055 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:36.621+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:36.832+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:37.424 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:37 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:37.012+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:37.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:37 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:37.167+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:37.760 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:37 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3[51471]: 2024-05-09T09:45:37.424+0000 7f0e4acaf700 -1 osd.3 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.703891+0000 front 2024-05-09T09:45:11.703912+0000 (oldest deadline 2024-05-09T09:45:35.804085+0000) 2024-05-09T09:45:38.055 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:37.647+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:38.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:37.879+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:38.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:38 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:38.048+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:38.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:38 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:38.141+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:38.906 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:38.606+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:38.940 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:38 smithi069 podman[101005]: 2024-05-09 09:45:38.626695573 +0000 UTC m=+5.819303214 container died 1d3cd7b10f71357daf9a5f80a52989e30142c4dfa8f9a0dbeb5bf5455d09f727 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:45:39.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:39 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:39.026+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:39.261 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101005]: 2024-05-09 09:45:39.032021548 +0000 UTC m=+6.224629189 container cleanup 1d3cd7b10f71357daf9a5f80a52989e30142c4dfa8f9a0dbeb5bf5455d09f727 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-05-09T09:45:39.261 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 bash[101005]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3 2024-05-09T09:45:39.305 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:38.906+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:39.617 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101034]: 2024-05-09 09:45:39.308728209 +0000 UTC m=+0.683827946 container remove 1d3cd7b10f71357daf9a5f80a52989e30142c4dfa8f9a0dbeb5bf5455d09f727 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-09T09:45:39.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:39 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:39.180+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:39.868 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:39.576+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:39.945 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101121]: 2024-05-09 09:45:39.617955338 +0000 UTC m=+0.114172298 container create f6967d941a40765966472444a8215b8eaba1ad78e64a0b797605bc8405e3dd5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3-deactivate, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-05-09T09:45:39.946 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101121]: 2024-05-09 09:45:39.521440795 +0000 UTC m=+0.017657739 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:39.946 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101121]: 2024-05-09 09:45:39.845857389 +0000 UTC m=+0.342074309 container init f6967d941a40765966472444a8215b8eaba1ad78e64a0b797605bc8405e3dd5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T09:45:39.946 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101121]: 2024-05-09 09:45:39.849215201 +0000 UTC m=+0.345432120 container start f6967d941a40765966472444a8215b8eaba1ad78e64a0b797605bc8405e3dd5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:45:39.946 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:39 smithi069 podman[101121]: 2024-05-09 09:45:39.945871258 +0000 UTC m=+0.442088192 container attach f6967d941a40765966472444a8215b8eaba1ad78e64a0b797605bc8405e3dd5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=) 2024-05-09T09:45:40.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:40 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:40.044+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:40.261 INFO:journalctl@ceph.osd.3.smithi069.stdout:May 09 09:45:40 smithi069 podman[101121]: 2024-05-09 09:45:40.148708379 +0000 UTC m=+0.644925316 container died f6967d941a40765966472444a8215b8eaba1ad78e64a0b797605bc8405e3dd5c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-3-deactivate, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:40.305 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:39.868+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:40.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:40 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:40.219+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:40.810 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.3.service' 2024-05-09T09:45:40.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:40.849 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-05-09T09:45:40.849 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-05-09T09:45:40.849 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.4 2024-05-09T09:45:40.871 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:40.582+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:41.174 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:40 smithi103 systemd[1]: Stopping Ceph osd.4 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:41.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:41 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:41.021+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:41.305 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:40.871+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:41.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:41.207+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:41.618 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:45:41.175+0000 7f69739ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:41.618 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:45:41.175+0000 7f69739ba700 -1 osd.4 102 *** Got signal Terminated *** 2024-05-09T09:45:41.618 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:41 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:45:41.175+0000 7f69739ba700 -1 osd.4 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:42.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:41.824+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:42.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:41.589+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:42.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:42 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:42.007+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:42.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:42 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:42.249+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:42.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:42 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:42.460+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:43.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:42.792+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:43.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:42.619+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:43.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:42.619+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:43.444 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:43.033+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:43.444 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:43.033+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:43.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:43 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:43.251+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:43.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:43 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:43.444+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:44.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:43.750+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:44.055 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:43.664+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:44.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:43 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:43.664+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:44.456 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:44 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:44.072+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:44.456 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:44 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:44.072+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:44.612 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:44.249+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:44.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:44 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:44.456+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:44.867 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:44 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:45:44.612+0000 7f696ffae700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.661012+0000 front 2024-05-09T09:45:18.660953+0000 (oldest deadline 2024-05-09T09:45:43.961014+0000) 2024-05-09T09:45:45.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:44 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:44.743+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:45.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:44 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:44.689+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:45.056 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:44 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:44.689+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:45.449 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:45.105+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:45.449 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:45.105+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:45.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:45.222+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:45.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:45.243+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:45.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:45 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:45.449+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:46.008 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:45 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:45.770+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:46.008 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:45 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:45.724+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:46.008 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:45 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:45.724+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:46.117 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:45 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4[45952]: 2024-05-09T09:45:45.638+0000 7f696ffae700 -1 osd.4 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.661012+0000 front 2024-05-09T09:45:18.660953+0000 (oldest deadline 2024-05-09T09:45:43.961014+0000) 2024-05-09T09:45:46.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:46 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:46.008+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:46.440 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:46.107+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:46.441 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:46.107+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:46.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:46 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:46.283+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:46.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:46 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:46.242+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:46.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:46.441+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:46.967 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:46 smithi103 podman[73518]: 2024-05-09 09:45:46.643109085 +0000 UTC m=+5.595529011 container died 1cc7535215d7ee612af74c0fef5fe036bdcb1e2821ba46e08b81a551f7801f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, ceph=True) 2024-05-09T09:45:47.017 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:46 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:46.809+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:47.017 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:46 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:46.809+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:47.018 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:46 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:46.725+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:47.018 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:46 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:46.725+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:47.218 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73518]: 2024-05-09 09:45:47.076507143 +0000 UTC m=+6.028927076 container cleanup 1cc7535215d7ee612af74c0fef5fe036bdcb1e2821ba46e08b81a551f7801f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T09:45:47.219 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 bash[73518]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4 2024-05-09T09:45:47.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:47.018+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:47.453 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:47.072+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:47.453 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:47.072+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:47.618 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73536]: 2024-05-09 09:45:47.291173259 +0000 UTC m=+0.651975933 container remove 1cc7535215d7ee612af74c0fef5fe036bdcb1e2821ba46e08b81a551f7801f95 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-05-09T09:45:47.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:47 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:47.252+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:47.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:47 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:47.218+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:47.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:47 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:47.218+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:47.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:47 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:47.454+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:47.970 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:47.760+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:47.970 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:47.760+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:47.971 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:47.693+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:47.971 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:47.693+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:47.981 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73630]: 2024-05-09 09:45:47.593787431 +0000 UTC m=+0.019181128 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:47.981 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73630]: 2024-05-09 09:45:47.720774947 +0000 UTC m=+0.146168661 container create 280ddf30ddf8983ec8cd7c8b76cfea41f4111734cfe22fbed734cbbd1201e2b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4-deactivate, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507) 2024-05-09T09:45:47.981 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73630]: 2024-05-09 09:45:47.98175299 +0000 UTC m=+0.407146667 container init 280ddf30ddf8983ec8cd7c8b76cfea41f4111734cfe22fbed734cbbd1201e2b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4-deactivate, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:45:48.233 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:48 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:48.204+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:48.234 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:47 smithi103 podman[73630]: 2024-05-09 09:45:47.985078173 +0000 UTC m=+0.410471845 container start 280ddf30ddf8983ec8cd7c8b76cfea41f4111734cfe22fbed734cbbd1201e2b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:48.234 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:48 smithi103 podman[73630]: 2024-05-09 09:45:48.08192718 +0000 UTC m=+0.507320853 container attach 280ddf30ddf8983ec8cd7c8b76cfea41f4111734cfe22fbed734cbbd1201e2b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-05-09T09:45:48.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:47 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:47.971+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:48.409 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:48.061+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:48.409 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:48.061+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:48.617 INFO:journalctl@ceph.osd.4.smithi103.stdout:May 09 09:45:48 smithi103 podman[73630]: 2024-05-09 09:45:48.278476661 +0000 UTC m=+0.703870350 container died 280ddf30ddf8983ec8cd7c8b76cfea41f4111734cfe22fbed734cbbd1201e2b8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240507, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:45:48.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:48 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:48.233+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:48.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:48 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:48.233+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:48.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:48 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:48.409+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:48.964 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.4.service' 2024-05-09T09:45:49.004 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:49.004 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-05-09T09:45:49.004 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-05-09T09:45:49.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.5 2024-05-09T09:45:49.010 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:48 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:48.769+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:49.010 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:48 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:48.769+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:49.011 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:48 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:48.741+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:49.011 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:48 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:48.741+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:49.305 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 systemd[1]: Stopping Ceph osd.5 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:49.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:49.011+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:49.433 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:49.101+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:49.433 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:49.101+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:49.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:49 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:49.195+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:49.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:49 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:49.228+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:49.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:49 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:49.228+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:49.729 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:49.315+0000 7fe00142e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:49.729 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:49.315+0000 7fe00142e700 -1 osd.5 102 *** Got signal Terminated *** 2024-05-09T09:45:49.729 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:49.315+0000 7fe00142e700 -1 osd.5 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:49.730 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:49.729+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:49.730 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:49.729+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:49.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:49.433+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:50.011 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:49.781+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:50.012 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:49 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:49.781+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:50.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:50.012+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:50.423 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:50 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:50.121+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:50.423 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:50 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:50.121+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:50.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:50 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:50.146+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:50.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:50 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:50.270+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:50.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:50 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:50.270+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:50.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:50 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:50.423+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:51.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:50.750+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:51.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:50.750+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:51.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:50.750+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:51.057 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:50.756+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:51.057 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:50 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:50.756+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:51.436 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:51 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:51.126+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:51.436 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:51 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:51.126+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:51.519 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:51 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:51.160+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:51.519 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:51 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:51.305+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:51.519 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:51 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:51.305+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:51.519 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:51 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:51.305+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:51.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:51.057+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:51.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:51 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:51.436+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:52.022 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:51.804+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:52.023 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:51.804+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:52.023 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:51.804+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:29.284807+0000 front 2024-05-09T09:45:29.284780+0000 (oldest deadline 2024-05-09T09:45:51.485530+0000) 2024-05-09T09:45:52.023 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:51.712+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:52.023 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:51.712+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:52.023 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:51 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:51.712+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:52.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:52.023+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:52.466 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:52.147+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:52.467 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:52.147+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:52.467 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:52.147+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:52.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:52 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:52.160+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:52.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:52 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:52.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:52.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:52 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:52.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:52.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:52 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:52.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:52.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:52.467+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:52.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:52.467+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:52.994 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:52.721+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:52.994 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:52.721+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:52.994 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:52.721+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:52.995 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:52.823+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:52.995 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:52.823+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:52.995 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:52.823+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:29.284807+0000 front 2024-05-09T09:45:29.284780+0000 (oldest deadline 2024-05-09T09:45:51.485530+0000) 2024-05-09T09:45:53.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:52 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:52.995+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:53.454 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:53.156+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:53.454 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:53.156+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:53.454 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:53.156+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:53.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:53 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:53.176+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:53.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:53 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:53.268+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:53.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:53 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:53.268+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:53.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:53 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:53.268+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:53.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:53.454+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:53.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:53 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:53.454+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:53.988 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:53.671+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:53.988 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:53.671+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:53.988 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:53.671+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:53.988 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:53.816+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:12.382050+0000 front 2024-05-09T09:45:12.381934+0000 (oldest deadline 2024-05-09T09:45:36.382768+0000) 2024-05-09T09:45:53.989 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:53.816+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:16.383293+0000 front 2024-05-09T09:45:16.383279+0000 (oldest deadline 2024-05-09T09:45:42.283273+0000) 2024-05-09T09:45:53.989 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5[46482]: 2024-05-09T09:45:53.816+0000 7fdff9a1a700 -1 osd.5 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:29.284807+0000 front 2024-05-09T09:45:29.284780+0000 (oldest deadline 2024-05-09T09:45:51.485530+0000) 2024-05-09T09:45:54.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:53.988+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:54.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:53 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:53.988+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:54.462 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:54.203+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:54.463 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:54.203+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:54.463 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:54.203+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:54.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:54.156+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:54.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:54.257+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:54.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:54.257+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:54.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:54 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:54.257+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:54.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:54.463+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:54.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:54 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:54.463+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:54.979 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:54 smithi163 podman[75895]: 2024-05-09 09:45:54.918721948 +0000 UTC m=+5.713573879 container died 4ac1380d2892d7ded4233d7700f11121784618c64304523549840fb2a6064ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, GIT_CLEAN=True, ceph=True) 2024-05-09T09:45:54.980 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:54.683+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:54.980 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:54.683+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:54.980 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:54.683+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:55.233 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:54.979+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:55.233 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:54 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:54.979+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:55.508 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:55 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:55.210+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:55.509 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:55 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:55.210+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:55.509 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:55 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:55.210+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:55.555 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:55 smithi163 podman[75895]: 2024-05-09 09:45:55.349246789 +0000 UTC m=+6.144098717 container cleanup 4ac1380d2892d7ded4233d7700f11121784618c64304523549840fb2a6064ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux ) 2024-05-09T09:45:55.555 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:55 smithi163 bash[75895]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5 2024-05-09T09:45:55.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:55 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:55.163+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:55.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:55 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:55.240+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:55.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:55 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:55.240+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:55.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:55 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:55.240+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:55.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:55 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:55.509+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:55.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:55 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:55.509+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:55.953 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:55 smithi163 podman[75912]: 2024-05-09 09:45:55.592368253 +0000 UTC m=+0.666848531 container remove 4ac1380d2892d7ded4233d7700f11121784618c64304523549840fb2a6064ba9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:45:55.954 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:55.656+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:55.954 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:55.656+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:55.954 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:55.656+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:56.236 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:55.954+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:56.236 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:55 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:55.954+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:56.236 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:55.936967717 +0000 UTC m=+0.022419551 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:45:56.237 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:56.06023616 +0000 UTC m=+0.145687977 container create 8226d160eb0ab11adcaa3422dcc164e5fd4935f305183dd59518c009179534f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5-deactivate, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True) 2024-05-09T09:45:56.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:56.254+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:56.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:56.254+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:56.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:56.254+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:56.555 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:56.321269512 +0000 UTC m=+0.406721330 container init 8226d160eb0ab11adcaa3422dcc164e5fd4935f305183dd59518c009179534f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2) 2024-05-09T09:45:56.555 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:56.324750259 +0000 UTC m=+0.410202077 container start 8226d160eb0ab11adcaa3422dcc164e5fd4935f305183dd59518c009179534f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5-deactivate, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-05-09T09:45:56.555 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:56.396309726 +0000 UTC m=+0.481761552 container attach 8226d160eb0ab11adcaa3422dcc164e5fd4935f305183dd59518c009179534f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:45:56.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:56.211+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:56.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:56.211+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:45:56.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:56.204+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:56.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:56.204+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:56.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:56 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:56.204+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:56.982 INFO:journalctl@ceph.osd.5.smithi163.stdout:May 09 09:45:56 smithi163 podman[76015]: 2024-05-09 09:45:56.620217099 +0000 UTC m=+0.705668918 container died 8226d160eb0ab11adcaa3422dcc164e5fd4935f305183dd59518c009179534f0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-05-09T09:45:56.982 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:56 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:56.638+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:56.982 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:56 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:56.638+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:56.983 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:56 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:56.638+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:57.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:56.512+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:57.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:56 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:56.512+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:57.291 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:56 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:56.982+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:57.292 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:56 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:56.982+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:57.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.5.service' 2024-05-09T09:45:57.388 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:45:57.388 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-05-09T09:45:57.388 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-05-09T09:45:57.389 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.6 2024-05-09T09:45:57.558 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 systemd[1]: Stopping Ceph osd.6 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:45:57.558 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:57.266+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:57.558 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:57.266+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:57.558 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:57.266+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:57.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:57 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:57.180+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:57.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:57 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:57.180+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:45:57.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:57 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:57.179+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:57.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:57 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:57.179+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:57.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:57 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:57.179+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:58.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:57.557+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:58.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:57.557+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:58.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:57.710+0000 7fed8074a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:45:58.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:57.710+0000 7fed8074a700 -1 osd.6 102 *** Got signal Terminated *** 2024-05-09T09:45:58.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:57 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:57.710+0000 7fed8074a700 -1 osd.6 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:45:58.032 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:57 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:57.635+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:58.032 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:57 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:57.635+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:58.032 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:57 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:57.635+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:58.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:58 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:58.032+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:58.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:58 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:58.032+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:58.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:58.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:58.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:58.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:58.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:58.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:58.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:58.199+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:58.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:58.199+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:45:58.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:58.227+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:58.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:58.227+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:58.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:58 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:58.227+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:59.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:58.533+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:45:59.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:58.533+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:45:59.033 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:58 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:58.642+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:45:59.033 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:58 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:58.642+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:45:59.033 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:58 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:58.642+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:45:59.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:59 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:59.034+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:45:59.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:45:59 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:45:59.034+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:45:59.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:59.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:45:59.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:59.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:45:59.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:45:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:45:59.236+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:45:59.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:59.219+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:45:59.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:45:59.219+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:45:59.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:59.272+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:45:59.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:59.272+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:45:59.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:59.272+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:45:59.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:45:59 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:45:59.272+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:00.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:59.522+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:46:00.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:45:59 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:45:59.522+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:46:00.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:59 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:59.603+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:00.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:59 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:59.603+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:00.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:45:59 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:45:59.603+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:00.504 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:00.205+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:00.504 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:00.205+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:00.504 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:00.205+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:00.504 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:00.205+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:00.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:00.083+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:00.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:00.083+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:00.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:00.083+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:00.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:00.239+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:00.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:00.239+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:00.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:00.239+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:00.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:00.275+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:00.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:00.275+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:00.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:00.275+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:00.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:00 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:00.275+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:00.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:00.504+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:46:00.760 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:00 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:00.504+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:46:01.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:00.648+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:01.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:00.648+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:01.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:00 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:00.648+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:01.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:01.251+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:01.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:01.251+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:01.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:01.251+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:01.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:01.251+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:01.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:01.094+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:01.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:01.094+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:01.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:01.094+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:01.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:01.210+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:01.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:01.210+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:01.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:01.210+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:01.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:01.280+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:01.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:01.280+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:01.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:01.280+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:01.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:01 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:01.280+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:02.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:01.529+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:46:02.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:01 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:01.529+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:46:02.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:01.652+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:02.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:01.652+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:02.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:01 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:01.652+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:02.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:02.231+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:02.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:02.231+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:02.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:02.231+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:02.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:02.231+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:02.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:02.130+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:02.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:02.130+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:02.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:02.130+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:02.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:02.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:02.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:02.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:02.245+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:02.246+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:02.246+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:02.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:02.246+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:03.011 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:02.518+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.199004+0000 front 2024-05-09T09:45:20.199122+0000 (oldest deadline 2024-05-09T09:45:41.899212+0000) 2024-05-09T09:46:03.011 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:02.518+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.000502+0000 front 2024-05-09T09:45:26.000443+0000 (oldest deadline 2024-05-09T09:45:51.900580+0000) 2024-05-09T09:46:03.011 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:02 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6[58015]: 2024-05-09T09:46:02.518+0000 7fed78d36700 -1 osd.6 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:37.602866+0000 front 2024-05-09T09:45:37.602823+0000 (oldest deadline 2024-05-09T09:46:01.702940+0000) 2024-05-09T09:46:03.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:02.647+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:03.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:02.647+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:03.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:02 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:02.647+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:03.292 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:03.209+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:03.292 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:03.209+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:03.292 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:03.209+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:03.293 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:03 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:03.209+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:03.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:03.127+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:03.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:03.127+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:03.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:03.127+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:03.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:03.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:03.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:03.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:03.242+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:03.242+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:03.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:03 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:03.242+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:03.660 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:03 smithi069 podman[101226]: 2024-05-09 09:46:03.379689626 +0000 UTC m=+5.793779937 container died f03f5af84cdd61486dfa07969fb14766a677454ef69269716e43b816dacaaa0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-05-09T09:46:04.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:03 smithi069 podman[101226]: 2024-05-09 09:46:03.76865325 +0000 UTC m=+6.182743556 container cleanup f03f5af84cdd61486dfa07969fb14766a677454ef69269716e43b816dacaaa0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-05-09T09:46:04.010 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:03 smithi069 bash[101226]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6 2024-05-09T09:46:04.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:03.609+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:04.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:03.609+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:04.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:03 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:03.609+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:04.355 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101247]: 2024-05-09 09:46:04.053509207 +0000 UTC m=+0.670847637 container remove f03f5af84cdd61486dfa07969fb14766a677454ef69269716e43b816dacaaa0d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:46:04.356 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:04 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:04.191+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:04.356 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:04 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:04.191+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:04.356 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:04 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:04.191+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:04.356 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:04 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:04.191+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:04.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:04.081+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:04.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:04.081+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:04.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:04.081+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:04.614 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.259943249 +0000 UTC m=+0.018407198 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:04.614 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.371218072 +0000 UTC m=+0.129682036 container create 63147206874fc64e7ab3e6df1ea740459f0134d69f682580a5e833046dfaa8f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:46:04.614 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.615245394 +0000 UTC m=+0.373709316 container init 63147206874fc64e7ab3e6df1ea740459f0134d69f682580a5e833046dfaa8f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-05-09T09:46:04.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:04.204+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:04.204+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:04.204+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:04.215+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:04.215+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:04.215+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:04.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:04 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:04.215+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:04.908 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.618757292 +0000 UTC m=+0.377221214 container start 63147206874fc64e7ab3e6df1ea740459f0134d69f682580a5e833046dfaa8f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-09T09:46:04.908 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.723632267 +0000 UTC m=+0.482096200 container attach 63147206874fc64e7ab3e6df1ea740459f0134d69f682580a5e833046dfaa8f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-05-09T09:46:04.908 INFO:journalctl@ceph.osd.6.smithi069.stdout:May 09 09:46:04 smithi069 podman[101345]: 2024-05-09 09:46:04.909213338 +0000 UTC m=+0.667677262 container died 63147206874fc64e7ab3e6df1ea740459f0134d69f682580a5e833046dfaa8f7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-6-deactivate, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-09T09:46:05.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:04.654+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:05.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:04.654+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:05.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:04 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:04.654+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:05.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:05 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:05.177+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:05.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:05 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:05.177+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:05.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:05 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:05.177+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:05.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:05 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:05.177+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:05.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:05.067+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:05.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:05.067+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:05.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:05.067+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:05.561 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.6.service' 2024-05-09T09:46:05.602 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:05.603 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-05-09T09:46:05.603 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-05-09T09:46:05.603 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.7 2024-05-09T09:46:05.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:05.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:05.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.202+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:05.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:05.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:05.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:05.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:05.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:05.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:05.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:05.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:05.938 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 systemd[1]: Stopping Ceph osd.7 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:06.036 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:05.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:06.036 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:05.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:06.036 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:05 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:05.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:06.221 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.938+0000 7f61f6b84700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:06.222 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.938+0000 7f61f6b84700 -1 osd.7 102 *** Got signal Terminated *** 2024-05-09T09:46:06.222 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:05 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:05.938+0000 7f61f6b84700 -1 osd.7 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:46:06.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:06.036+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:06.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:06.036+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:06.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:06.036+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:06.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:06 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:06.165+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:06.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:06 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:06.165+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:06.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:06 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:06.165+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:06.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:06 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:06.165+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:06.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:06.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:06.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:06.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:06.247+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:06.222+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:06.222+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:06.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:06 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:06.222+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:07.051 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:06.688+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:07.052 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:06.688+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:07.052 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:06 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:06.688+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:07.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:07.051+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:07.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:07.051+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:07.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:07.051+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:07.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:07.051+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:07.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:07.192+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:07.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:07.192+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:07.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:07.192+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:07.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:07 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:07.192+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:07.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:07.231+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:07.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:07.231+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:07.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:07.231+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:07.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:07.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:07.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:07.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:07.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:07.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:07.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:07.255+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:08.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:07.680+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:08.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:07.680+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:08.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:07 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:07.680+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:08.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:08 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:08.212+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:08.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:08 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:08.212+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:08.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:08 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:08.212+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:08.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:08 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:08.212+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:08.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:08.059+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:08.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:08.059+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:08.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:08.059+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:08.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:08.059+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:08.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:08.191+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:08.191+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:08.191+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:08.295+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:08.295+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:08.295+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:08.295+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:08.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:08 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:08.295+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:09.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:08.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:09.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:08.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:09.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:08.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:09.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:08 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:08.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:09.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:09 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:09.230+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:09.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:09 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:09.230+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:09.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:09 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:09.230+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:09.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:09 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:09.230+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:09.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:09 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:09.230+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:09.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:09.055+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:09.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:09.055+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:09.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:09.055+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:09.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:09.055+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:09.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:09.238+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:09.238+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:09.238+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:09.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:09.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:09.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:09.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:09.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:09 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:09.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:10.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:09.620+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:10.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:09.620+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:10.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:09.620+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:10.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:09 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:09.620+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:10.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:10.069+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:10.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:10.069+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:10.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:10.069+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:10.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:10.069+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:10.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:10.203+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:18.446129+0000 front 2024-05-09T09:45:18.446160+0000 (oldest deadline 2024-05-09T09:45:44.847165+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:10.203+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:24.847716+0000 front 2024-05-09T09:45:24.847565+0000 (oldest deadline 2024-05-09T09:45:55.448127+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:10.203+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.448739+0000 front 2024-05-09T09:45:35.448853+0000 (oldest deadline 2024-05-09T09:45:59.548726+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7[52629]: 2024-05-09T09:46:10.203+0000 7f61ef170700 -1 osd.7 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:45.350179+0000 front 2024-05-09T09:45:45.350210+0000 (oldest deadline 2024-05-09T09:46:09.450362+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:10.306+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:10.306+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:10.306+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:10.306+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:10.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:10 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:10.306+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:10.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:10 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:10.267+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:10.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:10 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:10.267+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:10.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:10 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:10.267+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:10.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:10 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:10.267+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:10.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:10 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:10.267+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:11.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:10.630+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:11.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:10.630+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:11.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:10.630+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:11.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:10 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:10.630+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:11.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:11.080+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:11.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:11.080+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:11.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:11.080+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:11.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:11.080+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:11.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:11 smithi103 podman[73747]: 2024-05-09 09:46:11.389224285 +0000 UTC m=+5.591476610 container died 52d2a990c24743947eff57dceb4af59573359a6a31ffc1f71ada0daf7654fa3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-05-09T09:46:11.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:11.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:11.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:11.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:11.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:11.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:11.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:11.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:11.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:11.303+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:11.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:11 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:11.275+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:11.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:11 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:11.275+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:11.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:11 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:11.275+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:11.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:11 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:11.275+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:11.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:11 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:11.275+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:12.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:11.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:12.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:11.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:12.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:11.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:12.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:11 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:11.666+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:12.057 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:11 smithi103 podman[73747]: 2024-05-09 09:46:11.789546327 +0000 UTC m=+5.991798655 container cleanup 52d2a990c24743947eff57dceb4af59573359a6a31ffc1f71ada0daf7654fa3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2) 2024-05-09T09:46:12.057 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:11 smithi103 bash[73747]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7 2024-05-09T09:46:12.350 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73767]: 2024-05-09 09:46:12.057799694 +0000 UTC m=+0.667146175 container remove 52d2a990c24743947eff57dceb4af59573359a6a31ffc1f71ada0daf7654fa3e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:46:12.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:12.098+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:12.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:12.098+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:12.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:12.098+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:12.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:12.098+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:12.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.261881557 +0000 UTC m=+0.018242147 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:12.617 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.400494111 +0000 UTC m=+0.156854747 container create 0adb4b1c5079397f9a30960afd0493266a17e2f0bb14480d90731aab0d290aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True) 2024-05-09T09:46:12.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:12 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:12.349+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:12.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:12 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:12.349+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:12.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:12 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:12.349+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:12.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:12 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:12.349+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:12.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:12 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:12.349+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:12.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:12 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:12.307+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:12.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:12 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:12.307+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:12.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:12 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:12.307+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:12.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:12 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:12.307+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:12.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:12 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:12.307+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:12.984 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.695767221 +0000 UTC m=+0.452127811 container init 0adb4b1c5079397f9a30960afd0493266a17e2f0bb14480d90731aab0d290aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-05-09T09:46:12.984 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.69902862 +0000 UTC m=+0.455389215 container start 0adb4b1c5079397f9a30960afd0493266a17e2f0bb14480d90731aab0d290aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-09T09:46:12.984 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.794995329 +0000 UTC m=+0.551355931 container attach 0adb4b1c5079397f9a30960afd0493266a17e2f0bb14480d90731aab0d290aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0) 2024-05-09T09:46:13.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:12.655+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:13.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:12.655+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:13.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:12.655+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:13.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:12 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:12.655+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:13.347 INFO:journalctl@ceph.osd.7.smithi103.stdout:May 09 09:46:12 smithi103 podman[73853]: 2024-05-09 09:46:12.984888159 +0000 UTC m=+0.741248747 container died 0adb4b1c5079397f9a30960afd0493266a17e2f0bb14480d90731aab0d290aba (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-7-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-05-09T09:46:13.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:13.138+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:13.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:13.138+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:13.555 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:13.138+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:13.556 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:13.138+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:13.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:13 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:13.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:13.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:13 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:13.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:13.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:13 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:13.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:13.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:13 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:13.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:13.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:13 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:13.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:13.652 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.7.service' 2024-05-09T09:46:13.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:13.732 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-05-09T09:46:13.733 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-05-09T09:46:13.733 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.8 2024-05-09T09:46:13.742 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:13 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:13.331+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:13.742 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:13 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:13.331+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:13.742 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:13 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:13.331+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:13.742 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:13 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:13.331+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:13.742 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:13 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:13.331+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:14.036 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:13 smithi163 systemd[1]: Stopping Ceph osd.8 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:14.036 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.035+0000 7f76edd21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:14.036 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.035+0000 7f76edd21700 -1 osd.8 102 *** Got signal Terminated *** 2024-05-09T09:46:14.036 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.035+0000 7f76edd21700 -1 osd.8 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:46:14.036 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:13.607+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:14.037 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:13.607+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:14.037 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:13.607+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:14.037 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:13 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:13.607+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:14.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.105+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:14.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.105+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:14.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.105+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:14.306 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:14.105+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:14.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:14.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:14.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:14.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:14.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:14.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:14.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:14.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:14.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:14 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:14.347+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:14.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:14.356+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:14.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:14.356+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:14.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:14.356+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:14.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:14.356+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:14.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:14 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:14.356+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:15.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:14.574+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:15.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:14.574+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:15.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:14.574+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:15.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:14 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:14.574+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:15.537 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:15.122+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:15.537 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:15.122+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:15.537 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:15.122+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:15.537 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:15.122+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:15.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:15.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:15.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:15.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:15.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:15.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:15 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:15.360+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:15.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:15.537+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:15.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:15.537+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:15.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:15.537+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:15.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:15 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:15.537+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:15.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:15.375+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:15.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:15.375+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:15.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:15.375+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:15.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:15.375+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:15.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:15.375+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:16.500 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:16.144+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:16.500 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:16.144+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:16.500 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:16.144+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:16.500 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:16.144+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:16.501 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:16.144+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:51.956230+0000 front 2024-05-09T09:45:51.956254+0000 (oldest deadline 2024-05-09T09:46:15.456425+0000) 2024-05-09T09:46:16.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:16.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:16.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:16.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:16.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:16.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:16 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:16.369+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:16.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:16.500+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:16.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:16.500+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:16.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:16.500+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:16.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:16.500+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:16.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:16 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:16.500+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:16.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:16 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:16.368+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:16.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:16 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:16.368+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:16.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:16 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:16.368+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:16.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:16 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:16.368+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:16.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:16 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:16.368+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:17.490 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:17.157+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:17.490 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:17.157+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:17.490 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:17.157+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:17.490 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:17.157+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:17.491 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:17.157+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:51.956230+0000 front 2024-05-09T09:45:51.956254+0000 (oldest deadline 2024-05-09T09:46:15.456425+0000) 2024-05-09T09:46:17.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:17.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:17.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:17.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:17.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:17.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:17 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:17.357+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:17.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:17.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:17.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:17.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:17.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:17.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:17 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:17.328+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:17.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:17.490+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:17.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:17.490+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:17.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:17.490+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:17.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:17.490+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:17.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:17 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:17.490+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:18.496 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:18.137+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:19.250862+0000 front 2024-05-09T09:45:19.250716+0000 (oldest deadline 2024-05-09T09:45:45.150853+0000) 2024-05-09T09:46:18.496 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:18.137+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.052130+0000 front 2024-05-09T09:45:28.052071+0000 (oldest deadline 2024-05-09T09:45:53.352272+0000) 2024-05-09T09:46:18.496 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:18.137+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.653507+0000 front 2024-05-09T09:45:35.653456+0000 (oldest deadline 2024-05-09T09:45:59.153546+0000) 2024-05-09T09:46:18.496 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:18.137+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:46.155189+0000 front 2024-05-09T09:45:46.155175+0000 (oldest deadline 2024-05-09T09:46:06.655156+0000) 2024-05-09T09:46:18.496 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8[53199]: 2024-05-09T09:46:18.137+0000 7f76ea315700 -1 osd.8 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:51.956230+0000 front 2024-05-09T09:45:51.956254+0000 (oldest deadline 2024-05-09T09:46:15.456425+0000) 2024-05-09T09:46:18.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:18.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:18.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:18.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:18.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:18.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:18 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:18.309+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:18.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:18 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:18.305+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:18.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:18.496+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:18.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:18.496+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:18.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:18.496+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:18.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:18.496+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:18.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:18 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:18.496+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:19.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:19.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:19.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:19.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:19.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:19.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:19 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:19.315+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:19.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:19.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:19.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:19.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:19.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:19.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:19 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:19.281+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:19.805 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:19 smithi163 podman[76124]: 2024-05-09 09:46:19.608162941 +0000 UTC m=+5.685944562 container died e33eafbf32b427e5ce05adac5d1cadd78b43937799749383cee133260a7925c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-05-09T09:46:19.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:19.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:19.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:19.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:19.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:19.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:19.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:19.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:19.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:19.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:20.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76124]: 2024-05-09 09:46:20.061833848 +0000 UTC m=+6.139615514 container cleanup e33eafbf32b427e5ce05adac5d1cadd78b43937799749383cee133260a7925c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240507, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=) 2024-05-09T09:46:20.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 bash[76124]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8 2024-05-09T09:46:20.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:20.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:20.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:20.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:20.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:20.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:20 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:20.345+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:20.723 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76141]: 2024-05-09 09:46:20.396705144 +0000 UTC m=+0.785945414 container remove e33eafbf32b427e5ce05adac5d1cadd78b43937799749383cee133260a7925c6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, RELEASE=squid-96805dc) 2024-05-09T09:46:20.723 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:20 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:20.482+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:20.723 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:20 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:20.482+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:20.723 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:20 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:20.482+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:20.723 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:20 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:20.482+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:20.724 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:20 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:20.482+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:20.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:20 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:20.330+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:21.049 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76232]: 2024-05-09 09:46:20.627282282 +0000 UTC m=+0.018574932 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:21.049 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76232]: 2024-05-09 09:46:20.73919739 +0000 UTC m=+0.130490067 container create cd59521fe7d0deb29953bc83b3518a110a5f02dffa03963eb14b0aae09ee3cf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T09:46:21.049 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76232]: 2024-05-09 09:46:20.958646446 +0000 UTC m=+0.349939096 container init cd59521fe7d0deb29953bc83b3518a110a5f02dffa03963eb14b0aae09ee3cf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:46:21.049 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:20 smithi163 podman[76232]: 2024-05-09 09:46:20.961771748 +0000 UTC m=+0.353064379 container start cd59521fe7d0deb29953bc83b3518a110a5f02dffa03963eb14b0aae09ee3cf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T09:46:21.050 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:21 smithi163 podman[76232]: 2024-05-09 09:46:21.050203999 +0000 UTC m=+0.441496643 container attach cd59521fe7d0deb29953bc83b3518a110a5f02dffa03963eb14b0aae09ee3cf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8-deactivate, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507) 2024-05-09T09:46:21.305 INFO:journalctl@ceph.osd.8.smithi163.stdout:May 09 09:46:21 smithi163 podman[76232]: 2024-05-09 09:46:21.26082049 +0000 UTC m=+0.652113133 container died cd59521fe7d0deb29953bc83b3518a110a5f02dffa03963eb14b0aae09ee3cf1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-8-deactivate, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=) 2024-05-09T09:46:21.579 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:21 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:21.528+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:21.579 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:21 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:21.528+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:21.579 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:21 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:21.528+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:21.579 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:21 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:21.528+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:21.580 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:21 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:21.528+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:21.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:21.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:21.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:21.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:21.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:21.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:21 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:21.334+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:21.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:21.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:21.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:21.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:21.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:21.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:21.362+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:21.892 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.8.service' 2024-05-09T09:46:21.929 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:21.929 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-05-09T09:46:21.929 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-05-09T09:46:21.929 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.9 2024-05-09T09:46:22.254 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:21 smithi069 systemd[1]: Stopping Ceph osd.9 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:22.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.253+0000 7fded6c12700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.253+0000 7fded6c12700 -1 osd.9 102 *** Got signal Terminated *** 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.253+0000 7fded6c12700 -1 osd.9 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:22.511 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:22 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:22.347+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:22.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:22.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:22.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:22.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:22.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:22.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:22 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:22.285+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:23.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:22.564+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:23.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:22.564+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:23.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:22.564+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:23.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:22.564+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:23.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:22 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:22.564+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:23.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:23.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:23.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:23.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:23.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:23.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:23 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:23.322+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:23.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:23.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:23.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:23.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:23.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:23.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:23 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:23.376+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:23.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:23.518+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:23.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:23.518+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:23.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:23.518+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:23.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:23.518+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:23.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:23.518+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:24.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:24.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:24.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:24.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:24.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:24.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:24 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:24.328+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:24.730 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:24.471+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:24.730 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:24.471+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:24.730 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:24.471+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:24.730 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:24.471+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:24.730 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:24 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:24.471+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:24.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:24.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:24.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:24.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:24.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:24.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:24 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:24.384+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:25.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:25.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:25 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:25.348+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:25.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:25.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:25.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:25.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:25.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:25.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:25 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:25.414+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:25.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:25 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:25.489+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:25.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:25 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:25.489+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:25.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:25 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:25.489+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:25.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:25 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:25.489+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:25.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:25 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:25.489+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:26.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.303996+0000 front 2024-05-09T09:45:11.303776+0000 (oldest deadline 2024-05-09T09:45:33.603970+0000) 2024-05-09T09:46:26.760 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.005879+0000 front 2024-05-09T09:45:20.005966+0000 (oldest deadline 2024-05-09T09:45:42.305925+0000) 2024-05-09T09:46:26.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:28.207364+0000 front 2024-05-09T09:45:28.207397+0000 (oldest deadline 2024-05-09T09:45:51.707523+0000) 2024-05-09T09:46:26.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.808705+0000 front 2024-05-09T09:45:35.808729+0000 (oldest deadline 2024-05-09T09:45:59.908899+0000) 2024-05-09T09:46:26.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:44.610195+0000 front 2024-05-09T09:45:44.610323+0000 (oldest deadline 2024-05-09T09:46:08.710332+0000) 2024-05-09T09:46:26.761 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:26 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9[64846]: 2024-05-09T09:46:26.461+0000 7fded3206700 -1 osd.9 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:52.112118+0000 front 2024-05-09T09:45:52.112302+0000 (oldest deadline 2024-05-09T09:46:15.012275+0000) 2024-05-09T09:46:26.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:26 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:26.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:26.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:26 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:26.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:26.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:26 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:26.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:26.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:26 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:26.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:26.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:26 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:26.516+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:26.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:26.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:26 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:26.377+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:27.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:27.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:27.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:27.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:27.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:27.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:27.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:27.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:27.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:27 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:27.415+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:28.253 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:27 smithi069 podman[101453]: 2024-05-09 09:46:27.895124732 +0000 UTC m=+5.773025698 container died 753948ff2fd72bdb37d643bb8b05b7ad6dd007688805f81764b0c72aeefd0d6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-05-09T09:46:28.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:28 smithi069 podman[101453]: 2024-05-09 09:46:28.353246757 +0000 UTC m=+6.231147729 container cleanup 753948ff2fd72bdb37d643bb8b05b7ad6dd007688805f81764b0c72aeefd0d6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T09:46:28.510 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:28 smithi069 bash[101453]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:28.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:28 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:28.477+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:28.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:28.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:28 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:28.452+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:28.971 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:28 smithi069 podman[101470]: 2024-05-09 09:46:28.621503132 +0000 UTC m=+0.726080037 container remove 753948ff2fd72bdb37d643bb8b05b7ad6dd007688805f81764b0c72aeefd0d6a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T09:46:29.260 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:28 smithi069 podman[101558]: 2024-05-09 09:46:28.972275803 +0000 UTC m=+0.111264116 container create 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc) 2024-05-09T09:46:29.261 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:28 smithi069 podman[101558]: 2024-05-09 09:46:28.878819227 +0000 UTC m=+0.017807541 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:29.261 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 podman[101558]: 2024-05-09 09:46:29.166309303 +0000 UTC m=+0.305297617 container init 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-05-09T09:46:29.261 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 podman[101558]: 2024-05-09 09:46:29.169743478 +0000 UTC m=+0.308731798 container start 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T09:46:29.743 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 podman[101558]: 2024-05-09 09:46:29.26616616 +0000 UTC m=+0.405154484 container attach 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:46:29.743 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 podman[101558]: 2024-05-09 09:46:29.457367634 +0000 UTC m=+0.596355940 container died 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-05-09T09:46:29.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:29.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:29.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:29.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:29.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:29.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:29 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:29.461+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:29.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:29 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:29.477+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:29.974 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.9.service' 2024-05-09T09:46:29.998 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 podman[101584]: 2024-05-09 09:46:29.952789515 +0000 UTC m=+0.485279629 container remove 7de2ea35cf1e41273b0dc973a35926f22d27d144990ee5dc3361773c4c10b82c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-9-deactivate, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T09:46:29.998 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.9.service: Deactivated successfully. 2024-05-09T09:46:29.998 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 systemd[1]: Stopped Ceph osd.9 for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:46:29.998 INFO:journalctl@ceph.osd.9.smithi069.stdout:May 09 09:46:29 smithi069 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.9.service: Consumed 15min 32.869s CPU time. 2024-05-09T09:46:30.015 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:30.015 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-05-09T09:46:30.015 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-05-09T09:46:30.015 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.10 2024-05-09T09:46:30.343 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 systemd[1]: Stopping Ceph osd.10 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.343+0000 7f8807c0b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.343+0000 7f8807c0b700 -1 osd.10 102 *** Got signal Terminated *** 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.343+0000 7f8807c0b700 -1 osd.10 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:30.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:30.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:30.619 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:30 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:30.481+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:30.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:30 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:30.483+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:31.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:31.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:31.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:31.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:31.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:31.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:31.441+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:31.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:31.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:31 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:31.457+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:32.617 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:32.618 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:32 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:32.439+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:32.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:32.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:32.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:32.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:32.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:32.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:32 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:32.480+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:33.806 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:33 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:33.529+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:33.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:33.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:33 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:33.405+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:34.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:11.333751+0000 front 2024-05-09T09:45:11.333622+0000 (oldest deadline 2024-05-09T09:45:33.033727+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.635555+0000 front 2024-05-09T09:45:20.635593+0000 (oldest deadline 2024-05-09T09:45:46.535759+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:26.536220+0000 front 2024-05-09T09:45:26.536253+0000 (oldest deadline 2024-05-09T09:45:51.236424+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6814 osd.3 since back 2024-05-09T09:45:35.237989+0000 front 2024-05-09T09:45:35.237868+0000 (oldest deadline 2024-05-09T09:45:58.638577+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.339394+0000 front 2024-05-09T09:45:43.339350+0000 (oldest deadline 2024-05-09T09:46:08.039428+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:50.940538+0000 front 2024-05-09T09:45:50.940665+0000 (oldest deadline 2024-05-09T09:46:16.840391+0000) 2024-05-09T09:46:34.868 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:34 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10[59371]: 2024-05-09T09:46:34.390+0000 7f88041ff700 -1 osd.10 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.542373+0000 front 2024-05-09T09:46:02.542349+0000 (oldest deadline 2024-05-09T09:46:24.842284+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:35.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:34 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:34.566+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:36.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:36.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:35 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:35.597+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:36.201 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:35 smithi103 podman[73972]: 2024-05-09 09:46:35.922520674 +0000 UTC m=+5.704000807 container died ca1f32f32ea312e4889c69bd5bca142cc640cef920198a4722bc7fd00242e3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:46:36.568 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:36 smithi103 podman[73972]: 2024-05-09 09:46:36.301841921 +0000 UTC m=+6.083322107 container cleanup ca1f32f32ea312e4889c69bd5bca142cc640cef920198a4722bc7fd00242e3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc) 2024-05-09T09:46:36.569 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:36 smithi103 bash[73972]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10 2024-05-09T09:46:36.867 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:36 smithi103 podman[73990]: 2024-05-09 09:46:36.569780582 +0000 UTC m=+0.650628248 container remove ca1f32f32ea312e4889c69bd5bca142cc640cef920198a4722bc7fd00242e3f1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T09:46:37.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.640+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:37.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:36 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:36.641+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:37.155 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:36 smithi103 podman[74080]: 2024-05-09 09:46:36.785417818 +0000 UTC m=+0.018393252 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:37.155 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:36 smithi103 podman[74080]: 2024-05-09 09:46:36.90415014 +0000 UTC m=+0.137125573 container create 9e838bab83b0fb5eed23686e220f7bced4293481263011c69154c32787d28c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-05-09T09:46:37.155 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:37 smithi103 podman[74080]: 2024-05-09 09:46:37.15661135 +0000 UTC m=+0.389586782 container init 9e838bab83b0fb5eed23686e220f7bced4293481263011c69154c32787d28c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2) 2024-05-09T09:46:37.446 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:37 smithi103 podman[74080]: 2024-05-09 09:46:37.159750087 +0000 UTC m=+0.392725529 container start 9e838bab83b0fb5eed23686e220f7bced4293481263011c69154c32787d28c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10-deactivate, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-05-09T09:46:37.446 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:37 smithi103 podman[74080]: 2024-05-09 09:46:37.231723123 +0000 UTC m=+0.464698557 container attach 9e838bab83b0fb5eed23686e220f7bced4293481263011c69154c32787d28c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10-deactivate, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.schema-version=1.0) 2024-05-09T09:46:37.742 INFO:journalctl@ceph.osd.10.smithi103.stdout:May 09 09:46:37 smithi103 podman[74080]: 2024-05-09 09:46:37.446929596 +0000 UTC m=+0.679905024 container died 9e838bab83b0fb5eed23686e220f7bced4293481263011c69154c32787d28c71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-05-09T09:46:38.038 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.10.service' 2024-05-09T09:46:38.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:38.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:38.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:38.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:38.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:38.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:38.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:37 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:37.681+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:38.075 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:38.075 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-05-09T09:46:38.075 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-05-09T09:46:38.075 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.11 2024-05-09T09:46:38.364 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 systemd[1]: Stopping Ceph osd.11 for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:38.681 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.365+0000 7f7c9ee25700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:38.682 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.365+0000 7f7c9ee25700 -1 osd.11 102 *** Got signal Terminated *** 2024-05-09T09:46:38.682 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.365+0000 7f7c9ee25700 -1 osd.11 102 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-05-09T09:46:39.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:39.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:38 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:38.682+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:40.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:40.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:39 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:39.650+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:41.057 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:41.058 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:40 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:40.605+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:42.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:42.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:42.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:42.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:42.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:42.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:42.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:41 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:41.583+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:43.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6806 osd.0 since back 2024-05-09T09:45:10.823027+0000 front 2024-05-09T09:45:10.823180+0000 (oldest deadline 2024-05-09T09:45:34.322980+0000) 2024-05-09T09:46:43.055 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6804 osd.1 since back 2024-05-09T09:45:20.224405+0000 front 2024-05-09T09:45:20.224291+0000 (oldest deadline 2024-05-09T09:45:46.124391+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6804 osd.2 since back 2024-05-09T09:45:27.225402+0000 front 2024-05-09T09:45:27.225518+0000 (oldest deadline 2024-05-09T09:45:50.125619+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6812 osd.4 since back 2024-05-09T09:45:43.627580+0000 front 2024-05-09T09:45:43.627655+0000 (oldest deadline 2024-05-09T09:46:08.327624+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6812 osd.5 since back 2024-05-09T09:45:53.330415+0000 front 2024-05-09T09:45:53.330449+0000 (oldest deadline 2024-05-09T09:46:16.230543+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.69:6822 osd.6 since back 2024-05-09T09:46:02.031999+0000 front 2024-05-09T09:46:02.031948+0000 (oldest deadline 2024-05-09T09:46:26.731912+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.103:6820 osd.7 since back 2024-05-09T09:46:10.133670+0000 front 2024-05-09T09:46:10.133788+0000 (oldest deadline 2024-05-09T09:46:33.633580+0000) 2024-05-09T09:46:43.056 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:42 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11[60211]: 2024-05-09T09:46:42.629+0000 7f7c97411700 -1 osd.11 102 heartbeat_check: no reply from 172.21.15.163:6820 osd.8 since back 2024-05-09T09:46:15.934445+0000 front 2024-05-09T09:46:15.934424+0000 (oldest deadline 2024-05-09T09:46:41.834485+0000) 2024-05-09T09:46:44.305 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:43 smithi163 podman[76349]: 2024-05-09 09:46:43.945069251 +0000 UTC m=+5.669775106 container died cd7114a9398dac09f77e2b7f80fe9d30630bee71b3b1510de7b9b0e4112b3c72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-05-09T09:46:44.591 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:44 smithi163 podman[76349]: 2024-05-09 09:46:44.389925848 +0000 UTC m=+6.114631694 container cleanup cd7114a9398dac09f77e2b7f80fe9d30630bee71b3b1510de7b9b0e4112b3c72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:46:44.591 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:44 smithi163 bash[76349]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11 2024-05-09T09:46:45.004 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:44 smithi163 podman[76366]: 2024-05-09 09:46:44.591600081 +0000 UTC m=+0.637409974 container remove cd7114a9398dac09f77e2b7f80fe9d30630bee71b3b1510de7b9b0e4112b3c72 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-05-09T09:46:45.255 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76457]: 2024-05-09 09:46:44.908018915 +0000 UTC m=+0.017863009 image pull d7acac7d56572abea1532ef3a63a1b1ff716d7f1f7a4701680860a80b80cd662 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac 2024-05-09T09:46:45.255 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76457]: 2024-05-09 09:46:45.009076995 +0000 UTC m=+0.118921115 container create 85f29534c9097de3a7fc06777379405d1dcaca23dd59ba3b1d13d95f28981d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11-deactivate, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:46:45.255 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76457]: 2024-05-09 09:46:45.253305867 +0000 UTC m=+0.363149945 container init 85f29534c9097de3a7fc06777379405d1dcaca23dd59ba3b1d13d95f28981d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=) 2024-05-09T09:46:45.255 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76457]: 2024-05-09 09:46:45.256391757 +0000 UTC m=+0.366235836 container start 85f29534c9097de3a7fc06777379405d1dcaca23dd59ba3b1d13d95f28981d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-05-09T09:46:45.549 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76457]: 2024-05-09 09:46:45.345044456 +0000 UTC m=+0.454888537 container attach 85f29534c9097de3a7fc06777379405d1dcaca23dd59ba3b1d13d95f28981d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-05-09T09:46:45.805 INFO:journalctl@ceph.osd.11.smithi163.stdout:May 09 09:46:45 smithi163 podman[76484]: 2024-05-09 09:46:45.579729294 +0000 UTC m=+0.018694230 container died 85f29534c9097de3a7fc06777379405d1dcaca23dd59ba3b1d13d95f28981d42 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-osd-11-deactivate, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.build-date=20240507, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-05-09T09:46:46.111 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@osd.11.service' 2024-05-09T09:46:46.148 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:46.149 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-05-09T09:46:46.149 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-05-09T09:46:46.149 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.a 2024-05-09T09:46:46.446 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:46 smithi069 systemd[1]: Stopping Ceph mds.a for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:46.760 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a[71201]: 2024-05-09T09:46:46.446+0000 7f41f9407700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:46.760 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:46 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a[71201]: 2024-05-09T09:46:46.446+0000 7f41f9407700 -1 mds.a *** got signal Terminated *** 2024-05-09T09:46:48.212 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:47 smithi069 podman[101681]: 2024-05-09 09:46:47.94132635 +0000 UTC m=+1.598380630 container died faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-05-09T09:46:48.510 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:48 smithi069 podman[101681]: 2024-05-09 09:46:48.312550879 +0000 UTC m=+1.969605158 container cleanup faa5f4b26d2e9d38e5c977f94a7678c5ab206e44a4102a8ed209c9972c637252 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.license=GPLv2) 2024-05-09T09:46:48.510 INFO:journalctl@ceph.mds.a.smithi069.stdout:May 09 09:46:48 smithi069 bash[101681]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-a 2024-05-09T09:46:48.731 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.a.service' 2024-05-09T09:46:48.767 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:48.767 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-05-09T09:46:48.767 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-05-09T09:46:48.767 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.d 2024-05-09T09:46:49.010 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:48 smithi069 systemd[1]: Stopping Ceph mds.d for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:49.510 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d[70247]: 2024-05-09T09:46:49.125+0000 7f14de80a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:49.510 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:49 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d[70247]: 2024-05-09T09:46:49.125+0000 7f14de80a700 -1 mds.d *** got signal Terminated *** 2024-05-09T09:46:52.141 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:51 smithi069 podman[101801]: 2024-05-09 09:46:51.890304675 +0000 UTC m=+2.900501023 container died eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-05-09T09:46:52.510 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:52 smithi069 podman[101801]: 2024-05-09 09:46:52.224292785 +0000 UTC m=+3.234489143 container cleanup eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-05-09T09:46:52.510 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:52 smithi069 bash[101801]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d 2024-05-09T09:46:52.510 INFO:journalctl@ceph.mds.d.smithi069.stdout:May 09 09:46:52 smithi069 podman[101817]: 2024-05-09 09:46:52.383762236 +0000 UTC m=+0.486252922 container remove eaa752f966d3b2867062b451256f87bcc6d47f6d67ed30e0bd2b77f1884cb164 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-d, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, RELEASE=squid-96805dc, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-05-09T09:46:52.595 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.d.service' 2024-05-09T09:46:52.627 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:52.628 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-05-09T09:46:52.628 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-05-09T09:46:52.628 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.g 2024-05-09T09:46:52.953 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:52 smithi069 systemd[1]: Stopping Ceph mds.g for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:53.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g[70840]: 2024-05-09T09:46:52.952+0000 7f55393ba700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:53.260 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:52 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g[70840]: 2024-05-09T09:46:52.952+0000 7f55393ba700 -1 mds.g *** got signal Terminated *** 2024-05-09T09:46:57.236 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:56 smithi069 podman[101923]: 2024-05-09 09:46:56.960716264 +0000 UTC m=+4.114233586 container died 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.build-date=20240507) 2024-05-09T09:46:57.510 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:57 smithi069 podman[101923]: 2024-05-09 09:46:57.327727589 +0000 UTC m=+4.481244919 container cleanup 6ddbe82800b85527775605e1ca9d4c3463b4f3ed708a0741799fb57aacab55be (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, ceph=True, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-05-09T09:46:57.510 INFO:journalctl@ceph.mds.g.smithi069.stdout:May 09 09:46:57 smithi069 bash[101923]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-g 2024-05-09T09:46:57.748 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.g.service' 2024-05-09T09:46:57.780 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:46:57.780 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-05-09T09:46:57.780 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-05-09T09:46:57.780 DEBUG:teuthology.orchestra.run.smithi069:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.j 2024-05-09T09:46:58.114 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:46:57 smithi069 systemd[1]: Stopping Ceph mds.j for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:46:58.510 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:46:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-j[69897]: 2024-05-09T09:46:58.114+0000 7f313c4d9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:46:58.510 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:46:58 smithi069 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-j[69897]: 2024-05-09T09:46:58.114+0000 7f313c4d9700 -1 mds.j *** got signal Terminated *** 2024-05-09T09:47:01.197 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:47:00 smithi069 podman[102043]: 2024-05-09 09:47:00.943398393 +0000 UTC m=+2.936398668 container died bef9aa007455a4382982cf5fdaa1d039e142caec6f7ef6bfdd0e368ad878799f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-j, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-05-09T09:47:01.510 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:47:01 smithi069 podman[102043]: 2024-05-09 09:47:01.297884771 +0000 UTC m=+3.290885043 container cleanup bef9aa007455a4382982cf5fdaa1d039e142caec6f7ef6bfdd0e368ad878799f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-05-09T09:47:01.510 INFO:journalctl@ceph.mds.j.smithi069.stdout:May 09 09:47:01 smithi069 bash[102043]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-j 2024-05-09T09:47:01.717 DEBUG:teuthology.orchestra.run.smithi069:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.j.service' 2024-05-09T09:47:01.750 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:01.751 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-05-09T09:47:01.751 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-05-09T09:47:01.751 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.b 2024-05-09T09:47:02.069 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:01 smithi103 systemd[1]: Stopping Ceph mds.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:02.367 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: 2024-05-09T09:47:02.069+0000 7f0f97990700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:02.367 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:02 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b[65033]: 2024-05-09T09:47:02.069+0000 7f0f97990700 -1 mds.b *** got signal Terminated *** 2024-05-09T09:47:06.367 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 podman[74188]: 2024-05-09 09:47:06.084718471 +0000 UTC m=+4.116566770 container died 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240507) 2024-05-09T09:47:06.669 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 podman[74188]: 2024-05-09 09:47:06.486272758 +0000 UTC m=+4.518121065 container cleanup 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240507, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=squid-96805dc, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-05-09T09:47:06.670 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 bash[74188]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b 2024-05-09T09:47:06.916 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.b.service' 2024-05-09T09:47:06.937 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 podman[74213]: 2024-05-09 09:47:06.670851661 +0000 UTC m=+0.588780238 container remove 6dee82707971dde31dca361a4a8285da041f3d7a48713029472acad52e0d6d64 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-b, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240507, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-05-09T09:47:06.937 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.b.service: Deactivated successfully. 2024-05-09T09:47:06.938 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 systemd[1]: Stopped Ceph mds.b for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:47:06.938 INFO:journalctl@ceph.mds.b.smithi103.stdout:May 09 09:47:06 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.b.service: Consumed 21min 13.250s CPU time. 2024-05-09T09:47:06.951 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:06.951 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-05-09T09:47:06.951 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-05-09T09:47:06.951 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.e 2024-05-09T09:47:07.339 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:07 smithi103 systemd[1]: Stopping Ceph mds.e for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:07.618 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: 2024-05-09T09:47:07.339+0000 7f1de12f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:07.618 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:07 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e[64414]: 2024-05-09T09:47:07.339+0000 7f1de12f8700 -1 mds.e *** got signal Terminated *** 2024-05-09T09:47:10.507 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:10 smithi103 podman[74319]: 2024-05-09 09:47:10.196905356 +0000 UTC m=+3.021700768 container died 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240507) 2024-05-09T09:47:10.849 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:10 smithi103 podman[74319]: 2024-05-09 09:47:10.590598687 +0000 UTC m=+3.415394100 container cleanup 1605d788409f0c3bc51a06c6cd627ba6dce5a9ced33528ae231c677c6466f34c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e, GIT_CLEAN=True, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-05-09T09:47:10.849 INFO:journalctl@ceph.mds.e.smithi103.stdout:May 09 09:47:10 smithi103 bash[74319]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-e 2024-05-09T09:47:10.962 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.e.service' 2024-05-09T09:47:10.994 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:10.994 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-05-09T09:47:10.994 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-05-09T09:47:10.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.h 2024-05-09T09:47:11.117 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:11 smithi103 systemd[1]: Stopping Ceph mds.h for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:11.618 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:47:11.310+0000 7faba18f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:11.618 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:11 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h[64721]: 2024-05-09T09:47:11.310+0000 7faba18f8700 -1 mds.h *** got signal Terminated *** 2024-05-09T09:47:14.468 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:14 smithi103 podman[74440]: 2024-05-09 09:47:14.203882399 +0000 UTC m=+3.000055509 container died 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240507) 2024-05-09T09:47:14.845 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:14 smithi103 podman[74440]: 2024-05-09 09:47:14.569374243 +0000 UTC m=+3.365547351 container cleanup 7143ffd289bd2d76d9be2d79c4a5f2d0ab411310a84a2fd183a410ccf7690347 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h, ceph=True, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240507) 2024-05-09T09:47:14.845 INFO:journalctl@ceph.mds.h.smithi103.stdout:May 09 09:47:14 smithi103 bash[74440]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-h 2024-05-09T09:47:14.994 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.h.service' 2024-05-09T09:47:15.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:15.026 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-05-09T09:47:15.026 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-05-09T09:47:15.026 DEBUG:teuthology.orchestra.run.smithi103:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.k 2024-05-09T09:47:15.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:15 smithi103 systemd[1]: Stopping Ceph mds.k for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:15.867 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k[64094]: 2024-05-09T09:47:15.373+0000 7fa352f5e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:15.867 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:15 smithi103 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k[64094]: 2024-05-09T09:47:15.373+0000 7fa352f5e700 -1 mds.k *** got signal Terminated *** 2024-05-09T09:47:18.367 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 podman[74563]: 2024-05-09 09:47:18.112644473 +0000 UTC m=+2.869762166 container died e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-05-09T09:47:18.641 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 podman[74563]: 2024-05-09 09:47:18.448914762 +0000 UTC m=+3.206032476 container cleanup e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-05-09T09:47:18.641 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 bash[74563]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k 2024-05-09T09:47:18.872 DEBUG:teuthology.orchestra.run.smithi103:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.k.service' 2024-05-09T09:47:18.892 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 podman[74577]: 2024-05-09 09:47:18.64148152 +0000 UTC m=+0.522335783 container remove e77657395d3439d20c539484540b26a3cbec3a9a15041c6e02b19535392ea36a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-k, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=squid-96805dc, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux ) 2024-05-09T09:47:18.893 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.k.service: Deactivated successfully. 2024-05-09T09:47:18.893 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 systemd[1]: Stopped Ceph mds.k for 8e3a7b52-0de3-11ef-bc98-c7b262605968. 2024-05-09T09:47:18.893 INFO:journalctl@ceph.mds.k.smithi103.stdout:May 09 09:47:18 smithi103 systemd[1]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.k.service: Consumed 1.168s CPU time. 2024-05-09T09:47:18.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:18.903 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-05-09T09:47:18.903 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-05-09T09:47:18.903 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.c 2024-05-09T09:47:19.239 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:18 smithi163 systemd[1]: Stopping Ceph mds.c for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:19.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: 2024-05-09T09:47:19.239+0000 7f989ab5c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:19.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:19 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c[65538]: 2024-05-09T09:47:19.239+0000 7f989ab5c700 -1 mds.c *** got signal Terminated *** 2024-05-09T09:47:22.555 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:22 smithi163 podman[76563]: 2024-05-09 09:47:22.201829692 +0000 UTC m=+3.083889034 container died 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=squid-96805dc, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240507, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:47:22.947 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:22 smithi163 podman[76563]: 2024-05-09 09:47:22.647390397 +0000 UTC m=+3.529449751 container cleanup 0519309381ca7367811825d0f106140b09cfe1779279e44ef0d6a8352c714132 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS) 2024-05-09T09:47:22.947 INFO:journalctl@ceph.mds.c.smithi163.stdout:May 09 09:47:22 smithi163 bash[76563]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-c 2024-05-09T09:47:23.061 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.c.service' 2024-05-09T09:47:23.096 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:23.096 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-05-09T09:47:23.096 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-05-09T09:47:23.096 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.f 2024-05-09T09:47:23.305 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:23 smithi163 systemd[1]: Stopping Ceph mds.f for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:23.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: 2024-05-09T09:47:23.418+0000 7fd645b82700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:23.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:23 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f[65214]: 2024-05-09T09:47:23.418+0000 7fd645b82700 -1 mds.f *** got signal Terminated *** 2024-05-09T09:47:26.486 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:26 smithi163 podman[76696]: 2024-05-09 09:47:26.199703582 +0000 UTC m=+2.885987596 container died aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240507, RELEASE=squid-96805dc, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504) 2024-05-09T09:47:26.805 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:26 smithi163 podman[76696]: 2024-05-09 09:47:26.584958338 +0000 UTC m=+3.271242349 container cleanup aad47eda99fe9a097d2f5926107d074195439f8253d975f5ebf62dba432dd6b4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, ceph=True, org.label-schema.build-date=20240507, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-05-09T09:47:26.806 INFO:journalctl@ceph.mds.f.smithi163.stdout:May 09 09:47:26 smithi163 bash[76696]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-f 2024-05-09T09:47:26.999 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.f.service' 2024-05-09T09:47:27.032 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:27.032 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-05-09T09:47:27.032 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-05-09T09:47:27.033 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.i 2024-05-09T09:47:27.555 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:27 smithi163 systemd[1]: Stopping Ceph mds.i for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:27.555 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:47:27.330+0000 7f06fffb6700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:27.555 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:27 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i[65846]: 2024-05-09T09:47:27.330+0000 7f06fffb6700 -1 mds.i *** got signal Terminated *** 2024-05-09T09:47:30.555 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:30 smithi163 podman[76819]: 2024-05-09 09:47:30.199182575 +0000 UTC m=+2.954411892 container died 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i, ceph=True, RELEASE=squid-96805dc, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240507, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=) 2024-05-09T09:47:31.055 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:30 smithi163 podman[76819]: 2024-05-09 09:47:30.805560422 +0000 UTC m=+3.560789742 container cleanup 87c090455bbc0ef0466b52a8cb7f2e84b4e589608891b5f2ea3dcd3e1aef0986 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=squid-96805dc, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS) 2024-05-09T09:47:31.055 INFO:journalctl@ceph.mds.i.smithi163.stdout:May 09 09:47:30 smithi163 bash[76819]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-i 2024-05-09T09:47:31.207 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.i.service' 2024-05-09T09:47:31.238 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:31.238 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-05-09T09:47:31.238 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-05-09T09:47:31.239 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.l 2024-05-09T09:47:31.805 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:31 smithi163 systemd[1]: Stopping Ceph mds.l for 8e3a7b52-0de3-11ef-bc98-c7b262605968... 2024-05-09T09:47:31.805 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l[64906]: 2024-05-09T09:47:31.558+0000 7fca8821e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-05-09T09:47:31.805 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:31 smithi163 ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l[64906]: 2024-05-09T09:47:31.558+0000 7fca8821e700 -1 mds.l *** got signal Terminated *** 2024-05-09T09:47:35.545 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:35 smithi163 podman[76938]: 2024-05-09 09:47:35.146582811 +0000 UTC m=+3.691523326 container died 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240507, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-09T09:47:35.805 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:35 smithi163 podman[76938]: 2024-05-09 09:47:35.61804818 +0000 UTC m=+4.162988700 container cleanup 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=squid-96805dc, org.label-schema.build-date=20240507, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-05-09T09:47:35.806 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:35 smithi163 bash[76938]: ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l 2024-05-09T09:47:35.806 INFO:journalctl@ceph.mds.l.smithi163.stdout:May 09 09:47:35 smithi163 podman[76952]: 2024-05-09 09:47:35.785746134 +0000 UTC m=+0.545208635 container remove 6f32cfbe681312d9a97f69aade03aaa3fdc71768c9ed194539075a9d23a8deaa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:3b573cb0f3e3b36c210f25843c0a0bb7a496943c2eb5ca04944a503c246a55ac, name=ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968-mds-l, org.label-schema.license=GPLv2, org.label-schema.build-date=20240507, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=627304247bdbcb1f80af6cad8aac9a1ee2979504, CEPH_POINT_RELEASE=, RELEASE=squid-96805dc, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True) 2024-05-09T09:47:36.035 DEBUG:teuthology.orchestra.run.smithi163:> sudo pkill -f 'journalctl -f -n 0 -u ceph-8e3a7b52-0de3-11ef-bc98-c7b262605968@mds.l.service' 2024-05-09T09:47:36.066 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-05-09T09:47:36.067 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-05-09T09:47:36.067 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:47:36.258 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:47:38.847 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:47:39.037 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:47:41.458 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force --keep-logs 2024-05-09T09:47:41.649 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T09:47:44.106 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:47:44.139 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:47:44.171 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-05-09T09:47:44.197 INFO:tasks.cephadm:Archiving crash dumps... 2024-05-09T09:47:44.199 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi069/crash 2024-05-09T09:47:44.199 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:47:44.236 INFO:teuthology.orchestra.run.smithi069.stderr:tar: /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:47:44.236 INFO:teuthology.orchestra.run.smithi069.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:47:44.238 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi103/crash 2024-05-09T09:47:44.239 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:47:44.285 INFO:teuthology.orchestra.run.smithi103.stderr:tar: /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:47:44.286 INFO:teuthology.orchestra.run.smithi103.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:47:44.288 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi163/crash 2024-05-09T09:47:44.288 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash -- . 2024-05-09T09:47:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:tar: /var/lib/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/crash: Cannot open: No such file or directory 2024-05-09T09:47:44.329 INFO:teuthology.orchestra.run.smithi163.stderr:tar: Error is not recoverable: exiting now 2024-05-09T09:47:44.330 INFO:tasks.cephadm:Checking cluster log for badness... 2024-05-09T09:47:44.331 DEBUG:teuthology.orchestra.run.smithi069:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v 'filesystem is online with fewer MDS than max_mds' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v PG_AVAILABILITY | egrep -v PG_DEGRADED | egrep -v 'Reduced data availability' | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-05-09T09:47:44.409 INFO:tasks.cephadm:Compressing logs... 2024-05-09T09:47:44.409 DEBUG:teuthology.orchestra.run.smithi069:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:47:44.411 DEBUG:teuthology.orchestra.run.smithi103:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:47:44.414 DEBUG:teuthology.orchestra.run.smithi163:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-05-09T09:47:44.446 INFO:teuthology.orchestra.run.smithi069.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T09:47:44.447 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:47:44.448 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:47:44.448 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:47:44.449 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.g.log 2024-05-09T09:47:44.450 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.j.log 2024-05-09T09:47:44.450 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.3.log 2024-05-09T09:47:44.450 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.0.log 2024-05-09T09:47:44.450 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.x.log 2024-05-09T09:47:44.450 INFO:teuthology.orchestra.run.smithi103.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.a.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.0.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.6.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.a.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.a.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.d.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.a.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log: 86.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.9.log 2024-05-09T09:47:44.453 INFO:teuthology.orchestra.run.smithi069.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-05-09T09:47:44.460 INFO:teuthology.orchestra.run.smithi069.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.d.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.9.log: /var/log/ceph/ceph-client.admin.log: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-05-09T09:47:44.460 INFO:teuthology.orchestra.run.smithi069.stderr: 94.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.g.log.gz 2024-05-09T09:47:44.460 INFO:teuthology.orchestra.run.smithi069.stderr: 94.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.j.log.gz 2024-05-09T09:47:44.461 INFO:teuthology.orchestra.run.smithi069.stderr: 94.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.d.log.gz 2024-05-09T09:47:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:47:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:47:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:47:44.461 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.7.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.b.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.4.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.1.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.y.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.h.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.10.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.h.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.e.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.b.log 2024-05-09T09:47:44.462 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.k.log 2024-05-09T09:47:44.463 INFO:teuthology.orchestra.run.smithi163.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory 2024-05-09T09:47:44.463 INFO:teuthology.orchestra.run.smithi163.stderr: -5 --verbose -- /var/log/ceph/cephadm.log 2024-05-09T09:47:44.463 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.2.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.f.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.2.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.f.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.5.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.c.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.c.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.z.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.c.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.i.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.i.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.11.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.8.log 2024-05-09T09:47:44.464 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log 2024-05-09T09:47:44.465 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.l.log 2024-05-09T09:47:44.527 INFO:teuthology.orchestra.run.smithi103.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.b.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.k.log: 83.6% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:47:44.586 INFO:teuthology.orchestra.run.smithi103.stderr: 94.6% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.k.log.gz 2024-05-09T09:47:44.589 INFO:teuthology.orchestra.run.smithi163.stderr:/var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log: /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.l.log: 93.6% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.z.log.gz 2024-05-09T09:47:44.595 INFO:teuthology.orchestra.run.smithi163.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:47:44.612 INFO:teuthology.orchestra.run.smithi163.stderr: 86.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:47:44.615 INFO:teuthology.orchestra.run.smithi069.stderr: 90.2% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:47:44.621 INFO:teuthology.orchestra.run.smithi163.stderr: 90.4% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:47:44.623 INFO:teuthology.orchestra.run.smithi163.stderr: 83.5% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:47:44.630 INFO:teuthology.orchestra.run.smithi069.stderr: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:47:44.650 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:47:44.672 INFO:teuthology.orchestra.run.smithi103.stderr: 93.5% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.y.log.gz 2024-05-09T09:47:44.710 INFO:teuthology.orchestra.run.smithi163.stderr: 91.0% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:47:44.717 INFO:teuthology.orchestra.run.smithi103.stderr: 90.4% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.audit.log.gz 2024-05-09T09:47:44.804 INFO:teuthology.orchestra.run.smithi069.stderr: 83.3% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.cephadm.log.gz 2024-05-09T09:47:44.809 INFO:teuthology.orchestra.run.smithi069.stderr: 94.7% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.a.log.gz 2024-05-09T09:47:44.916 INFO:teuthology.orchestra.run.smithi103.stderr: 86.6% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph.log.gz 2024-05-09T09:47:45.096 INFO:teuthology.orchestra.run.smithi103.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-05-09T09:47:45.188 INFO:teuthology.orchestra.run.smithi103.stderr: 91.0% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-volume.log.gz 2024-05-09T09:47:45.448 INFO:teuthology.orchestra.run.smithi163.stderr: 94.6% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.l.log.gz 2024-05-09T09:47:47.078 INFO:teuthology.orchestra.run.smithi069.stderr: 90.5% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mgr.x.log.gz 2024-05-09T09:47:54.429 INFO:teuthology.orchestra.run.smithi103.stderr: 91.8% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.b.log.gz 2024-05-09T09:48:00.648 INFO:teuthology.orchestra.run.smithi163.stderr: 91.8% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.c.log.gz 2024-05-09T09:48:10.993 INFO:teuthology.orchestra.run.smithi069.stderr: 92.3% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mon.a.log.gz 2024-05-09T09:48:42.949 INFO:teuthology.orchestra.run.smithi163.stderr: 91.2% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.11.log.gz 2024-05-09T09:49:08.571 INFO:teuthology.orchestra.run.smithi069.stderr: 90.9% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.9.log.gz 2024-05-09T09:49:33.879 INFO:teuthology.orchestra.run.smithi069.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.6.log.gz 2024-05-09T09:50:26.572 INFO:teuthology.orchestra.run.smithi069.stderr: 91.3% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.0.log.gz 2024-05-09T09:50:41.461 INFO:teuthology.orchestra.run.smithi069.stderr: 91.0% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.3.log.gz 2024-05-09T09:50:41.463 INFO:teuthology.orchestra.run.smithi069.stderr: 2024-05-09T09:50:41.463 INFO:teuthology.orchestra.run.smithi069.stderr:real 2m57.039s 2024-05-09T09:50:41.463 INFO:teuthology.orchestra.run.smithi069.stderr:user 4m51.393s 2024-05-09T09:50:41.463 INFO:teuthology.orchestra.run.smithi069.stderr:sys 0m10.582s 2024-05-09T09:51:22.492 INFO:teuthology.orchestra.run.smithi163.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.8.log.gz 2024-05-09T09:52:39.539 INFO:teuthology.orchestra.run.smithi163.stderr: 94.9% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.f.log.gz 2024-05-09T09:52:49.965 INFO:teuthology.orchestra.run.smithi163.stderr: 94.8% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.c.log.gz 2024-05-09T09:55:18.042 INFO:teuthology.orchestra.run.smithi103.stderr: 95.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.e.log.gz 2024-05-09T09:59:07.451 INFO:teuthology.orchestra.run.smithi163.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.2.log.gz 2024-05-09T09:59:09.070 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.10.log.gz 2024-05-09T09:59:13.631 INFO:teuthology.orchestra.run.smithi103.stderr: 91.4% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.1.log.gz 2024-05-09T10:02:16.264 INFO:teuthology.orchestra.run.smithi163.stderr: 90.9% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.5.log.gz 2024-05-09T10:04:57.251 INFO:teuthology.orchestra.run.smithi163.stderr: 93.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.i.log.gz 2024-05-09T10:04:57.253 INFO:teuthology.orchestra.run.smithi163.stderr: 2024-05-09T10:04:57.253 INFO:teuthology.orchestra.run.smithi163.stderr:real 17m12.826s 2024-05-09T10:04:57.253 INFO:teuthology.orchestra.run.smithi163.stderr:user 9m4.187s 2024-05-09T10:04:57.253 INFO:teuthology.orchestra.run.smithi163.stderr:sys 0m24.723s 2024-05-09T10:06:37.646 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.7.log.gz 2024-05-09T10:07:39.660 INFO:teuthology.orchestra.run.smithi103.stderr: 91.1% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-osd.4.log.gz 2024-05-09T10:32:05.784 INFO:teuthology.orchestra.run.smithi103.stderr: 93.3% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.h.log.gz 2024-05-09T10:33:17.444 INFO:teuthology.orchestra.run.smithi103.stderr: 93.2% -- replaced with /var/log/ceph/8e3a7b52-0de3-11ef-bc98-c7b262605968/ceph-mds.b.log.gz 2024-05-09T10:33:17.488 INFO:teuthology.orchestra.run.smithi103.stderr: 2024-05-09T10:33:17.488 INFO:teuthology.orchestra.run.smithi103.stderr:real 45m33.064s 2024-05-09T10:33:17.489 INFO:teuthology.orchestra.run.smithi103.stderr:user 14m40.435s 2024-05-09T10:33:17.489 INFO:teuthology.orchestra.run.smithi103.stderr:sys 0m42.556s 2024-05-09T10:33:17.489 INFO:tasks.cephadm:Archiving logs... 2024-05-09T10:33:17.491 DEBUG:teuthology.misc:Transferring archived files from smithi069:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi069/log 2024-05-09T10:33:17.492 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T10:34:11.064 DEBUG:teuthology.misc:Transferring archived files from smithi103:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi103/log 2024-05-09T10:34:11.065 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T10:36:30.967 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi163/log 2024-05-09T10:36:30.969 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /var/log/ceph -- . 2024-05-09T10:38:02.549 INFO:tasks.cephadm:Removing cluster... 2024-05-09T10:38:02.550 DEBUG:teuthology.orchestra.run.smithi069:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force 2024-05-09T10:38:02.837 INFO:teuthology.orchestra.run.smithi069.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T10:38:03.521 DEBUG:teuthology.orchestra.run.smithi103:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force 2024-05-09T10:38:04.326 INFO:teuthology.orchestra.run.smithi103.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T10:38:05.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 8e3a7b52-0de3-11ef-bc98-c7b262605968 --force 2024-05-09T10:38:05.853 INFO:teuthology.orchestra.run.smithi163.stdout:Deleting cluster with fsid: 8e3a7b52-0de3-11ef-bc98-c7b262605968 2024-05-09T10:38:06.787 INFO:tasks.cephadm:Removing cephadm ... 2024-05-09T10:38:06.787 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T10:38:06.807 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T10:38:06.826 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf /home/ubuntu/cephtest/cephadm 2024-05-09T10:38:06.842 INFO:tasks.cephadm:Teardown complete 2024-05-09T10:38:06.842 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-05-09T10:38:06.855 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-05-09T10:38:06.855 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T10:38:06.858 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T10:38:06.868 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-05-09T10:38:07.039 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-05-09T10:38:07.050 INFO:teuthology.task.clock:Checking final clock skew... 2024-05-09T10:38:07.050 DEBUG:teuthology.orchestra.run.smithi069:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:38:07.053 DEBUG:teuthology.orchestra.run.smithi103:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:38:07.056 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-05-09T10:38:07.068 INFO:teuthology.orchestra.run.smithi069.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:38:07.069 INFO:teuthology.orchestra.run.smithi163.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:38:07.070 INFO:teuthology.orchestra.run.smithi103.stderr:bash: line 1: ntpq: command not found 2024-05-09T10:38:07.073 INFO:teuthology.orchestra.run.smithi069.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:38:07.073 INFO:teuthology.orchestra.run.smithi069.stdout:=============================================================================== 2024-05-09T10:38:07.073 INFO:teuthology.orchestra.run.smithi069.stdout:^* hv01.front.sepia.ceph.com 3 7 377 54 +555us[ +563us] +/- 42ms 2024-05-09T10:38:07.074 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -1188us[-1188us] +/- 51ms 2024-05-09T10:38:07.074 INFO:teuthology.orchestra.run.smithi069.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 52 +280us[ +280us] +/- 68ms 2024-05-09T10:38:07.074 INFO:teuthology.orchestra.run.smithi069.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:=============================================================================== 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:^* hv01.front.sepia.ceph.com 3 7 377 114 +603us[ +595us] +/- 41ms 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv02.front.sepia.ceph.com 3 7 377 119 -1095us[-1103us] +/- 80ms 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 57 +297us[ +297us] +/- 68ms 2024-05-09T10:38:07.098 INFO:teuthology.orchestra.run.smithi163.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:=============================================================================== 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:^* hv01.front.sepia.ceph.com 3 6 377 51 +633us[ +499us] +/- 42ms 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 53 -1097us[-1231us] +/- 51ms 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 53 +347us[ +214us] +/- 68ms 2024-05-09T10:38:07.117 INFO:teuthology.orchestra.run.smithi103.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-05-09T10:38:07.118 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-05-09T10:38:07.129 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-05-09T10:38:07.129 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-05-09T10:38:07.160 DEBUG:teuthology.orchestra.run.smithi069:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T10:38:07.288 DEBUG:teuthology.orchestra.run.smithi103:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T10:38:07.441 DEBUG:teuthology.orchestra.run.smithi163:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-05-09T10:38:07.557 DEBUG:teuthology.orchestra.run.smithi069:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T10:38:07.589 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:07.589 DEBUG:teuthology.orchestra.run.smithi103:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T10:38:07.620 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:07.620 DEBUG:teuthology.orchestra.run.smithi163:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-05-09T10:38:07.646 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:07.647 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-05-09T10:38:07.658 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-05-09T10:38:07.687 INFO:teuthology.task.internal:Duration was 5837.020133 seconds 2024-05-09T10:38:07.687 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-05-09T10:38:07.699 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-05-09T10:38:07.700 DEBUG:teuthology.orchestra.run.smithi069:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T10:38:07.702 DEBUG:teuthology.orchestra.run.smithi103:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T10:38:07.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-05-09T10:38:07.751 INFO:teuthology.orchestra.run.smithi163.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:38:07.753 INFO:teuthology.orchestra.run.smithi069.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:38:07.761 INFO:teuthology.orchestra.run.smithi103.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-05-09T10:38:08.184 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-05-09T10:38:08.185 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi069.front.sepia.ceph.com 2024-05-09T10:38:08.185 DEBUG:teuthology.orchestra.run.smithi069:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T10:38:08.206 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi103.front.sepia.ceph.com 2024-05-09T10:38:08.206 DEBUG:teuthology.orchestra.run.smithi103:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T10:38:08.229 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2024-05-09T10:38:08.230 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-05-09T10:38:08.271 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-05-09T10:38:08.271 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T10:38:08.274 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T10:38:08.277 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-05-09T10:38:08.347 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-05-09T10:38:08.348 DEBUG:teuthology.orchestra.run.smithi069:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T10:38:08.350 DEBUG:teuthology.orchestra.run.smithi103:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T10:38:08.352 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-05-09T10:38:08.735 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-05-09T10:38:08.746 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-05-09T10:38:08.747 DEBUG:teuthology.orchestra.run.smithi069:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T10:38:08.778 DEBUG:teuthology.orchestra.run.smithi103:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T10:38:08.817 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-05-09T10:38:08.841 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-05-09T10:38:08.854 DEBUG:teuthology.orchestra.run.smithi069:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.857 DEBUG:teuthology.orchestra.run.smithi103:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.859 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.885 INFO:teuthology.orchestra.run.smithi069.stdout:kernel.core_pattern = core 2024-05-09T10:38:08.901 INFO:teuthology.orchestra.run.smithi103.stdout:kernel.core_pattern = core 2024-05-09T10:38:08.926 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2024-05-09T10:38:08.943 DEBUG:teuthology.orchestra.run.smithi069:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:08.958 DEBUG:teuthology.orchestra.run.smithi103:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.975 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:08.975 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2024-05-09T10:38:08.989 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-05-09T10:38:08.990 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-05-09T10:38:09.003 INFO:teuthology.task.internal:Transferring archived files... 2024-05-09T10:38:09.004 DEBUG:teuthology.misc:Transferring archived files from smithi069:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi069 2024-05-09T10:38:09.004 DEBUG:teuthology.orchestra.run.smithi069:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T10:38:09.048 DEBUG:teuthology.misc:Transferring archived files from smithi103:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi103 2024-05-09T10:38:09.048 DEBUG:teuthology.orchestra.run.smithi103:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T10:38:09.084 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717/remote/smithi163 2024-05-09T10:38:09.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-05-09T10:38:09.122 INFO:teuthology.task.internal:Removing archive directory... 2024-05-09T10:38:09.122 DEBUG:teuthology.orchestra.run.smithi069:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T10:38:09.125 DEBUG:teuthology.orchestra.run.smithi103:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T10:38:09.127 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2024-05-09T10:38:09.170 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-05-09T10:38:09.183 INFO:teuthology.task.internal:Not uploading archives. 2024-05-09T10:38:09.184 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-05-09T10:38:09.232 INFO:teuthology.task.internal:Tidying up after the test... 2024-05-09T10:38:09.232 DEBUG:teuthology.orchestra.run.smithi069:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T10:38:09.235 DEBUG:teuthology.orchestra.run.smithi103:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T10:38:09.237 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-05-09T10:38:09.247 INFO:teuthology.orchestra.run.smithi069.stdout: 262165 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:38 /home/ubuntu/cephtest 2024-05-09T10:38:09.249 INFO:teuthology.orchestra.run.smithi103.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:38 /home/ubuntu/cephtest 2024-05-09T10:38:09.251 INFO:teuthology.orchestra.run.smithi163.stdout: 262161 4 drwxr-xr-x 2 ubuntu ubuntu 4096 May 9 10:38 /home/ubuntu/cephtest 2024-05-09T10:38:09.253 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-05-09T10:38:09.279 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-05-09T10:38:09.429 INFO:teuthology.nuke:Checking targets against current locks 2024-05-09T10:38:09.454 DEBUG:teuthology.nuke:shortname: smithi069 2024-05-09T10:38:09.455 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:38:09.480 DEBUG:teuthology.nuke:shortname: smithi103 2024-05-09T10:38:09.480 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:38:09.485 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi069.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.282262', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBX9THh0AhAnU7ZbVL9H/c8OP5pzCaiV1UYunD6EkJPf32epHj6b05KR8z/e5Wj3mGvjRk5aokO9d7SrweOm5pY='} 2024-05-09T10:38:09.506 DEBUG:teuthology.nuke:shortname: smithi163 2024-05-09T10:38:09.506 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-05-09T10:38:09.511 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi103.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.284308', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM60uisW48ClpMmF1puX7MslpBdWqxj1j9FocMvCZ5jhgBMsWjqySzYa0FxPhOEsSt/m41GP/XP2P9hH1avQ3CM='} 2024-05-09T10:38:09.545 INFO:teuthology.orchestra.console.smithi069:Power off 2024-05-09T10:38:09.545 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T10:38:09.554 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-05-08_21:24:04-fs-squid-distro-default-smithi/7698717', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-05-09 08:51:27.283476', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFn8t6I/B1Ndke2vAFl68IOQ3TOq7hSo/j4FuloIK8aChMbV8GlABziQNKfxDENANehxem1LFs8bNCWt6zwliDs='} 2024-05-09T10:38:09.570 DEBUG:teuthology.orchestra.console.smithi069:power off output: Chassis Power Control: Down/Off 2024-05-09T10:38:09.570 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:09.587 INFO:teuthology.orchestra.console.smithi103:Power off 2024-05-09T10:38:09.587 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T10:38:09.601 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-05-09T10:38:09.611 DEBUG:teuthology.orchestra.console.smithi103:power off output: Chassis Power Control: Down/Off 2024-05-09T10:38:09.611 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:09.626 INFO:teuthology.orchestra.console.smithi163:Power off 2024-05-09T10:38:09.626 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-05-09T10:38:09.636 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-05-09T10:38:09.649 DEBUG:teuthology.orchestra.console.smithi163:power off output: Chassis Power Control: Down/Off 2024-05-09T10:38:09.649 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:09.693 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-05-09T10:38:13.601 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:13.637 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:13.694 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:13.711 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-05-09T10:38:13.747 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is on 2024-05-09T10:38:13.804 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is on 2024-05-09T10:38:17.714 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:17.748 DEBUG:teuthology.orchestra.console.smithi103:pexpect command: ipmitool -H smithi103.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:17.806 DEBUG:teuthology.orchestra.console.smithi163:pexpect command: ipmitool -H smithi163.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:17.824 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is on 2024-05-09T10:38:17.859 DEBUG:teuthology.orchestra.console.smithi103:check power output: Chassis Power is off 2024-05-09T10:38:17.917 DEBUG:teuthology.orchestra.console.smithi163:check power output: Chassis Power is off 2024-05-09T10:38:17.960 INFO:teuthology.orchestra.console.smithi103:Power off completed 2024-05-09T10:38:18.018 INFO:teuthology.orchestra.console.smithi163:Power off completed 2024-05-09T10:38:21.829 DEBUG:teuthology.orchestra.console.smithi069:pexpect command: ipmitool -H smithi069.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-05-09T10:38:21.938 DEBUG:teuthology.orchestra.console.smithi069:check power output: Chassis Power is off 2024-05-09T10:38:22.039 INFO:teuthology.orchestra.console.smithi069:Power off completed 2024-05-09T10:38:22.144 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate 3-modules} clusters/1a11s-mds-1c-client-3node conf/{client mds mgr mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/yes} objectstore-ec/bluestore-comp-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts pg_health session_timeout} ranks/multi/{balancer/automatic export-check n/3 replication/always} standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/dbench}} duration: 5837.020133256912 failure_reason: 'error during scrub thrashing: rank damage found: {''backtrace''}' flavor: default owner: scheduled_teuthology@teuthology status: fail success: false 2024-05-09T10:38:22.144 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-05-09T10:38:22.318 INFO:teuthology.run:FAIL